Yes I have wondered about that. I feel the same about Attention based networks, may be we are not using the most befitting coordinate system to understand them.

https://news.ycombinator.com/item?id=46544981