It's not about doing it without a mouse, it's about doing it fast, comfortably, and powerfully. Pointers are a useful addition to graphic capable vi implementations, but if you do everything with mouse or trackpad movements then I don't think you can claim the zen of vi.
My fascination is that the original motions were not invented to escape the imprecision of graphical pointers, even though that's their main selling point today. They were invented to minimize (1) excessive tty printing, because printing is slow (2) keystrokes.
That, in 2020, these optimizations made in 1960 allow you to avoid the imprecision and slowness of a graphic pointer, we get for free :)
It's fascinating to read comments praising how great it is that you can edit text without a mouse.
ed was written to interface with tty systems, where mice weren't even a thing to be avoided. :)