Could be the motion blur. Vision Pro is in this really weird cross section of insanely good visuals but really bad motion blur. Generally a little motion blur is OK, but the better your visuals get, the worse and more apparent motion blur can be.

Personally I found AVP to be most draining if I'm moving my head around a lot and experiencing this motion blur. If I'm just looking at the screen in front of me, I get fatigued less