I play MSFS a lot with realistic add-on planes like the A32NX or the PMDG 737, even though the flight model isn't quite "as good" as XP or P3D, the visual dominance of MSFS adds so much to the immersion for me.
Well, I honestly put flight modelling over graphics every single day. X-Plane 11 might not be as good as MSFS in terms of graphics but everything else (i.e. the really important things) is.
For me, I am not a real pilot (yet) and can't really tell the difference in 99% of cases. Also, the fidelity of 3rd party addon planes in MSFS is just as good if not better than in XP.
I heard somewhere it can be counterproductive to learn stick and rudder skills in sim, that sim time is better spent learning how to use instruments, automation, fly complex procedures like DME arc, etc. So maybe the fidelity of aircraft system models (vs aerodynamics) is also really important?
There's some amount of the physical feedback you get in a real plane (from resistance in flight controls to actual physical sensation of the movement) is not something you can get without a lot more sophisticated sim setup than most people will have at home and then also as you mention the model of the flight characteristics not matching the real airplane
There was a "flight simulator" ported from SGI GL to OpenGL+Windows when Microsoft embraced OpenGL. There were a small variety of aircraft in this program. I could take off in the 747, kill the engines and glide to a safe landing. Considering I have no experience whatsoever with real aircraft, that's probably not realistic.