My experience is with gcc and clang on x86. I generally agree with you regarding accuracy, which is why I was quite surprised when I first discovered that -Ofast breaks isnan/isinf.

Even if I don't care about the accuracy differences, I still need a way to check for invalid input data. The upshot is that I had to roll my own isnan and isinf to be able to use -Ofast (because it's actually the underlying __builtin_xxx intrinsics that are broken), which still seems wrong to me.