It isn't just that. -ffast-math also allows the compiler to ignore infinites. In fact for GCC with -ffast-math, isinf always returns false. Something similar happens for NaNs/isnan.
I lump this into "floating points are tricky". NaNs and inf are definitely legitimate floating point values. They are also things that a lot of applications will break on they ever encounter them.
