I don't, and I'm pretty baffled that it does affect em units for text-zoom (despite the two having very similar purposes).
Perhaps the difference is that with min font sizes everything gets clamped, which can make two objects of different sizes get the same size if they used em units (if em units affect things). Which is both good and bad, depending on the intent. (Whereas with text-zoom everything is nicely scaled so it's less of a problem).
text-zoom isn't a web feature, it's only a feature Firefox/Gecko has.
Servo doesn't implement it either, only Stylo (the confluence of Servo's style system and Gecko). In Servo mode all the code for text-zoom becomes a no-op.
Perhaps the difference is that with min font sizes everything gets clamped, which can make two objects of different sizes get the same size if they used em units (if em units affect things). Which is both good and bad, depending on the intent. (Whereas with text-zoom everything is nicely scaled so it's less of a problem).