Let's say the man is the moderator of a predominantly female workforce, like nurses. At best, this would come off as unprofessional but flattering. But the real problem is the underlying assumption that a good-looking person who happens to be in a predominantly other-gendered profession is going to be treated like chum to the sharks. In other words, the thinking behind saying something like that is "wow, that person is really good looking and everyone wants to have sex with them, so what's a good way to deflect that?" It's really more of an insult to the audience than the speaker.