Well yes, that's what "polarizing" means, that everything is pushed towards one of two groups, rather than being allowed to be anything in the middle. That's why it seems to me that Rogan is not polarizing, because his opinions don't strictly belong to one of the two groups.