> How can you smugly dismiss their point like that?

Because "it's fine if you're rich" is a shitty place to terminate the matter. It's not a useful counterargument in any sense. Of course it's fine if you're rich, that's why people want to be rich. What about, you know, the rest of the planet?