Use those other things for punishment then, don't sting them additionally just because they're wealthier (or rather, have a higher income).
Getting off track here, but I'd rather there were as few tax rates as practical. Perhaps even one under a basic income system. Then the rate is upfront, clear and honest. Not a compounding system of stealth taxes.
Worth noting that wealth is not the same as income, but we tend to tax income rather than wealth, at least until death.
It needs to sting for everyone, not just the poor.
Do you reject higher tax brackets for the wealthy under that same line of reasoning?
Surely wealth is an important factor in deciding what people owe to society