I think it's because short sellers are seen as pessimists -- on some level, you're betting for a business to fail (or perform poorly), and regardless of whether or not you're right, that means you're betting on people losing jobs, dreams going up in smoke, etc. etc.
I don't subscribe to this narrative, but it's an easy one to spin when your stock is the one being short sold.
I think there is a difference in the company itself being shady and investors being shady to influence the stock price. The long pump and dumpers definitely fall into the latter category and similar to the shorts described in my link.