I think there have been two main changes that have hurt the web:
1) The shift from spammy shit content as something to squash to something to allow and even promote over better content, provided it follows certain (Google's) rules. This shift (in terms of Google's behavior) happened ~2008-2010 and we haven't seen a period of spammy crap content getting heavily downranked since then, like we used to when they were still trying to stay ahead of it rather than give it a "legitimate" avenue as a method of control. Google's still being the most important search provider to appease has left the rest unable to direct behavior toward anything better than what does well on Google, so their results aren't much better.
2) A move away from actual or de facto open systems & protocols to deliberately carved up communities. The only thing keeping chat, Twitter-like services, and other social media—hell, even Youtube, so far as some kind of format for hosting video with metadata—from being standards or protocols is that business incentives reward "owning" a userbase (so you can better spy on them, and to keep anyone from providing a better, perhaps less-spying-laden client and "stealing" ad-viewing eyeballs)
Both of these are fundamentally problems of the spyvertising economy taking over the Web and I think a lot of the issues would go away if we could (legally—I don't think tech will do it) permanently and completely break that. More specifically a big part of the problem is Google, though of course the rest of the Web giants are gleefully following similar bad incentives.
I don't think search engines are promoting spammy $#!+ content per se; what they're doing is heavily promoting newer content that's relevant to the most common search queries, as this gives them the only real hope of staying ahead of the spam. Of course the "small", long-lasting, independent Web is heavily disadvantaged by this shift.
One development that would be good for small web sites to look into is schema.org linked-data formats. Those might simply be too effort-intensive for the spammers to adopt (at a high level of detail) and perhaps too much of a commitment to quality and transparency (they would have to actively forge the info, which would leave them open to bans given the lack of plausible deniability), so they might become a viable signal of quality and lead to higher visibility in SERP.
(Similar for things like proper separation of style from content, that have always been advocated for in the web-standards community but are not really commercially viable.)
I'm not quite sure if others have experimented with this stuff already, but it seems worth trying.
1) The shift from spammy shit content as something to squash to something to allow and even promote over better content, provided it follows certain (Google's) rules. This shift (in terms of Google's behavior) happened ~2008-2010 and we haven't seen a period of spammy crap content getting heavily downranked since then, like we used to when they were still trying to stay ahead of it rather than give it a "legitimate" avenue as a method of control. Google's still being the most important search provider to appease has left the rest unable to direct behavior toward anything better than what does well on Google, so their results aren't much better.
2) A move away from actual or de facto open systems & protocols to deliberately carved up communities. The only thing keeping chat, Twitter-like services, and other social media—hell, even Youtube, so far as some kind of format for hosting video with metadata—from being standards or protocols is that business incentives reward "owning" a userbase (so you can better spy on them, and to keep anyone from providing a better, perhaps less-spying-laden client and "stealing" ad-viewing eyeballs)
Both of these are fundamentally problems of the spyvertising economy taking over the Web and I think a lot of the issues would go away if we could (legally—I don't think tech will do it) permanently and completely break that. More specifically a big part of the problem is Google, though of course the rest of the Web giants are gleefully following similar bad incentives.