Sorry for the late reply—I wanted to give you a serious answer and didn't have time earlier.
The criteria for "substantive" are what the HN guidelines (along with https://news.ycombinator.com/newswelcome.html) say about which stories are on topic: intellectually interesting (as opposed to sensational) and so on. HN's culture is well-enough established by now that I think most community members can probably agree, not of course on what interests each of us personally, but on a working set of candidate stories that roughly fit the criteria. That's our hypothesis, anyhow.
HN sees perhaps a thousand new stories a day. Most aren't "intellectually interesting" in HN's sense (i.e. gratifying intellectual curiosity), and the weeds have grown too thick for the upvote system alone to reliably surface the potential gems. To comb through /newest looking for them has become too much work.
Some interesting stories (again, "interesting" in the HN sense) do fine under the upvote system. Breaking news, stories about fashionable technologies, and anything controversial reliably attract upvotes. But the quieter and deeper submissions are often not immediately recognizable. Those deserve closer attention than they were getting, so we've been experimenting with different approaches to finding them.
In the spirit of "do things that don't scale", it's mostly us trying different things manually for now, but what we're looking for is a mechanism that can be opened to anyone willing to put in the effort. The upvote system will work exactly as it always has, but we're hoping to add a new one that complements it and compensates for its weaknesses.
Since it requires significant effort to look through the story stream hunting for out-of-the-way pieces, one idea (my favorite) is to make this a new way of earning karma on the site. But first we have to find something repeatable that will work.
