> there's huge pressure to report positive results.
This is basically just noting Goodhart's Law, or as I like to say "Warning." Evaluation is incredibly difficult and every metric can be hacked. Making "meritocracy" a fool's errand since a metric alone is not enough (it needs the associated uncertainty value, which is non-zero).
The problem here is that we're searching through an incredibly high dimensional non-convex solution space and assuming that a simple optimization function is a global optimizer.
Instead I'd just love to see academics pursuing their ideas and discussing them in the open. I don't think you'd have a significant problem getting people to generally align in a single direction, as currently happens, but I think it also allows for others to challenge those directions as well. Essentially research is performing some sampling in that search space but we're turning the learning rate way down. So we get plenty of works where novelty and incrementalism is indeterminable, due to the speed in which we must produce. Instead let the academics determine their own parameters and let time sort it out. Ensembling is a very useful technique, especially when uncertainty is high.
This is basically just noting Goodhart's Law, or as I like to say "Warning." Evaluation is incredibly difficult and every metric can be hacked. Making "meritocracy" a fool's errand since a metric alone is not enough (it needs the associated uncertainty value, which is non-zero).
The problem here is that we're searching through an incredibly high dimensional non-convex solution space and assuming that a simple optimization function is a global optimizer.
Instead I'd just love to see academics pursuing their ideas and discussing them in the open. I don't think you'd have a significant problem getting people to generally align in a single direction, as currently happens, but I think it also allows for others to challenge those directions as well. Essentially research is performing some sampling in that search space but we're turning the learning rate way down. So we get plenty of works where novelty and incrementalism is indeterminable, due to the speed in which we must produce. Instead let the academics determine their own parameters and let time sort it out. Ensembling is a very useful technique, especially when uncertainty is high.