I'm not sure it's still the case (and the website is down), but the rules used to state that once someone hit 10%, they would wait a month before awarding the prize, to see if anyone else could beat their solution.
In any case, I'd love to see Netflix start another challenge round. A million dollars isn't a whole lot of money for them, it brings in a lot of publicity, and provides a 'fun' way to get some research done in the data-mining field.
To qualify for the Grand Prize the RMSE of a Participant’s submitted predictions on the test subset must be less than or equal to 90% of 0.9525, or 0.8572 (the "qualifying RMSE"). After three (3) months have elapsed from the start of the Contest, when the RMSE of a submitted prediction set on the quiz subset improves beyond the qualifying RMSE an electronic announcement will inform all registered Participants that they have thirty (30) days to submit additional candidate prediction sets to be considered for judging.
This starts a 30 day period during which all the teams can submit their final versions. After that, I believe Netflix will run the algorithms against the unreleased set of movie rating data.
Since the current teams were using the squared error ratings on the standard test data (not this new test data) to combine their old algorithms, it's quite possible that in one month we will learn that the prize has not yet been won. As I understand the rules, anyway.
No. from the rules -
"The RMSE for the first "quiz" subset will be reported publicly on the Site; the RMSE for the second "test" subset will not be reported publicly but will be employed to qualify a submission as described below. The reported RMSE scores on the quiz subset provide a public announcement that a potential qualifying score has been reached and provide feedback to Participants on both their absolute and relative performance."
The final call announcement would not go out unless a team beat 10% on the test set. That announcement went out at 4:30pm PST today.
It is still quite possible that the entries can be rejected if the reviewers find the system description lacking or if we are unable to reproduce their result. That clock starts after the 30 days runs out.
1) There's not enough information returned by the submission process (a single RMSE number per day).
2) The information you do get is on half of your submission, but it is the other half that counts. So you get no information back on that.
The only candidate I think to make this a surprising finish is the 'Grand Prize Team' which has another of the long-time leaders 'Gravity' involved and has structured itself to accept any and all team members who care to join.
Probably no one would have won this contest if it wasn't for the fact that many of the top teams published their results, and so there was cross-pollination of ideas. Pragmatic Theory didn't contribute in this manner. Kudos to them for making I think a significant improvement on the best of the published models however.
Netflix gets a non-exclusive license from the winner, so they can sell it to other parties, and it seems like something that various parties would be willing to pay for.
An open source version would be a significant act of charity. It would be nice to see older versions or versions that were crippled enough to be education without watering down the reward that the winnners receive.
Algorithms have already been published for the Progress Prizes for 2007 and 2008.
Open sources projects have already been created in Python, Java, C, and C++. None of these projects include the full algorithms from the 2007 or 2008 papers.
The projects can be found from links on the Netflixprize forum:
Agreed. PragmaticTheory team deserves a special commendation since last month they developed an enhancement that was included into the winner's submission and pushed it into the #1 position. Read about PT on Wired: http://www.wired.com/epicenter/2009/06/1-million-netflix-pri...
Did anyone here take a crack at this? I had some ideas that I wanted to try this summer but now its too late. If anyone already has a model built and made submissions using the KNN and SVD methods perhaps we can talk. I thought of a different approach that was not discussed in any of the papers, forums or websites that I read. I know its a long shot but it could happen... yhfin at yahoo dot kom.
Actually receiving the prize will stimulate more people to do ML research. I would be happy to see some other AI competitions similar to IOI, IMO or TopCoder with big publicity.
Netflix are pioneers in such challenges and have the opportunity to be some of the founders of such future competitions.
The teams who created the solutions own them, but they're obligated to license them to Netflix. It doesn't have to be an exclusive license though. Quite generous on Netflix's part if you ask me.
In any case, I'd love to see Netflix start another challenge round. A million dollars isn't a whole lot of money for them, it brings in a lot of publicity, and provides a 'fun' way to get some research done in the data-mining field.