NetFlix Prize a Case Study in Incentives for Innovation

The use of prizes to drive innovations is an idea that is not exploited as much as it should. One corporation that has put this idea to the firmest of tests is Netflix which put out an announcement to award the Netflix Prize of US$ 1 million to any individual or team which would draw an algorithm that would improve its ability to predict the movies that its customers would like by 10% from an existing level. A good number of teams using sophisticated mathematical and reasoning techniques have been crunching the numbers to since 2006.

Generating an improvement of 10% sounds easy until one finds that the last mile is the most difficult. NYT Magazine’s columnist, Clive Thompson traces the interesting experiment here and reports on some of the contenders for the prize, in addition to the frustrations faced by the competing teams as they try to reach the last rung. The difficulty of the task is discernible from the fact that the top team has gone through close to 95% of the tasks and yet the ultimate formula has not yielded.

For any student of applied economics, it is interesting to note that the corporation defined the task by building in one large ultimate prize and periodic prizes to keep the motivation of the teams. Secondly, the information that each team generates is soon available to the others and that helps to increase the overall amount of knowledge that is generated. The most fascinating fact though is that a number of teams are comprised of professionals who work on the problem on a part time basis or teams constituted in academia.

While I am neither good with models nor writing computer code, I intend to enter the contest on my own just so I may gain access to that interesting database of movies. I think that it can be used in many more ways than just cracking the 10% improvement threshold.

This Netflix approach is not new in principle but the results will perhaps change public views on how longstanding technical problems could be tackled. Regarding the subject here, I venture that one of the teams will crack the threshold before the prize expires.