Microsoft Researcher Correctly Predicted 21 Out Of 24 Categories In Oscars

David Rothschild, an economist from Microsoft Research New York City predicted the winners of the Oscars in 2013 using various data available on the web. He was accurate with his prediction in 19 of the 24 Academy Awards categories. For Oscars 2014, he posted his predictions for 24 categories weeks back. The Oscars 2014 event just got over and what about his accuracy of prediction this year? He correctly predicted 21 out of 24 categories which is a great improvement over last year. The three categories which he got wrong were Best Animated Short Film, Best Documentary Feature and Best Live Action Short Film. I guess the lack of enough data is one of the reason behind the above three errors.

David commented the following on the results,

There are two ways to judge accuracy. First, and most obvious, you want a small error. Correctly predicting 21 out of 24 categories and having non-negligible probability for the three other winners is indicative of a small error. Second, you want the predictions to be well calibrated. The top nominee in each category averaged 86.5% likelihood of victory. Thus, multiplied out over 24 categories we expected to pick the winner 20.76 categories. Winning 21 out of 24 means that our predictions were perfectly calibrated. If we picked any more categories we should have had higher probabilities of victory (i.e., if you get 24 out of 24, you should have 100% likelihood for the winner in each of the categories) and if we picked any less we should have had lower probabilities.

My final thought for the night is that this is another case of pundits and insiders advertising a close event when the proper aggregation of data said it was not. As I noted on Twitter earlier, my acceptance speech is short. I would like to thank prediction markets for efficiently aggregating dispersed and idiosyncratic data.

Source: Predictwise