Communications of the ACM
Award-Winning Paper Reveals Key to Netflix Prize
"The way I rate movies today can be very different from how I rate them even tomorrow," says Yehuda Koren of Yahoo! Research Israel. Those differences are accounted for by the improved algorithm that makes Koren's team the likely winner of the NetFlix Pri
Credit: Yehuda Koren
When the organizers of the Netflix Prize contest announced late last week that one team had met the requirement for the $1 million Grand Prize, Yehuda Koren, a member of the seven-person multinational team, was in Paris to present a paper at KDD-09, the 15th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. The ideas he laid out won the conference's Best Paper Award — and, not coincidentally, had much to do with reaching the contest's target of improving the accuracy of Netflix movie recommendations by 10 percent.
In the paper, Koren, a senior research scientist at Yahoo! Research Israel in Haifa, showed a way to improve Netflix's recommender algorithm by using information about changes in the ratings over time. His role on the winning team, BellKor's Pragmatic Chaos, involved refining the model that deals with temporal dynamics.
While movies themselves stay the same, the humans who rate them are anything but static. As Koren puts it, "The way I rate movies today can be very different from how I rate them even tomorrow." To the frustration of Netflix Prize contenders, a four-star rating can mean "great" or merely "so-so" depending on the user's current mood or comparisons with other recently seen movies. Besides such erratic shifts in the rating scale, people's actual tastes tend to change over time — as when someone tires of action films, for example, and develops a yen for screwball comedies. There's an overall rise in ratings over the years, as well: for various reasons, a typical movie's ratings become more favorable as the DVD ages.
Other investigators had noticed such time-based dynamics in the Netflix dataset, but, unlike Koren, they had trouble putting them to successful use. "This is the first paper that does a good job of giving a mathematical model of these phenomena," says Charles Elkan, a Netflix Prize judge and Professor of Computer Science and Engineering at the University of California, San Diego.
Past attempts at modeling temporal dynamics in collaborative filtering or recommender systems weren't sensitive enough, Koren explains: although recent data may reveal more about a user's current preferences than older data, simply underweighting older ratings loses too much valuable information for that approach to work. The trick to not tossing the baby with the bathwater is to retain everything that predicts the user's long-term behavior while filtering out temporary noise. To that end, Koren's model has one component for absorbing the lasting signal and another for detecting signs of transient noise, such as certain abrupt changes in the ratings.
Though BellKor’s Pragmatic Chaos surpassed current Netflix accuracy by 10.05 percent, the Grand Prize technically isn't theirs just yet: other participants have until July 26 to match or beat BellKor's feat. But the time-based methods are already a winner. According to Koren, they've helped Yahoo! improve the relevance of the ever-changing News stories displayed on the site's home page.
Marina Krakovsky (marinakrakovsky.com) is a writer in the San Francisco Bay Area.
Comments
Deepak Agarwal
I like this paper,it shows how incorporating time effects can help improve predictive accuracy in recommender systems. We have successfully used such a strategy for optimizing content on Yahoo! front page (the application referred to in the last line of the article). Although time based methods have helped us tremendously for this application, the methods used are different from what has been described by Yehuda in his KDD paper; there are some key differences between movie recommendation and web application. In particular, we have effectively used explore/exploit methods coupled with time series models with great success. The methods that have been used to optimize content on Yahoo! are described in the three papers below. There is another paper under review with Journal of the American Statistical Association that describes the explore/exploit methods in detail.
D.Agarwal, B.Chen, P.Elango,R.Ramakrishnan,S.Park,S.Roy,N.Motgi and J.Zachariah.
Online models for content optimization
In NIPS, 2008.
D.Agarwal, B.Chen and P.Elango
Spatio-Temporal models for estimating click-through rates
In WWW, 2009
D.Agarwal and Bee-Chung Chen
Regression based Latent Factor Models
In KDD, 2009
Displaying 1 comment