Last March I pointed out a short piece by Tessa Lau about how good interaction design trumps smart algorithms. Today I have a followup. In particular, Xavier Amatriain has a good writeup of the recently concluded Netflix contest. Some of the lessons learned by going through the process are related to the importance of good evaluation metrics, the effect of (lapsed) time, matrix factorization, algorithm combination, and the value of data.
Data is always important, but what struck me in the writeup was his discovery that the biggest advances came not from accumulation of massive amount of data, log files, clicks, etc. Rather, while dozens and dozens of researchers around the world were struggling to reach that coveted 10% improvement by eking out every last drop of value from large data-only methods, Amatriain comparatively easily blew past that ceiling and hit 14%.
How? He simply asked users to denoise their existing data by rerating a few items. In short, Amatriain resorted to HCIR:
Well, that is exactly what we have done in a paper that has been accepted for publication in Recsys09 (NY). The paper is entitled “Rate It Again” and you can access a preprint copy here. The basic idea in our approach is to ask users to re-rate items that they already rated in the past. We can then denoise ratings that prove to be inconsistent by minimizing their contribution to the recommendation process…We measured relative improvement in terms of RMSE up to 14% and we verified that this is consistent regardless of the particular recommendation algorithm (item and user-based CF, SVD, etc…).
Amatraian chalks this up as another victory for data-driven methods. And it’s true; collecting a little more data yielded vast improvements. But my interpretation of Amatriain’s work is not that of typical data-driven methodology. This isn’t the collection of mass- or web-scale amounts of unlabeled data, as is recommended by Halevy, Norvig, and Pereira. This is an interactive, small-scale, dialogue-oriented data. This is a small amount of human intelligence injected back into an otherwise faceless algorithm. This is what HCIR is all about:
Marchionini’s main thesis is that “HCIR aims to empower people to explore large-scale information bases but demands that people also take responsibility for this control by expending cognitive and physical energy.”
How best to elicit this interaction from the users, convincing the users of value while simultaneously minimizing the amount of effort (or the perceived amount of effort) required of the users is the purpose of interaction design of the sort Lau writes, above. And much more work needs to be done in this area. But I do find it interesting that this work shows that putting users back into the process yields much larger improvements than the combination of hundreds of data-only driven algorithms.
See also my previous posts: