I just got back from a PARC open forum, in which Marissa Mayer gave a talk about the Physics of Data, and Innovation at Google. All in all, it was fine. Maybe a third of the talk was about new possibilities enabled by large quantities of data (Google Flu Trends, better search, etc.) The other two-thirds were dedicated to introducing the audience to some longer-tail Google products that many folks might not have known about. So I’m not going to go into detail about the talk as a whole, but I will point out two tidbits that were the most interesting to me. One is positive, one is negative.
On the positive side, Mayer fielded a lot of questions about large data and A/B testing. There is a lot of talk in the web community about large scale A/B testing, and how once you’ve used it to gauge the most effective column width, relevance algorithm, or link color, you can deploy that change to your entire user base. Mayer did say something, though, that I’ve never heard anyone else say in this domain. She said that they try not to overfit. Perhaps that’s par for the course, and something I should have known. But my impression of A/B testing is that once there has been a clear, statistical winner, companies will put that winner into action, at least until the next round of testing. Mayer instead suggested that if you always deploy the winner, you run the risk of overfitting. And you might not necessarily want to do that. Interesting. My respect for Google went up a bit.
On the negative side, I was put off by some of her responses to questions about advertising. One fellow a few rows in front of me asked the question that I often ask on this blog: If Google search were to return in the organic results the same pages to which the ads on the right already link, would there be a need for those same advertisements (implicit answer: no). So isn’t there a financial incentive, he asked, for Google not to make its organic results too good? (Not that Google would actively worsen those results, but that it wouldn’t try as hard to make those results as good as they could be, because to do so would be to lower ad revenue.) I was disappointed that Marissa essentially dodged the question. She gave the standard answer, which is that all advertising is clearly labeled. I felt it was a hollow answer, and did not address the questioner’s core point.
But dodging a question isn’t what irked me. After Mayer got finished explaining that a web results page with advertising is better than one without advertising, based on a 6.5 year-long A/B test that she had conducted, another audience member asked her whether there could ever be too much of a good thing, in that perhaps web advertising (number of ads shown on your results) has increased to the point where there is simply too much of it. She smiled, and said that she did not think that advertising had increased.
Now, I know for a fact that is not true. Five years ago I used to do Google searches, and only 1-8 ads would show up on the right hand side. Maximum 8. Today, not only do 1-8 ads show up on the right, but 1-3 ads show up above the top organic result. Maximum 11. Today there are up to 11 ads on a page with 10 organic links. The amount of advertising shown on a results page has indeed increased. And you would think that, with all the massive quantities of data available, Ms. Mayer would be able to spot the same fact that is available to an average user. So her answer was a bit disappointing.
Overall, I left with mixed feelings. Some good, some bad, but overall it was fun.