Long Term versus Evolutionary Thinking (Part 2 of 2)

Continued from Part 1.

Now that I’ve fully (perhaps too much so) explained the analogy that I will be using, I’d like to ground this discussion in the subject of information retrieval.  And I’ll start with an example that O’Reilly used in his talk: Google. (This is an Information Retrieval blog, after all, and Google was the example that Tim used.)  The company, he says, successfully exhibits both long term and evolutionary thinking.  It takes the long term view through its very mission statement: “To organize the world’s information”.  What could be more long term, more global, than that?  At the same time, Google has a very evolutionary approach in that it starts with simple, elegant solutions and couples them with ongoing user measurements.  If and when changes to Google’s engine are made, they are made based on small evolutionary steps that become apparent through the actions of the user.  It’s a point of pride within the Google organization that every change to the engine is scrupulously measured and A/B tested so as to be able to tell whether the change was better or worse.  The user provides the fitness function, the arrow that points in the uphill direction, toward which the search engine evolves. 

So the question is, does Google suffer from this conflict between long term and evolutionary thinking?  My contention is that they do.

Early on in the company’s history, the observation was made that “simpler is better”.  Rather than distracting the user with non-relevant clutter, Google observed that users were more successful at their information seeking tasks when the interface was simple and streamlined.  So for over ten years now, Google’s home page is no more complex than a single line input box and a search button. As a result of this focus on simplicity, all the steps that Google subsequently took with respect to improving their engine did not, could not, violate this simplicity principle.  It’s almost as if, in the process of evolutionarily, iteratively hill-climbing its way to the best solution, Google got caught on the local hill of simplicity:

Why do I think that “simplicity” is a local maximum on which Google got snagged?  I’ve read the research, in particular an interesting paper by Belkin et al entitled “Query Length in Interactive Retrieval“.  These researchers had three main hypotheses:

1. A search interface which asks searchers to describe their information problems at length will lead to longer queries than one which asks searchers to simply input a query as a list of words or phrases.
2. A system which encourages long queries will lead to better performance in the search task than one which does not.
3. Query length will be positively correlated with performance in the search task. 

The researchers tested these hypotheses using a 5-line query input box with the following instructions in a label above the box: “Information problem description (the more you say, the better the results are likely to be)“.  The conclusions of the study indicated the following three findings:

• This technique does indeed result in significantly longer queries [5.45 query terms on average] than even a somewhat enhanced baseline technique, and is at least as usable, and perhaps more usable, than the baseline technique; 
• The technique results in increased searcher satisfaction with search results, and “objective” performance in the task equivalent to the baseline, but with fewer query iterations; 
• Longer queries irrespective of query elicitation mode are  significantly associated with increased searcher satisfaction with search results, and longer initial queries lead to equivalent “objective” results with fewer query iterations. 

The researchers concluded:

Thus, we conclude that our quite simple interface-based query  elicitation technique results in significantly longer, and more useful searcher queries in a Web searching task than typical query elicitation, for a best-match information retrieval system. Furthermore, we conclude that longer searcher queries result in increased search effectiveness in general, indicating that more words from the searcher describing the person’s information problem results in better interactive IR performance. Taken together, our results mean that getting longer queries from searchers in a best-match Web searching environment is not only possible, but desirable and useful.

These hypotheses stand in stark contrast to Google’s focus on simplicity.  A 5-line input box is more complex than a 1-line field, and so the 5-line box is not a solution that Google would be likely to explore.  And a label above the query input box, asking users to be as descriptive as possible adds “clutter” to the interface, and would be an evolutionary step away from the simplicity locally maximal hill, and so would not be something that Google would try.  

As a result of Google being stuck on this simplicity locally-maximal hill, Google’s users enter significantly shorter queries on average (last I’d heard it was approximately 2.2 terms per query) and more than likely end up having to iterate a lot more than they otherwise would have to if they had used longer queries in the first place.  Despite the fact that this approach demands more total work for the user, Google actively encourages this behavior.  Google’s Guidelines for Better Search help page explicitly instructs the user to:

Describe what you need with as few terms as possible. The goal of each word in a query is to focus it further. Since all words are used, each additional word limits the results. If you limit too much, you will miss a lot of useful information. The main advantage to starting with fewer keywords is that, if you don’t get what you need, the results will likely give you a good indication of what additional words are needed to refine your results on the next search.

So even though research has found that longer queries lead to more satisfied users, and that larger query input boxes lead to longer queries, Google is unable to take an evolutionary step in that direction.  That step violates their current locally-maximum hill principle of simplicity.  They seem fundamentally incapable of passing through the valley of complexity to reach an even higher effectiveness peak because evolutionary thinking does not allow them to take that large leap necessary.  They can only follow their current gradient.  In ten years of using Google, I don’t think that I have ever seen, even for brief experimental time periods, a query input box that was taller than one line.  Thus, evolutionary thinking conflicts with long-term goals.

I think that Google understands, even though they don’t publicly admit, that they need to make changes to (1) their UI and (2) the back-end algorithmics that support that UI, in order to increase the interactivity of the search engine, and increase user productivity and satisfaction.  A couple of years ago, they created an experimental playground called “SearchMash”.  (See here and here and  here.)  The idea was that they could take larger leaps on SearchMash than they could on their main Google.com site.  SearchMash clearly shows that Google was trying to think longer term.  

But the very existence of a separate site, a separate search playground, reinforces my original point about how long term thinking and evolutionary thinking are in conflict.  Google found itself fundamentally incapable of running SearchMash-like experiments live on its own main search engine.   It was too bound by its own expectations, and the self-reinforcing expectations that it created in its users, around simplicity.  When it wanted to think longer-term it was unable to, because users could only handle small, evolutionary changes.  It therefore had to a create a whole separate, disconnected space to explore these ideas.  SearchMash was never promoted on the main Google page, and if you weren’t following the blogosphere you might have never known that it existed.  Is it any wonder that, after less than two years, the site is now shut down?  

Perhaps my hypothesis about the fundamental conflict between long term and evolutionary thinking is nothing more than a re-phrasal of Clayton Christensen’s Innovator’s Dilemma.  By sticking too close to their current users, and the expectation set up around those current customers, Google’s evolutionary thinking overrules its long term thinking.  But in order to think longer-term (e.g. by playing with things like SearchMash) Google has to destroy, or at least depart from, the current simplicity mantra that made it what it is today. It has to take a leap, rather than an evolutionary step: Google’s long term thinking overrules its evolutionary thinking.  Inasmuch as I understand it, the tension between these two extremes is the source of the Innovator’s Dilemma.  Google thinks that it is immune from this conflict; Tim O’Reilly (or at least my flawed understanding of Tim O’Reilly — definitely a possibility) doesn’t think that this tension exists.  I don’t think Google is immune, and I do think that this tension exists.

Google is not “bad” or “evil” for having to struggle with this tension.  Everybody does.  To the extent that they think they are immune from this tension they may be guilty of a little hubris.  But there is evidence that even that is changing as well.  A couple of years ago Google did not allow any technology-focused employee to have a job title other than “Engineer”.  Engineers evolutionarily iterate.  However, a few years ago Google started allowed the job title of “Researcher”, i.e. someone who takes larger, exploratory leaps.  So I see evidence that they are dealing with this conflict.