Ranked Lists and the Paradox of Choice

A thread in the comments section over on Daniel’s blog prompted me to think about the issue of ranked list results presentation in a new manner.  How do we decide how far to scroll down a ranked list of results, or when to switch and try a different search query or even a different search engine?  

When a search engine gives you nothing but line after line of results, it doesn’t matter if those results are segmented into pages of 10 or 100, or are infinitely scrolling.  You constantly are faced with the choice of whether or not you should go further in the results to find the information that you seek.  Having 2.7 million results come back as result to your query means that you are faced with 2.7 million choices.  You look at the 1st result and then have to decide whether you want to go on to the 2nd result.  You look at the 2nd result and have to decide whether to look at the 3rd, and so on.  

Conventional wisdom is that a sparse search engine interface, with just the results and not a lot of clutter (other than ads) goes a long way toward reducing the Paradox of Choice. It is said that users do not want to be over-burdened with too many exploratory interface options, query modification possibilities, or other peripheral data.  That only gets in the way of them assessing and choosing from the results.  However, let’s look at Paradox of Choice author Barry Schwartz’s strategy for navigating the overwhelming number of choices we are faced with:

1. Figure out your goal
2. Evaluate the importance of that goal
3. Array the options
4. Evaluate how likely each option is to meet your goal
5. Pick the winning option
6. Modify goal, rinse, and repeat

Now, when using a search engine, users have figured out steps 1 and 2 before typing in their query.  The search engine now comes back and presents the user with step 3, the array of options, the ranked list of results.  The big question is: How well does that array of options help the user evaluate how likely each option is to meet their goal (step 4)?  My answer is: Not very well. In an ideal world, the users goal(s) — relevant document(s) — would all be ranked at the top of the list.  But frequently they are not. When they are not, the user has to scan down the list (the “array of options”) and examine, individually, whether each option meets the goal.  This leads to the paradox of choice, an overwhelming array of options.  The only way to evaluate the next result is to read or skim that next result.

Let’s contrast sparse search engines with exploratory search engines.  Many modes of exploration are possible, but let’s examine “clustering” in light of this paradox of choice. Some clustering techniques are better than others, and there is no guarantee that any particular cluster will be 100% successful at giving the user an accurate goal likelihood estimate. But clustering groups together entire swaths of similar information, so that by reading a single keyword, phrase, or sentence the user is able to evaluate the likelihood of dozens or even hundreds of documents meeting his or her information need.  With an ounce of effort, a pound of benefit is gained.  

However, with sparse, list-only techniques, an ounce of effort (looking at the next result) only gives you an ounce of benefit (that result).

From that standpoint, I don’t see how anyone can make the claim that exploratory search interfaces and algorithms require more effort, or increase the amount of choices available to the user.  Clustering methods are not perfect, but they do help the user both winnow, refine, and refactor the thousands of available results, so that those thousands do not have to be examined one at a time.  Even when not perfect, they reduce the burden of choice more than they expand it.