Ranked Lists and the Paradox of Choice

A thread in the comments section over on Daniel’s blog prompted me to think about the issue of ranked list results presentation in a new manner.  How do we decide how far to scroll down a ranked list of results, or when to switch and try a different search query or even a different search engine?  

When a search engine gives you nothing but line after line of results, it doesn’t matter if those results are segmented into pages of 10 or 100, or are infinitely scrolling.  You constantly are faced with the choice of whether or not you should go further in the results to find the information that you seek.  Having 2.7 million results come back as result to your query means that you are faced with 2.7 million choices.  You look at the 1st result and then have to decide whether you want to go on to the 2nd result.  You look at the 2nd result and have to decide whether to look at the 3rd, and so on.  

Conventional wisdom is that a sparse search engine interface, with just the results and not a lot of clutter (other than ads) goes a long way toward reducing the Paradox of Choice. It is said that users do not want to be over-burdened with too many exploratory interface options, query modification possibilities, or other peripheral data.  That only gets in the way of them assessing and choosing from the results.  However, let’s look at Paradox of Choice author Barry Schwartz’s strategy for navigating the overwhelming number of choices we are faced with:

  1. Figure out your goal
  2. Evaluate the importance of that goal
  3. Array the options
  4. Evaluate how likely each option is to meet your goal
  5. Pick the winning option
  6. Modify goal, rinse, and repeat

Now, when using a search engine, users have figured out steps 1 and 2 before typing in their query.  The search engine now comes back and presents the user with step 3, the array of options, the ranked list of results.  The big question is: How well does that array of options help the user evaluate how likely each option is to meet their goal (step 4)?  My answer is: Not very well. In an ideal world, the users goal(s) — relevant document(s) — would all be ranked at the top of the list.  But frequently they are not. When they are not, the user has to scan down the list (the “array of options”) and examine, individually, whether each option meets the goal.  This leads to the paradox of choice, an overwhelming array of options.  The only way to evaluate the next result is to read or skim that next result.

Let’s contrast sparse search engines with exploratory search engines.  Many modes of exploration are possible, but let’s examine “clustering” in light of this paradox of choice. Some clustering techniques are better than others, and there is no guarantee that any particular cluster will be 100% successful at giving the user an accurate goal likelihood estimate. But clustering groups together entire swaths of similar information, so that by reading a single keyword, phrase, or sentence the user is able to evaluate the likelihood of dozens or even hundreds of documents meeting his or her information need.  With an ounce of effort, a pound of benefit is gained.  

However, with sparse, list-only techniques, an ounce of effort (looking at the next result) only gives you an ounce of benefit (that result).

From that standpoint, I don’t see how anyone can make the claim that exploratory search interfaces and algorithms require more effort, or increase the amount of choices available to the user.  Clustering methods are not perfect, but they do help the user both winnow, refine, and refactor the thousands of available results, so that those thousands do not have to be examined one at a time.  Even when not perfect, they reduce the burden of choice more than they expand it.

This entry was posted in Information Retrieval Foundations. Bookmark the permalink.

6 Responses to Ranked Lists and the Paradox of Choice

  1. tam says:

    What if the context of the web pages indexed by the search engine are better defined as well as the context of the search engine’s user?

    Google has a mean to do that, through the google account; It is just not happening as most of the english speaking internet seams to be stranded in a US centered cultural context.

  2. jeremy says:

    Tam, are you talking about Google’s attempts at result personalization? That is happening in the U.S., too. But it still, imho, doesn’t overcome this general problem. Even if you get personalized results, and cut down from 2.7 million to 1.1 million more targeted results, you still have 1.1 million choices to make. The same dilemma exist.

    On top of that, if the personalization doesn’t work, and it give you the wrong 1.1 million result subset from the original, non-personalized 2.7 million result set, you have no way of “turning off” that personalization and getting back to a different subset. Google’s engine is not transparent, and not steerable/customizable. So you’re stuck.

  3. jeremy says:

    I talk a little more about search engines and the paradox of choice here: http://thenoisychannel.com/2009/04/22/sigir-09-accepted-papers/comment-page-1/#comment-2954

  4. Pingback: Information Retrieval Gupf » Universal Search is not Exploratory Search

  5. Pingback: Information Retrieval Gupf » Google Search Options and the Paradox of Choice

  6. Pingback: Information Retrieval Gupf » 200 Signals, Still Only One Route

Leave a Reply

Your email address will not be published. Required fields are marked *