A Bird in the Hand…

As a researcher, I have more questions than answers.  And one of the questions that I have is in regards to the widely-accepted maxim that users are too lazy to give explicit relevance feedback to the search engine.  See Danny Sullivan’s take, here.

Perhaps I am stuck back in a view of Information Retrieval that is 10-15 years old, but I tend to find my views heavily shaped or influenced by things like the following bit from Marti Hearst’s chapter in Modern Information Retrieval:

An important part of the information access process is query reformulation, and a proven effective technique for query reformulation is relevance feedback. In its original form, relevance feedback refers to an interaction cycle in which the user selects a small set of documents that appear to be relevant to the query, and the system then uses features derived from these selected relevant documents to revise the original query. This revised query is then executed and a new set of documents is returned. Documents from the original set can appear in the new results list, although they are likely to appear in a different rank order. Relevance feedback in its original form has been shown to be an effective mechanism for improving retrieval results in a variety of studies and settings [salton90a][harman92c][buckley94b]. In recent years the scope of ideas that can be classified under this term has widened greatly.

Given that explicit relevance feedback works, why is it essentially non-existent on the web?  A bird in the hand (an explicit relevance judgment) is worth two in the bush (two implied or inferred relevance judgments).  Danny Sullivan (above) argues against explicit relevance interaction by pointing to a bunch of complicated, non-intuitive interfaces and laughing;  nothing beats a simple interface, he says.  But clicking thumbs up or down on information that you do or do not find relevant is not complicated.  Pandora uses it all the time in the music information retrieval domain.  By itself, that shows that users are not too lazy to use such a tool.  So why do we not have it in the web information retrieval domain?  Hearst continues:

Relevance feedback introduces important design choices, including which operations should be performed automatically by the system and which should be user initiated and controlled. Bates discusses this issue in detail [bates90b], asserting that despite the emphasis in modern systems to try to automate the entire process, an intermediate approach in which the system helps automate search at a strategic level is preferable. Bates suggests an analogy of an automatic camera versus one with adjustable lenses and shutter speeds. On many occasions, a quick, easy method that requires little training or thought is appropriate. At other times the user needs more control over the operation of the machinery, while still not wanting to know about the low level details of its operation.

I see the same thing today that Bates saw twenty years ago: Modern systems are trying to automate the entire process.  You often don’t even know that you’re doing relevance feedback. Full automation may be helpful in certain specific scenarios, in that it might save me a click or two, it also is not transparent enough for me to correct when it goes wrong.  Bates concludes, and I agree, that an intermediate, strategic approach is better.  The system should handle the low-level details, but not remove control of the strategic process from the hands of the user.

One of the reasons I am interested in this topic is related to my work in collaborative information seeking (algorithmic mediation of the explicitly collaborative process).  One of the challenges that we face in designing not only interfaces, but the underlying strategic roles that will be adopted by collaborating searchers, is how much of the process to make explicit and how much to leave implicit.  Certainly we do not want to require that collaborators do everything themselves.  This is the current state-of-the-art, in which the only way to collaborate on the web is to email links back and forth.  This is clearly a non-optimal solution.  But on the other hand, it may also cause the collaborators a good deal of frustration if the system attempts to do everything for them.  There has to be a balance.