+1 is Explicit, but is not Relevance Feedback

A week or so ago, Google introduced it’s answer to the Facebook “Like”.  It is called “+1″.  Here is a quote from the official announcement:

The +1 button is shorthand for “this is pretty cool” or “you should check this out.”  Click +1 to publicly give something your stamp of approval. Your +1′s can help friends, contacts, and others on the web find the best stuff when they search.

A discussion then ensued on Twitter about whether Google had finally introduced explicit relevance feedback to its system.  For a long time, the user has been able to give implicit signals of preference to the search engine algorithm in the form of click-throughs.  And conventional wisdom has held that users are too lazy or to disinterested to interact with a web search engine in any explicit manner beyond typing 2.7 keywords into the one-line search box.  But now Google has introduced the +1.  Does this mean that explicit relevance feedback is finally here?

My answer is no.  And it is important to understand why.

First of all, +1 in its current state does not cause any changes in the search algorithm to happen at all.  From a Mashable interview:

[Interviewer] Will the number of +1s affect search rankings?
[Google] Prosser says no, but adds that it’s something Google is “very interested” in incorporating in some form at some point.”

Immediately it becomes clear that because +1 does not affect the search rankings, we cannot call it explicit feedback.  It might be explicit rather than implicit.  But it is not feedback.

However, my point runs deeper than that.  Let’s suppose at some point in the future the +1 actually did start to affect the overall ranking algorithm (the system as a whole), as Google wants to eventually make happen.  Even if it did that, it must be noted that +1 would still not affect your current search.  I.e. Google is pitching this is helping you find things that your friends have already found interesting.  Therefore, when your friends do a +1, they aren’t actually the ones receiving the benefit.  You are.  Their searches aren’t actually improving.  Yours are.  Again from the official announcement:

Sometimes it’s easier to find exactly what you’re looking for when someone you know already found it. Get recommendations for the things that interest you, right when you want them, in your search results.  The next time you’re trying to remember that bed and breakfast your buddy was raving about, or find a great charity to support, a +1 could help you out. Just make sure you’re signed in to your Google Account.  In order to +1 things, you first need a public Google profile. This helps people see who recommended that tasty recipe or great campsite. When you create a profile, it’s visible to anyone and connections with your email address can easily find it. Your +1’s are stored in a new tab on your Google profile. You can show your +1’s tab to the world, or keep it private and just use it to personally manage the ever-expanding record of things you love around the web.

One Twitter commenter gave the analogy to voting, and said that because users are explicitly allowed to put a label on something (“vote”) for it, that makes it explicit feedback.  I would like to expand on that voting analogy.  Yes, the +1 is explicit, like a vote.  But because we’ve already established that the vote doesn’t “feed back” to your own current information need, but instead affects other users of the system for their future information needs, it would be like a citizen of Country A casting a vote for the leader of Country B, and a citizen of Country B casting a vote for the leader of Country C, and so on.  Sure, both citizens are voting.  But each is voting for someone else’s leader.  So when the leader of country B does something, it affects the citizen of country B, rather than the citizen (of country A) who actually voted for that leader.  The analogy here is that leaders of countries are the search algorithms, and the vote is the +1.  Sure, you can vote all you want.  But if your vote doesn’t actually go toward your own leader, then your vote doesn’t actually affect what happens to you.

That is not what traditional IR literature means when it discusses relevance feedback.  In traditional feedback, an individual user marks a subset of documents as relevant and non-relevant, and then the system updates his or her ranked list results, immediately, so as to increase the recall (and sometimes also the precision) of documents not yet seen.  There is a reason it is called feedback: the loop is closed.  Just like when you hold a microphone too close to a speaker and start to get audio feedback.  That’s only possible because the output of one input gets fed immediately back into that same input.  Not into someone else’s input.

+1 offers no such closed loop.  Instead, a user explicitly expresses a preference for certain pieces of information.  That preference gets (or will someday soon get) used to update the search engine’s algorithm.  That updated algorithm will then alter/affect/change the results that some other user, with a perhaps related but also perhaps different information need, would have seen.  The search engine results are indeed improved as a result of the explicit +1.  But not for the original user, and not on that user’s existing information need.  And especially not immediately.  No results are reranked.  No new or changed query suggestions are given in the moment that the user casts his or her +1 vote.  In short, there is no feedback.

There is a word for user preference information that is fed into a search algorithm for the purpose of making that algorithm better: It is called “training data”.  Search engine companies use training data to improve their overall algorithm, and make sure that future users of a system get better results than past users did.  Currently, much of that training data in the web search world is implicit data: Click-throughs.  But just making that data explicit rather than implicit (through the use of +1′s) does not change the fundamental nature of the data; it’s still training data for the search algorithm, not feedback to and from the user.  With +1 training data, the user loop is open, not closed.  No algorithmic updates are flowing from the user back to the user for assistance with and improvements on the user’s current information need.

In summary, my argument boils down to this: +1, while explicit, does not offer a closed loop for a user’s open, unsatisfied information need.  Therefore, the purpose of +1 as Google envisions it is as training data, and not as relevance feedback.  Explicit training data, yes.  And it most likely carries a much stronger signal:noise ratio than implicit training data (click-throughs).  But it is (or will be once Google incorporates the signal) still naught but training data.

Why it matters

Ok, so +1 is explicit training rather than explicit feedback.  Who cares?  Aren’t we just arguing over terminology, splitting hairs over my favorite versus your favorite word?  No.  it is important to understand the difference between training data and relevance feedback because it helps understand what a search engine is doing for you (the user) and why.  It boils down to a question of user information need type: Do you have a precision-oriented information need (e.g. home page finding, recipe finding, address finding, etc.) or do you have a recall-oriented information need (are you seeking to understand everything that was published in the news media about the housing bubble, prior to its collapse, so as to paint a picture of who might have known more than they’re letting on)?  If you have a precision-oriented need, then you don’t need explicit relevance feedback.  You are more than likely fine with a system that trains itself on both the explicit and implicit actions of other users (either worldwide or even just your friends — still is training data) so as to make your search for [coffee shop chicago] a little better.  Remember:

The +1 button is shorthand for “this is pretty cool” or “you should check this out.”  Click +1 to publicly give something your stamp of approval. Your +1′s can help friends, contacts, and others on the web find the best stuff when they search.

But if your information need is deeper, and you don’t just need to fine a small handful of nearby coffee shops, but instead are attempting to make sense of a larger issue, then explicit data used for system parameter estimation under a social Learning to Rank regimen is not going to be sufficient.  You want to be able to instruct the system in the moment about the pieces of information that you are finding, and have it correct itself in that same moment, for that exact task.  You want there to be a closed loop between your actions and the machine’s actions, with each immediately influencing the other.  In short, you want relevance feedback.  Some of that relevance feedback might be implicit, some might be explicit.  But in both cases, that information is being “fed back” to the task, and then new information is returning to you, in real time.  This lets you do deeper on your current task than you could if you were simply waiting for a hundred other friends to click +1 on all the newspaper articles that you are looking for.  Understanding this difference is key to understanding how you as a user approach a system, and what you do with it.  It is not productive if the user thinks that the system is doing one thing, and it’s actually doing another.

Now, in no wise does this mean +1 is not useful.  In fact, it’s so useful (for particular types of information needs) that folks such as Barry Smyth have been doing similar social things for 6-7 years now (see Heystaks).  My goal was not to comment on the efficacy of +1, and in fact I think it is rather nice that a forward step toward more user interaction in the form of explicit judgments is finally becoming important to Google.  Rather, it was to seek clarity and clear delineation on what that efficacy is actually trying to do, and for whom, and by what mechanism, and why.  The explicit judgment is explicit training data.  It is not explicit relevance feedback.

5 comments to +1 is Explicit, but is not Relevance Feedback

  • Sam

    Thanks for this, Sharoda. This was my mid-day distraction+coffee fix. Besides your suggestion that explicit feedback on specific results could be helpful for mid-search refinement, the argument that most interested me:

    If you have a precision-oriented need, then you don’t need explicit relevance feedback. You are more than likely fine with a system that trains itself on both the explicit and implicit actions of other users (either worldwide or even just your friends — still is training data) so as to make your search for [coffee shop chicago] a little better.

    Unless I misunderstand what the explicit actions of others users/friends are, isn’t this just the point of +1? Producing a clearer signal for ranking searches like this? I would imagine the explicit recommendations of my friends for [coffee shop seattle], for instance, is much better than click-through at finding me a coffee shop I would actually like, given that my friends and I are hipster café snobs. I would think that, when my friends google [coffee shop seattle], they are browsing more than anything, sullying click-through’s signal-to-noise wrt my results.

    Perhaps this is something you had in mind.

  • jeremy

    Hi Sam. I’m Jeremy, not Sharoda :-) http://irgupf.com/about/

    But yes, you are absolutely correct. The point of +1 is to help my friends find things that I have already found. But what +1 does not do is help me find things that I have not yet found, to go deeper into a set of results by reshuffling around the results of my search, so as to more closely match the information found in the thing that I have +1′ed. The whole point of my post was to clearly distinguish these two use cases.

  • jeremy

    For example, given that you and your friends are not only coffee snobs, but hipster coffee snobs, you might want to use your +1 to find cafes that your friends don’t know about yet. Imagine that one of those coffee shops is in an out-of-the-way area. You give it your +1, and then the search engine says, “oh, you like what this page is about? Let me find more pages that exhibit similar traits”. So it goes and it finds another unknown cafe three blocks over from your +1′ed cafe. Lo and behold, none of your friends know about it yet; there are no friend +1′s attached to is. You go there, enjoy yourself immensely, and then you can be the uberhipster…because you might not want to +1 that link at all. You found it by using your own +1 (aka explicit) action on a different link, rather than relying on the +1 of your friends.

    (And note that at the end of the day, you didn’t actually +1 this second cafe. You only found it via your own +1 on the first cafe. So you can keep the secret to yourself a little bit longer if you want.)

    This is the second use case, the thing that +1 doesn’t actually do, that I wish it did. And it’s called “relevance feedback”.

  • [...] +1 is Explicit, but is not Relevance Feedback [...]

Leave a Reply

  

  

  

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>