There is an interesting blogpost on the Official Google blog today, about seeing stars:
We’ve long believed that personalization makes search more relevant and fun. For nearly five years, we’ve been tailoring results with personalized search. Today we’re announcing a new feature in search that makes it easier for you to mark and rediscover your favorite web content — stars. With stars, you can simply click the star marker on any search result or map and the next time you perform a search, that item will appear in a special list right at the top of your results when relevant. That means if you star the official websites for your favorite football teams, you might see those results right at the top of your next search for [nfl].
So it sounds to me like this is a sort of bookmarking. What it not as obviously, however, is what this sentence means:“the next time you perform a search, that item will appear in a special list right at the top of your results when relevant”. Does that mean the next time you perform the same search (e.g. [nfl]) that starred item will appear at the top? Or is it more dynamic than that? I.e., if I happen to perform the search [new england patriots], and that same link that I’d previously starred after executing the [nfl] query happens to be ranked in the top k, will it again appear at the top of my list? (And if so, what is the cutoff/threshold for k?) Similarly, if Google’s ranking of my original [nfl] query changes, due to shifting PageRank calculations, changes in freshness, or any of the hundreds++ of other signals that go into the ranking algorithm, and my particular starred web page no longer appears in the top k because it is no longer relevant to the [nfl] query using the signal vector from the current state of index, will the starred item not appear? After all, Google says that the starred item will only appear if it is relevant, and if it is no longer relevant to the [nfl] query, as determined by Google’s relevance algorithm, then it won’t appear? Even though I had previously starred it with respect to that exact query?
The post continues:
In our testing, we learned that people really liked the idea of marking a website for future reference, but they didn’t like changing the order of Google’s organic search results. With stars, we’ve created a lightweight and flexible way for people to mark and rediscover web content.
Now I am thoroughly confused. People didn’t like changing the order of Google’s organic search results, but at the same time, they claim earlier in the post that “For nearly five years, we’ve been tailoring results with personalized search.” What does it mean to personalize search results, if not to change the order of Google’s organic search results? (Quoting the earlier post:
With the launch of Personalized Search, you can use that search history you’ve been building to get better results. You probably won’t notice much difference at first, but as your search history grows, your personalized results will gradually improve.
So if users didn’t like changing the order of the organic search results, does this mean that Google has turned off (or will be turning off) personalization completely for all signed-in users? Or does personalization co-exist with explicit starring/bookmarks? If so, how exactly does that work? Will Google change the order (personalize) your organic results using only the signals of query history and implicit relevance (i.e. clickthrough), but not the signal of explicit starring? That’s even more confusing…the amount of mental jazz involved is a bit overwhelming. Sure, the interface jazz is kept to a minimum, but at the expense of making the user’s mental model of what the search engine is actually doing for him or her even more muddled.
Perhaps the best way to sort out this confusion is to dive in headfirst and start playing around with the system, seeing what it actually does and when. But I personally have a difficult time generating the gumption to use a feature for which I have an unclear mental model, an unclear understanding of what it is trying to do for me, how it might change, when it might or might not magically appear. Especially when some of my actions affect the state of the system and others do not.
One thing I do like about this feature, however, is that it uses out-of-band displays to show different types of information. Rather than trying to mix global/non-personalized results, implicit personalized results, and starred results, it lets you know via a separate channel whether there is any information that you have previously starred. This is an IR design principle that I would like to see more of — separate goals in separate channels. Examples of different IR goals include navigation, re-finding, discovery, exploration, etc. Rather than trying to mix results from all of these goals into a single channel (a single ranked list) it is quite useful to separate each goal from the other. This new Google interface does that. What exactly the goal attached to that separate channel is, again, unclear. But the existence of a separate channel is an interesting and exciting approach, one that I hope to see more of.
I think Google was referring to their short-lived Search Wiki feature (http://googleblog.blogspot.com/2008/11/searchwiki-make-search-your-own.html) that allowed users to actually move results around.
I don’t think personalized results are going away. This new Star feature seems to simply be a way to highlight a search result for later, perhaps to help the users that repeat search queries for navigational purposes (“I got to that site last time I searched for XYZ – I’ll search for it again”), or “Information Re-Retrieval” (http://bit.ly/duW3iw).
Ah, when they said “people didn’t like changing the order of search results”, you think that they meant “people didn’t like explicitly having to drag search results up and down”? Ok, that sounds plausible. I just wish it were a little better written. Something like “people like when they can change Google’s organic results implicitly or automatically, via queries and clicks. But people do not like having to explicitly drag a result around.” The categorical, blanket statement of “people don’t like changing the order of search results” is what threw me off.
I’m still left scratching my head a bit, though. Like when they say that when you do “a search”, that starred result will appear again when relevant. Do they really mean when you do the same search that you did before, as you say? (i.e. I got to that site the last time I searched for XYZ, so I’ll type XYZ again”.) If so, why don’t they say, when you do “the search” instead of “a search”? It makes the whole feature seem a bit arbitrary and capricious. Like they’re saying “go ahead and star something, but don’t have any expectations about when or if it might appear again. Sometimes it will, sometimes it won’t. It’s anybody’s guess.”
Maybe I’m being pedantic. But I tend to prefer information retrieval systems that are a bit more transparent, that perhaps sacrifice a little bit of effectiveness for the sake of a little more conceptual consistency. It’s difficult to get excited about using software that does different things for you all the time, where you don’t really know what’s going to happen, why it happens, or how you can change it or stop it from happening. I value IR systems with more penetrability than that.
What does it mean “relevant”?
say, in the first 10 of the organic ranking? The first 100? The first 1000?
I’d prefer Google to be more explicit