Let me be clear about Aardvark: What that service does is help you comb through a network of people to find those individuals who have the highest likelihood of holding the answer to your information need.  Somebody has the answer; you just don’t know who it is.  So Aardvark helps you find that somebody.  The reason this is different from what I am talking about with explicit collaboration is that in this latter case, you already know who it is that you want to work with on resolving a shared information need.  You want to work with a relationship partner on finding an apartment.  You want to work with a business colleague on finding potential markets for a new product.  You want to work with some buddies on planning a road trip.  In all of these situations, your partner, your colleague, and your buddies don’t already have the answers that you seek.  But you do know that you want to work with them to find those answers because they have the same need that you do.  Your partner wants to live with you, your business colleague wants to work with you, and your buddies want to travel with you.  This is what explicitly collaborative information seeking is about, and it’s not the same thing as the “collaborative” category discussed in the panel.

Case in point: Take a look at the panel’s slides: http://www.slideshare.net/bmevans/introductory-slides.  Slide 9 outlines the two main social strategies: (1) Ask the network, and (2) embark alone.  This misses a third major, but as yet untapped, strategy: (3) embark together.

A good way to think about this is in terms of information seeking.  In both the (1) ask the network and (2) embark alone strategies, there is only a single user with an actual information need, a single person who is actively seeking information.  Using Aardvark, he or she is asking other people in the network if they are able to give an answer to satisfy that need.  But those other individuals do not actively share your information need.  They already either (1) have the information that you seek, and thus already have a satisfied information need, or (2) do not have the information you seek, but do not care, i.e. they do not share your information need (they aren’t going to move in with you, or go on that road trip with you).  When you ask the network, you are not actually involved in collaborative information seeking.  There is only a single seeker: You.  You are simply tapping into the network to find those people who already have the information you need.  It is still the single individual, not the network, that has the information need and that is actively engaged in the seeking process.

But embarking together with one or two other individuals who also lack information, i.e. engaging in explicitly collaborative information seeking, is a entirely different process.  In this case, there are at least two information seekers, two people who have a shared, as-yet-unsatisfied, information need.  Now, there are a number of different ways you can build systems and design interfaces to support these multiple seekers in their task.  I’ve written a lot about such systems on this blog and on the FXPAL blog, and will not go into it in further detail right now.  The point is simply that embarking together is an information seeking strategy that was not covered by any of the existing methods.  It is not the same as asking the network.  It is not the same as embarking alone.  It is a third process, a third strategy, and one that remains quite untapped in today’s marketplace.

Update: I have a final quick example.  On his live blog, Danny Sullivan paraphrases Max from Aardvark: “We want to do that across communication channels, so you can find partners to go bike with”.  That’s Aardvarkian social search: You want to find the people to go biking with.  Collaborative search is the next phase.  Once you’ve found the people that you explicitly know that you want to go biking with, how do you find out where you want to go?  You know about all the bike trails around your house.  Your new biking partner knows about all the trails near her house.  But neither of you know about the trails that exist halfway between both of your houses.  Ideally, you’d like to find one of those trails that is good for both of you, because neither of you is aware of them.  (And why should you have been? Before meeting your partner, you had no reason to venture away from your favorite nearby trails.) THAT is explicitly collaborative information seeking.  When both of you actively look for new bike trails, that is embarking together.

Collective (gathering advice from a crowd)

Friend Filtered (using your friends)

Collaborative (asking a friend — see also our The Rise Of Help Engines: Twitter & Aardvark article)

The version that was left out was the type of search in which you don’t just ask a friend for an answer (e.g. Twitter and Aardvark), but the type of search in which you actively engage with a specific person to work on on a jointly-shared information need.  For example, imagine a couple looking to rent an apartment.  It’s not like one person in the couple can ask the other one “where should we live?”  The point is that both people do not know.  And so you can imagine an information retrieval system that has, built in, the capability to be multi-searcher aware.  Both people can work on the same task at the same time.

This is not what Aardvark does.  This is not what Twitter does. This is also not friend-filtered; this is also not collective.  It is a fourth type, a distinction that seems to have been missed by the panel — search in which a small team of people actively work together, and the search system actively mediates between them, helping the group as a whole find information that no individual already knows, and that no individual would have easily found, had that person been working alone.   For more information on this oft-ignored area, please see our earlier series of posts.

The article focuses on YouTube’s “plans to rely more heavily on personalization and ties between users to refine recommendations” and “suggesting videos that users may want to watch based on what they have watched before, or on what others with similar tastes have enjoyed.”  What is striking about this is how little this has to do with search. As described in the article, what YouTube needs to do is entertain people who are bored but do not entirely know what they want. YouTube wants to get from users spending “15 minutes a day on the site” closer to the “five hours in front of the television.” This is entertainment, not search. Passive discovery, playlists of content, deep classification hierarchies, well maintained catalogs, and recommendations of what to watch next will play a part; keyword search likely will play a lesser role.

My feeling is that the dichotomy that is being drawn does not exhaustively cover the space.  I would characterize the space using the following two orthogonal dimensions: (1) Information Need Clarity and (2) User Engagement.  The first dimension (clarity) is related to the degree with which the user understands his or her own information need, i.e. has something specific in mind that he is looking for and/or understands what he needs to do to find it.  That need may either be well understood, or (to borrow Nick Belkin’s terminology) “anomalous”: The user doesn’t know what he or she doesn’t know.  The second dimension is related to the level at which the user applies himself to the information seeking process.  That level may be active or passive.

Greg points out two modes: “Active Understood” (typical navigational web search) and “Passive Anomalous” (entertainment/discovery/recommendation).  But I believe that there are more than these two modes.  A large, interesting design space opens up when one realizes that information seeking can be “Active Anomalous” and “Passive Understood“.

Exploratory Search is a good example of Active Anomalous seeking.  One doesn’t yet fully know or understand what it is that one is looking for, but at the same time one is willing to engage with an information system in order to discover what it is that he or she does not yet know.  And the system itself is designed not necessarily toward trying to answer a well understood need, but toward helping the user map out and better comprehend a space.

Collaborative Information Seeking (see here and here and here) is a good example of where an need may be well understood, but a user does not necessarily have to actively express every last query detail in order to get more information on a topic.  Why not?  Because when User #1 is explicitly collaborating with User #2, an algorithmic mediation engine can push some of User #2’s activity on to User #1 without requiring User #1 to make additional effort.  Note that I am not implying that every aspect of collaborative information seeking is passive; quite the contrary, as it requires at least one co-collaborator to be active.  I am only pointing out that it is a domain in which it becomes possible for a user to passively obtain specific information on a well understood need.

There is a lot discussion in the Information Retrieval Community on the similarities and differences between Search and Recommendation.  A fruitful tension opens up as one travels back and forth along the diagonal from Active Understood to Passive Anomalous; the two approaches often end up complementing each other.  Where I see much less discussion is on the tension that opens up along the other diagonal, between Passive Understood and Active Anomalous.  When Exploratory Search meets Collaborative Information Seeking, it yields Collaborative Exploratory Search and a whole host of interesting possibilities.  Over the coming year I will be blogging more about the tension along this alternative diagonal (both here at on the FXPAL blog) and what it means for the Information Retrieval systems I and others are designing.  Happy 2010!

Via Daniel Tunkelang’s recent post, I think that Government information might be a perfect domain in which to ask (and answer) these sorts of questions.  The U.S. Open Government Initiative has as its goal the release of loads of raw government data for use by any individual or organization.  How are people going to use this data?  What types of questions will they ask?  What types of questions could they ask, if given the proper tools (i.e. what might they not know that they want to ask, until it becomes possible?)

Two types of information retrieval might be perfect for this domain: Exploratory Search and (Explicitly) Collaborative Search.  In exploratory search, the goal of your information seeking is to learn, discover, compare, contrast, etc. In explicitly collaborative search, your goal is to do something similar, but with another set of like-minded partners working with you on the same task/topic.  Each partner may have different expertise; one may be an expert in energy policy, another might understand trade and commerce, and another might have experience with the inner workings of Congress and understand how it works on a practical level.  If you put all these people together right now, the only way they can work together on a shared task is to search separately and then email each other their results.  What if, however, you could design a system that not only mediated between them on an interface level (immediate notification of marked documents and passages, shared highlighting of seen documents, etc.) but mediated between them on an algorithmic level as well?  Algorithmic mediation of the collaborative process would mean that the retrieval system itself has a hand in both combining and partitioning the inputs and actions of the search team members, as necessary.  They might then be able to find important, valuable information that none of the searchers, had they been working alone, could have.

It seems like an interesting domain, and one with real, potentially quite important consequences and societal implications.  It will be interesting to watch as this develops.

This is hardly the first time that the so-called “wisdom of the crowds” has been called into question. The term, which implies that a diverse collection of individuals makes more accurate decisions and predications than individuals or even experts, has been used in the past to describe how everything from Wikipedia to user-generated news sites like Digg.com offer better services than anything created by a smaller group could do.

Of course, we now know that simply isn’t true. For one thing, Wikipedia isn’t written and edited by the “crowd” at all. In fact, 1% of Wikipedia users are responsible for half of the site’s edits. Even Wikipedia’s founder, Jimmy Wales, has been quoted as saying that the site is really written by a community, “a dedicated group of a few hundred volunteers.”

<snip>

Still, there [has] yet to be a perfect solution to the problem. Perhaps it’s time we give up the idea that the “wisdom of the crowds” was ever a driving force behind any socialized, user-generated anything and realize that, just like in life, there will always be active participants as well as the passive passerbys.

I have never quite liked the notion of “wisdom of crowds”, and the hype behind it even less, so I”m glad to see signs that the hype cycle is finally starting to wind down.  However, by having to confront exactly what it was that I didn’t like about the notion, I was intellectually forced to propose an alternative: Explicit Collaboration in Search.  As I wrote half a year ago:

[In early 2006], I was having a visceral reaction against all the hype surrounding “collective intelligence” and “wisdom of the crowds” as a primary basis for doing information retrieval. I was not interested in collaboration as massive data crunching on top an anonymous crowd, nor even on top of a known crowd of friends.  I wasn’t interested in crowds of any kind. Rather, I thought (and still do) there is a lot more value left to be extracted from content-based search methods.

However, where these thoughts soon led me was to a notion of collaboration in search quite different from “wisdom of crowds” methods. With Maribeth Back and Gene Golovchinsky, I envisioned collaboration as a sort of “musical jam session”, where a small set of common-goal searchers got together and “played” their search “tunes” together over a content-based retrieval back end.  The purpose of this jamming wasn’t to repeat each others’ notes (”people who play this note also play that note”) but to play melodies and baselines that were different, but that worked together toward the larger goal of creating a full “song”, a commonly constructed set of relevant information.  To date, this notion of collaboration in search has proven, and continues to be, quite fruitful.  There are all sort of research “melodies” left to be played, all sorts of songs left to be sung, by all sorts of researchers.  I continue to be excited about this notion of “search jamming”, and look forward to the solutions that the community will continue to invent.

I have not spend too much time blogging about this ongoing collaborative information seeking research on IRGupf.  Most of what (Gene and) I have written on that topic appears on the FXPAL Blog (see [1][2][3][4][5][6][7]).  Still, I found the aforementioned “There is no Crowd” post interesting enough to warrant a brief mention here.  I find Perez’s (the author’s) conclusion compelling: Perhaps it’s time we give up the idea that the “wisdom of the crowds” was ever a driving force behind any socialized, user-generated anything.  If you want to put the “social” into your information seeking systems and algorithms, then it makes sense to really make it social, i.e. there should be explicit interaction with a known person, rather than implicit, passive behavioral aggregation.

[Note that explicit collaboration in information seeking is only one of many possible alternatives to the "wisdom of the crowd".  Xavier proposes a different solution in a paper published at SIGIR 2009.]

Perhaps I am stuck back in a view of Information Retrieval that is 10-15 years old, but I tend to find my views heavily shaped or influenced by things like the following bit from Marti Hearst’s chapter in Modern Information Retrieval:

An important part of the information access process is query reformulation, and a proven effective technique for query reformulation is relevance feedback. In its original form, relevance feedback refers to an interaction cycle in which the user selects a small set of documents that appear to be relevant to the query, and the system then uses features derived from these selected relevant documents to revise the original query. This revised query is then executed and a new set of documents is returned. Documents from the original set can appear in the new results list, although they are likely to appear in a different rank order. Relevance feedback in its original form has been shown to be an effective mechanism for improving retrieval results in a variety of studies and settings [salton90a][harman92c][buckley94b]. In recent years the scope of ideas that can be classified under this term has widened greatly.

Given that explicit relevance feedback works, why is it essentially non-existent on the web?  A bird in the hand (an explicit relevance judgment) is worth two in the bush (two implied or inferred relevance judgments).  Danny Sullivan (above) argues against explicit relevance interaction by pointing to a bunch of complicated, non-intuitive interfaces and laughing;  nothing beats a simple interface, he says.  But clicking thumbs up or down on information that you do or do not find relevant is not complicated.  Pandora uses it all the time in the music information retrieval domain.  By itself, that shows that users are not too lazy to use such a tool.  So why do we not have it in the web information retrieval domain?  Hearst continues:

Relevance feedback introduces important design choices, including which operations should be performed automatically by the system and which should be user initiated and controlled. Bates discusses this issue in detail [bates90b], asserting that despite the emphasis in modern systems to try to automate the entire process, an intermediate approach in which the system helps automate search at a strategic level is preferable. Bates suggests an analogy of an automatic camera versus one with adjustable lenses and shutter speeds. On many occasions, a quick, easy method that requires little training or thought is appropriate. At other times the user needs more control over the operation of the machinery, while still not wanting to know about the low level details of its operation.

I see the same thing today that Bates saw twenty years ago: Modern systems are trying to automate the entire process.  You often don’t even know that you’re doing relevance feedback. Full automation may be helpful in certain specific scenarios, in that it might save me a click or two, it also is not transparent enough for me to correct when it goes wrong.  Bates concludes, and I agree, that an intermediate, strategic approach is better.  The system should handle the low-level details, but not remove control of the strategic process from the hands of the user.

One of the reasons I am interested in this topic is related to my work in collaborative information seeking (algorithmic mediation of the explicitly collaborative process).  One of the challenges that we face in designing not only interfaces, but the underlying strategic roles that will be adopted by collaborating searchers, is how much of the process to make explicit and how much to leave implicit.  Certainly we do not want to require that collaborators do everything themselves.  This is the current state-of-the-art, in which the only way to collaborate on the web is to email links back and forth.  This is clearly a non-optimal solution.  But on the other hand, it may also cause the collaborators a good deal of frustration if the system attempts to do everything for them.  There has to be a balance.

In April of 2006 I had the good fortune to attend a Dagstuhl Seminar on Content-Based Multimedia Information Retrieval (I am toward the upper left corner of the seminar group photo).  Ramesh Jain has a good writeup of Dagstuhl Seminars, what they are and how they work.  In the abstract of my Seminar presentation I wrote:

The classic problem of ad hoc information retrieval involves a user with an information need, a representation or expression of that information need (the query), and a system or retrieval engine that compares the query against a collection of items in order to return the most relevant items to the user information need. Despite numerous and obvious exceptions, in general text information retrieval has a fairly high correlation between the syntax of a query as expressed in language and the semantics of the information need. Textual similarity is highly correlated with relevance. On the other hand, in content-based multimedia retrieval (images, video, music, 3d models), objects encompass multitudinous semantics in many different dimensions. In music for example there are properties of pitch, tempo, rhythm, timbre, singer characteristics, genre, instrumentation, year of production, and so on. The correlation between similarity and relevance is much lower. Two music pieces might be similar because they both use similar instruments, timbres, tempos and singers, but they are not necessary both relevant to my information need if I am looking for waltzes, and one piece is in 3/4 and the other in 4/4.

The current popular solution to this problem, characterized by buzzwords such as “collective intelligence”, “wisdom of crowds” and “Web 2.0″, is to bypass content altogether. By instead aggregating the media interactions (playlists, tags, click behavior, etc.) of massive numbers of people, the collective intelligence approach hopes to be able to determine relevance directly, without the need for content-based methods. If people are not only the ultimate consumers, but also the ultimate producers of relevance, why waste any effort on a problem as difficult as content-based retrieval? In our presentation we reject this notion of complete reliance on collective intelligence methods and argue that content-based methods are necessary. Aggregate crowd relevance information may be able to tell us what should be retrieved, but it still will not tell us why something was retrieved. For that, we still need to rely on the explanatory power of content. Therefore, we propose the “cognitive disclosure” paradigm, in which semantic representations are chosen a priori by designers of a content retrieval system, i.e. content-features necessary to call a piece of music a “waltz”, or to call an image a “landscape”. These semantic categories are then revealed to users at retrieval time, to allow them more intelligent selection of the types of information that is relevant to them. This problem is still very difficult and there are no easy solutions. However, our purpose is simply to explain why “wisdom of crowds” approaches will inevitably fall short, and content-based methods are still going to be necessary.

At the time, I was having a visceral reaction against all the hype surrounding “collective intelligence” and “wisdom of the crowds” as a primary basis for doing information retrieval. I was not interested in collaboration as massive data crunching on top an anonymous crowd, nor even on top of a known crowd of friends.  I wasn’t interested in crowds of any kind. Rather, I thought (and still do) there is a lot more value left to be extracted from content-based search methods.

However, where these thoughts soon led me was to a notion of collaboration in search quite different from “wisdom of crowds” methods. With Maribeth Back and Gene Golovchinsky, I envisioned collaboration as a sort of “musical jam session”, where a small set of common-goal searchers got together and “played” their search “tunes” together over a content-based retrieval back end.  The purpose of this jamming wasn’t to repeat each others’ notes (”people who play this note also play that note”) but to play melodies and baselines that were different, but that worked together toward the larger goal of creating a full “song”, a commonly constructed set of relevant information.  To date, this notion of collaboration in search has proven, and continues to be, quite fruitful.  There are all sort of research “melodies” left to be played, all sorts of songs left to be sung, by all sorts of researchers.  I continue to be excited about this notion of “search jamming”, and look forward to the solutions that the community will continue to invent.

• Communicating about Collaboration
• Communicating about Collaboration: Intent
• Communicating about Collaboration: Synchronization
• Social Search
• Social Search Redux

I will post another recap once we finish the series.

http://palblog.fxpal.com/?p=350#comments

When we view collaboration from the IR perspective, it becomes clear that collaboration does not merely refer to a decision to work together.  Rather, it refers to a decision to work together on an information need.  Users who have actively and consciously decided to work with each other on a specific information need are explicitly(intentionally) engaged in collaborative information seeking.  A user whose information seeking activities are influenced (for the better, of course) by others, but without the user’s active involvement, is implicitly or non-intentionally collaborating with others.