It’s when you start to get to the long tail of information needs, the variety of users that want to use the information on the web for something other than looking up a home page. In those learning oriented, knowledge skimming- and synthesis-oriented, analysis-oriented types of search tasks, simply mining the logs may not give you the information you seek.

For example, one person (let’s say “User A”) might run 6 queries in a row, constantly clicking on the back button and trying another one, because they can’t find the “correct” query to get them to the known-item piece of information that they are seeking. That would be a failed search, using existing tools. However, another person (let’s say “User B”) might run 6 queries in a row, quickly skimming the summaries for the top few results to each query and then constantly clicking on the back button and trying another query, because user B is attempting to probe the conceptual boundaries of a topic area. That would be a successful search, but one in which the tool was used quite awkwardly, ie. the search engine really didn’t provide the right tool, so the user had to simulate a boundary-probing search tool, manually. Like using the prongs of a hammer to grab hold of and screw in a screw.. you can do it, but it ain’t pretty.

However, from the perspective of the log analysis, both users A and B look exactly the same. Both try 6 different queries, in rapid succession, and don’t click anything after running each query. But user A considered his/her search a failure, and user B considered his/her search a success.

So I can’t say for sure, because I am not at one of these companies. But it seems to me that it would be very difficult to tell these two types of users apart, to know for sure from the log analysis which if both users were of type A or type B or one of each. And even if you did user interviews, and discovered that user B exists, you still have to rely on log mining to know exactly how many people of type user B there are. And in that case you’re back to the old problem of knowing how to tell user A and user B apart in the logs, when their behaviors look exactly the same.

But, I agree with you that they’re not sufficient for understanding process failures. If you want to understand these with automatic machine learning methods, I’d guess that a much broader view of a “log” is need. For example, a capture all the actions associated with an information seeking task not just in the search engine but in the word processor, on your blog, on twitter, in your code editor, on your phone, and even personal interactions away from the computer. Its hard to imagine collecting this data at all, much less on a reasonably a large scale.

Everyone talks about how “user driven” they are, and so there must be some sort of process for understanding and measuring situations wherein the process itself is not working. That’s what I’ve been seeking to understand for many years now.

So is your Oxo example the answer? There really is no process for detecting “out of band” processes, and it is up to the search engine designers to come up with that stroke of creativity?

There’s an art to identifying those types of failures — as you’ve said, users may not know how to articulate what the failure may be, or may not even perceive it as a failure since they’re using the provided tool as it was intended to be used and as it is used by everyone else. Most people probably don’t even think there might be a better way, they just work around the provided constraints.

This is the case with all design, not just information seeking systems. Check out this video of the president of Oxo talking about their design process, especially the bit about their measuring cup. No one perceived the tool as lacking until the whole process was evaluated. Even then, it took a real stroke of creativity to break out of the established process.

But it’s quite a leap to go from that user statement, to the notion that some sort of exploratory interface and supporting algorithm is needed, never mind even what interface and algorithm would be best for satisfying this type of “off the beaten path” cafe search.

But yeah, as Daniel says, it could at least be a starting point to get engineers to think about new problems.