I have a question that has been bothering me, kicking around in my head, for at least half a decade now. And I can’t seem to come to any solid conclusion on it. I suppose it can’t hurt to throw it out here onto the web, and see if one of my 3 readers has any thoughts on the matter.
A large amount of web search effort goes into statistics and probabilistic modeling of user queries and behaviors. There seems to be a generally-accepted, widely-held belief that the way to go about web search is to look at vast quantities of data, and modify search engine parameters based on that observable data. With enough users, and enough data, the search engine can be made better.
The approach seems very reasonable. Where I get concerned is with the iterative feedback loop:
- The designers of the search engine provide specific functionality
- The user utilizes that functionality
- The search engine collects data/logs about the use of that functionality
- The designers use machine learning to analyze those logs for evidence of success and failure, and finally
- The designers modify the search engine to provide improved functionality that better satisfies these user behaviors
- Goto 2.
What I do not see in this whole process is a way for the user to tell the engine designers that they need anything other than what the search engine already provides. From another perspective: Analysis of failure in steps 3&4 seems to only really tell you how well the user did when using the existing functionality. It does not tell you that the user had an information need, the satisfaction method of which falls outside of that existing functionality.
So how do (or how should) the developers of a search engine learn to recognize user needs and behaviors that fall outside of the what the users are able to say (through log analysis)? I am assuming that user-interviews are too small-scale to gather the necessary information, at web scale. And A/B testing only works when you’ve got a B that you’ve designed in reaction to a known behavior. But how do you develop that B, if you’ve never observed the behavior that will make use of B in the first place.. not because that behavior doesn’t exist, but because you cannot observe it until B exists? In short, when adding interactive features and capabilities to a search engine, how does one take proactive action in the development of those features, rather than only re-active reaction?