Long Term versus Evolutionary Thinking (Part 1 of 2)

Last week I attended the O’Reilly eTech conference.  The first night, Tim O’Reilly gave his annual Radar talk, in which he surveys the technology landscape and comments on upcoming and interesting trends. I have heard this Radar talk for years, via the IT Conversations podcast network, but this was the first time I’d seen it in person. O’Reilly always has challenging, thought-provoking things to say, and this year was no different.  He did, however, mention two emerging trends or patterns that I thought contradicted each other, and I want to specifically comment on those.

In particular, O’Reilly pointed out that both the need for both long term thinking and evolutionary thinking with respect to the technologies that we develop.  Long term thinking is necessary because the problems we face are important enough that we cannot afford to be short-sighted.  Evolutionary thinking is necessary because the solutions to these problems are too complex to engineer in one fell swoop.  I agree with each of these points, taken individually.  We do need to think long term.  And we do need to iterate our solutions.  The problem comes in the fact that each of these approaches can often contradict the other.

As this is an information retrieval blog, I will use an information retrieval example to illustrate this contradiction.  However, before I get into my conjectures I need to explain the analogy that I will be using.  In the machine learning community, there is the idea of finding the maximum or optimal value of a function using a method called gradient ascent, which is a series of small, evolutionary steps.  For example:

This illustration represents the space of possible solutions to a problem along the x-axis, the quality of that solution along the y-axis.  Long term thinking means that we eventually want to end up at point A on the curve, the highest and best possible solution.  Currently, however, the solutions that we have are located at points B’ or B”.  They are not optimal.  This is where evolutionary thinking arrives.  Evolution moves in small increments, rather than in large leaps, toward locally-better solutions.  For example, a person standing at current solution point B” surveys the landscape, sees that a step to the left lead down the hill and a step to the right leads up the hill, and then steps up the hill.  With enough evolutionary iteration, each small upward step then leads to the long term, global optimum.  

So in the above example, long term and evolutionary thinking are not incompatible.  Long term goals coupled with evolutionary steps yield the best solution.  However, the reason why they are not compatible is because the solution space is globally convex; you cannot take a wrong step because all upward-facing paths lead to the global maximum.  In real life, especially when working on problems and challenges that really matter and make a difference in people’s lives, the solution space is not so simple.  It may look more like this:

This more realistic solution space still has a global maximum.  It also has a few local maximums, or peaks that are not as optimal as the global solution.  This is where the problem of evolutionary thinking comes in.  Our long-term goal is still to arrive at solution A, the global maximum.  However, if we are standing at point B’ or B”, there is no longer a single, ascending pathway that leads to A.  Suppose we are at point B”.  Evolutionary thinking walks us up the hill, until we get to local maximum point A”.  At this point, evolutionary thinking looks around, sees that both a step to the left and to the right leads to less optimal solutions.  Both directions lead down the hill.  Evolutionary thinking then quits, because it appears that the best possible solution has been reached.  Of course, this is not the case. What is needed in this case is a temporary suspension of evolutionary thinking.  A non-evolutionary, leap-oriented research lab needs to step in, to creatively engineer a new solution, one that takes the user from point A” to the other size of the point C” valley:

This research leap, rather than evolutionary step, is able to offer a solution that lands on the other side of the C” valley.  It does not matter that this new solution might initially be of lower quality that the solution at point A”.  Once this new solution is known, evolutionary thinking can step back in, and easily iterate its way up to the global maximum A.  

So in summary, long term thinking and evolutionary thinking are both needed.  However, if the solution landscape is complex, as it often is with problems that matter, these two modes of thinking are not enough on their own, and will lead to less-than-optimal solutions. The long-term desire to reach a high-point will contradict the evolutionary, short-term desire for small, incremental steps.   What is needed at time, instead of evolutionary thinking, is leap-oriented, big step thinking.  And we also need a willingness or patience — the long term thinking — to not immediately discard a leap-created solution simply because it currently appears to be less optimal than our previous local maximum.  That leap might just have started us on the pathway to and even larger global maximum.

A side note: If you prefer, you can think about this in terms of water flowing down the hill toward better (deeper) solutions, rather than hikers hiking up the hill, toward better (higher) solution.  Mathematically, they’re equivalent; the global and local minima problem still exists.  Simply turn the curve upside down.  A solution starting at point B” and flowing downhill will encounter the A” pothole, and get trapped or stuck.  Some leap based work is then required to push the water over the C” hill, so that it can flow again toward the global minimum A.

So how does this relate to information retrieval?  Now that I’ve given this background, my next post will get into that topic.

Continue to Part 2.