Daniel Kahneman, famous psychology researcher and winner of the 2002 Nobel Prize in Economics, tells the story how many years ago, after completing his undergraduate studies in psychology, he served in the Israeli Army’s Psychology Unit. One of the unit’s tasks was to evaluate possible candidates for officer training. Teams of potential candidates underwent some mental and physical challenges completing an obstacle course. Kahneman and his colleagues monitored these exercises closely to evaluate each candidate’s role and contribution to the team’s effort. Afterwards they had to review and submit recommendations which candidates were likely to become good officers and those who would not. Kahneman and his colleagues thought their recommendations would be fairly accurate as they saw exactly the behaviour of each candidate under these stressful situations.
The disheartening aspect of their work, however, was the regular feedback they got back from the army trainers of these officer candidates. It became clear that their initial analyses were often inaccurate and were only slightly better than blind guesses. Why? Kahneman realised over time that they based their decisions on too little evidence – basically their recommendations were based on artificial situations over a short period of time, and not during real training and battlefield situations. Moreover, they ignored the role of randomness or accidental luck that bedevilled their recommendations and often caused their expectations of a “star” performer to be highly exaggerated when the candidate was confronted with real situations.