Key Insight
Well, I’m still obsessed with the ” ‘hot hand fallacy’ fallacy.” Are you? As discussed previously, the classic “‘hot hand’ fallacy” studies purported to show that people are deluded when they perceive that basketball players and other athletes enjoy temporary “hot streaks” during which they display an above-average level of proficiency. The premise of the studies was that ordinary ... Read more
Well, I’m still obsessed with the ” ‘hot hand fallacy’ fallacy.” Are you?
As discussed previously , the classic “‘hot hand’ fallacy” studies purported to show that people are deluded when they perceive that basketball players and other athletes enjoy temporary “hot streaks” during which they display an above-average level of proficiency.
The premise of the studies was that ordinary people are prone to detect patterns and thus to confuse chance sequences of events (e.g., a consecutive string of successful dice rolls in craps) as evidence of some non-random process (e.g., a “hot streak,” in which a craps player can be expected to defy the odds for a specified period of time).
For sure, people are disposed to see signal in noise.
But the question is whether that cognitive bias truly accounts for the perception that athletes are on a “hot streak.”
The answer, according to an amazing paper by Joshua Miller & Adam Sanjurjo, is no .
Or in any case, they show that the purported proof of the “hot hand fallacy” itself reflects an alluring but false intuition about the the conditional independence of binary random events.
The “test” the “hot hand fallacy” researchers applied to determine whether a string of successes indicate a genuine “hot hand”–as opposed to the illusion associated with our over-active pattern-detection imaginations–was to examine how likely basketball players were to hit shots after some specified string of “hits” than they were to hit shots after an equivalent string of misses.
If the success rates for shots following strings of “hits” was not “significantly” different from the success rates for shots following strings of “failures,” then one could infer that the probability of hitting a shot after either a string of hits or misses was not significantly different from the probability of hitting a shot regardless of the outcome of previous shots. Strings of successful shots being no longer than what we should expect by chance in a random binary process, the “hot hand” could be dismissed as product of our vulnerability to see patterns where they ain’t, the researchers famously concluded.
This analytic strategy itself reflects a cognitive bias– an understanding about the relationship of independent events that is intuitively appealing but in fact incorrect .
Basically, the mistake — which for sure should now be called the ” ‘hot hand fallacy’ fallacy” — is to treat the conditional probability of success following a string of successes in a past sequence of outcomes as if it were the same as the conditional probability of success following a string of successes in a future or ongoing sequence . In the latter situation, the occurrence of independent events generated by a random process is (by definition) unconstrained by the past. But in the former situation — where one is examining a past sequence of such events — that’s not so.
In the completed past sequence, there is a fixed number of each outcome. If we are talking about successful shots by a basketball player, then in a season’s worth of shots, he or she will have made a specifiable number of “hits” and “misses.”
Accordingly, if we examine the sequence of shots after the fact , the probability the next shot in the sequence will be a “hit” will be lower immediately following a specified number of “hits” for the simple reason that the proportion of “hits” in the remainder of the sequence will necessarily be lower than it it was before the previous successful shot or shots.
By the same token, if we observe a string of “misses,” the proportion of “misses” in the remainder will be lower than it had been before the first shot in the string. As a result, following a string of “misses,” we can deduce that the probability has now gone up that the next shot in the sequence will turn out to have been a “hit.”
Thus, it is wrong to expect that, on average, when we examine a past sequence of random binary outcomes, P(success|specified string of successes) will be equal to P(success|specified string of failures). Instead, in that that situation, we should expect P(success|specified string of successes) to be less than P(success|specified string of failures).
That means the original finding of the “hot hand fallacy” researchers that P(success|specified string of successes) = P(success|specified string of failures) in their samples of basketball player performances wasn’t evidence that the “hot hand” perception is an illusion. If P(success|specified string of successes) = P(success|specified string of failures) within an adequate sample of sequences , then we are observing a higher success rate following a string of successes than we would expect to see by chance .
In other words, the data reported by the original “hot hand fallacy” studies supported the inference that there was a hot-hand effect after all!
So goes M&S’s extremely compelling proof, which I discussed in a previous blog . The M&S paper was featured in Andrew Gelman’s Statistical Modeling, Causal Inference blog, where the comment thread quickly frayed and broke, resulting in a state of total mayhem and bedlam!
How did the “hot hand fallacy” researchers make this error? Why did it go undetected for 30 yrs, during which the studies they did have been celebrated as classics in the study of “bounded rationality”? Why do so many smart people find it so hard now to accept that those studies themselves rest on a mistaken understanding of the logical properties of random processes?
The answer I’d give for all of these questions is the priority of affective perception to logical inference.