follow CCP

Recent blog entries
popular papers

Science Curiosity and Political Information Processing

What Is the "Science of Science Communication"?

Climate-Science Communication and the Measurement Problem

Ideology, Motivated Cognition, and Cognitive Reflection: An Experimental Study

'Ideology' or 'Situation Sense'? An Experimental Investigation of Motivated Reasoning and Professional Judgment

A Risky Science Communication Environment for Vaccines

Motivated Numeracy and Enlightened Self-Government

Making Climate Science Communication Evidence-based—All the Way Down 

Neutral Principles, Motivated Cognition, and Some Problems for Constitutional Law 

Cultural Cognition of Scientific Consensus
 

The Tragedy of the Risk-Perception Commons: Science Literacy and Climate Change

"They Saw a Protest": Cognitive Illiberalism and the Speech-Conduct Distinction 

Geoengineering and the Science Communication Environment: a Cross-Cultural Experiment

Fixing the Communications Failure

Why We Are Poles Apart on Climate Change

The Cognitively Illiberal State 

Who Fears the HPV Vaccine, Who Doesn't, and Why? An Experimental Study

Cultural Cognition of the Risks and Benefits of Nanotechnology

Whose Eyes Are You Going to Believe? An Empirical Examination of Scott v. Harris

Cultural Cognition and Public Policy

Culture, Cognition, and Consent: Who Perceives What, and Why, in "Acquaintance Rape" Cases

Culture and Identity-Protective Cognition: Explaining the White Male Effect

Fear of Democracy: A Cultural Evaluation of Sunstein on Risk

Cultural Cognition as a Conception of the Cultural Theory of Risk

« #scicomm question: what communicates essential information more effectively--unfilled overlapping pdd's or filled/transparency ones? | Main | Weekend update--it's baaaaaack! Our paper explaining why N=55, 95% liberal, is not a valid sample for "replicating" our "motivated numeracy" study »
Monday
Oct302017

More evidence of AOT's failure to counteract politically motivated reasoning 

Notice 2 things about this Figure:

1st, Stata 15 can now do transparencies!

2nd, this is even more evidence that “Actively open-minded thinking,” as commonly measured, furnishes no meaningful protection against politically motivated reasoning.

The results here are based on the same experimental design featured in the CCP Motivated Numeracy paper (Kahan, Peters et al. 2017). Subjects were asked what inference was supported by data presented in a 2x2 contingency table.  In one condition, the data were described as results of an experiment to test a new skin-rash cream.  In another, the data were described as results of an experiment to determine whether banning the carrying of concealed handguns in public increased or decreased crime.

In Motivated Numeracy, we found that individuals of opposing ideological orientations were substantially more likely to get the correct answer in the gun-control version if the data, properly interpreted, supported (or “affirmed”) the position associated with their ideology; when the data, properly interpreted did not not support their ideological group's position, individuals were more likely to select the wrong answer.

What’s more, the effect was stronger among the subjects of the highest degree of Numeracy, an aptitude to reason well with quantitative information.

The data here are pretty similar to those in Motivated Numeracy, except now it's “Actively Open-minded Thinking” (AOT) that is being shown to interact with ideology.  On the effectiveness of the new skin cream, individuals who score highest on a standard measure of AOT do better than those who score low, regardless of their political outlooks.

In the “gun control” condition, those who score highest on AOT do only slightly better on the version of the problem that presents ideologically congenial data. 

In the version that presents threatening or ideologically uncongenial evidence, however, those who score highest on AOT do no better than those who score the lowest.

This is not what you’d expect.

AOT is supposed to counteract ideologically motivated reasoning along with kindred forms of “my side bias” (e.g., Stanovich 2013; Baron  1995). Accordingly, in the "identity threatened" condition, one would expect those highest in AOT to do just as well as their high-scoring counterparts in the "identity affirmed" condition. One would expect, too, that the performance of those high in AOT would not show a level of degradation (-30%, +/- 14%) comparable to the degradation in performance shown by low scoring AOT subjects (-23%, +/-10%).

But it didn’t work this way here.

It also didn’t work that way in a study that Jon Corbin and I did last year, in which we showed that those highest in AOT, far from converging, were even more politically polarized on the danger posed by climate change (Kahan & Corbin 2016).

What to make of this?

Well, again, one possibility is that the version of AOT we are using simply is not valid.  I don’t buy that, really, because the measure has been validated in various settings (e.g., Baron et al. 2015).

The other possibility, which I think is more plausible, is that AOT--like Numeracy  (Kahan, Peters et al. 2017), Cognitive Reflection (Kahan 2013), and Ordinary Science Intelligence (Kahan 2016)—magnifies identity-protective reasoning where certain policy-relevant facts have become entangled with group-based identities (Kahan 2015).  Basically, where that’s the case, people use their critical reasoning proficiencies, of which AOT is clearly one, not to figure out the truth but rather to cement their status and relations with other group members (Stanovich & West 2007, 2008; Kahan & Stanovich 2016).

But I don’t want to be closed-minded toward other possibilities. 

So what do you think?

Refs

Baron, J. Myside bias in thinking about abortion. Thinking & Reasoning 1, 221-235 (1995).

Baron, J., Scott, S., Fincher, K. & Emlen Metz, S. Why does the Cognitive Reflection Test (sometimes) predict utilitarian moral judgment (and other things)? Journal of Applied Research in Memory and Cognition, 265-284 (2015).

Kahan, D. & Stanovich, K. Rationality and Belief in Human Evolution (2016), CCP/APPC Working paper available at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2838668.

Kahan, D.M. & Corbin, J.C. A note on the perverse effects of actively open-minded thinking on climate-change polarization. Research & Politics 3 (2016).

Kahan, D.M. ‘Ordinary science intelligence’: a science-comprehension measure for study of risk and science communication, with notes on evolution and climate change. J Risk Res, 1-22 (2016).

Kahan, D.M. Climate-Science Communication and the Measurement Problem. Advances in Political Psychology 36, 1-43 (2015).

Kahan, D.M. Ideology, Motivated Reasoning, and Cognitive Reflection. Judgment and Decision Making 8, 407-424 (2013).

Kahan, D.M., Peters, E., Dawson, E.C. & Slovic, P. Motivated numeracy and enlightened self-government. Behavioural Public Policy 1, 54-86 (2017).

Stanovich, K.E. Why humans are (sometimes) less rational than other animals: Cognitive complexity and the axioms of rational choice. Thinking & Reasoning 19, 1-26 (2013).

Stanovich, K. & West, R. On the failure of intelligence to predict myside bias and one-sided bias. Thinking & Reasoning 14, 129-167 (2008).

Stanovich, K.E. & West, R.F. Natural myside bias is independent of cognitive ability. Thinking & Reasoning 13, 225-247 (2007).

 

PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (19)

Suppose there were such a thing as "passively open-minded thinking"... then what?

I think the result (which I agree is very likely, although not a slam-dunk) of the skin-rash-gun-control test is that MS2R is a bit of a misnomer (to be added to Dan's growing collection of misnomers - such as "cultural" and "identity"). It's de-motivation, not motivation, that seems to be taking place - and it appears to be something in system 1 that is de-motivating (or otherwise disabling) some system 2 cognition. Perhaps this is splitting hairs here - but the skin rash part of the test shows that the high system 2 cognition (AOT or numeracy) was by default working in the control. So the gun control part shows de-motivating of that otherwise existing propensity (which was perhaps just motivation to get the correct answer) vs. the control. It doesn't show motivation to find a pleasing alternative to a displeasing result.

The high numeracy folks seem to be de-motivated to use their skills when faced with a pleasing result from their system 1 first pass in the gun control test. So, on this reading, and taking the "actively" of AOT seriously (maybe too seriously), one might suspect the result Dan gets above. Perhaps a similar expectation for any system 2 cognition: don't do any extra work that might weaken your own social standing!

But, what about "passively open-minded thinking"? Might there be such a thing - one that doesn't require motivation to use? Perhaps related to day-dreaming? Or, perhaps to threat-sensitivity? To a disposition to feel awe?

If so, then maybe it wouldn't so easily be disabled by a pleasing system 1 result.

Then, on to the question of how science curiosity fits in. One might suspect this is a system 2 behavior (it seems to require some kind of motivation, and involves some energy expenditure, etc.). Then it might be similarly disabled by pleasing system 1 results - assuming the above AOT result and the original high-numeracy result can be generalized into a full system 2 disabler. Does that imply SC is more likely a system 1 behavior?

Alternatively, perhaps SC is a weakening of the disabling effect, as due to a lack of concern over one's social standing, or perhaps a self-reflected inability to preserve one's social standing regardless of how hard one tries (geeks, square-pegs). Or, perhaps it is a group identity onto itself: Skeptics with a capital "S".

Or, perhaps (as I've read elsewhere), this whole system 1/system 2 dichotomy is overblown.

October 30, 2017 | Unregistered CommenterJonathan

Hi Dan,

These are interesting results that contradict a simple account of how AOT operates. But does it represent evidence for motivated reasoning?

The increase in accuracy for high AOT in the identity affirmed context is consistent with both accounts.

But the lack of difference between high and low AOT in the identity threatened context is inconsistent with both accounts (with the standard account predicting a positive correlation between AOT and accuracy, and the motivated reasoning account predicting a negative correlation). One could say, in fact, that this is equally contrary to both accounts.

So, perhaps we need another account...

-Gord

October 30, 2017 | Unregistered CommenterGordon Pennycook

An obvious point, but AOT would certainly imply not having an end point to shoot for while considering data. ThatBeingSaid, do we all think according to hypothesis-->test and so we would naturally have early hypotheses that are at or close to our preferred political position [availability]? These would be tested first and would, most likely, be the first to be considered *close enough* to what we consider "correct". Some cognitive fatigue could then be responsible for settling at or near the political position.

Perhaps the best test would be to test for this fatigue?

Thanks for this work.

October 30, 2017 | Unregistered CommenterCharles Kinsley

Possibly it is simply that people thought they already knew the answer, so didn't bother to calculate it?

October 30, 2017 | Unregistered CommenterNiV

@Gord-- I don't get your point. Conventional understanding of AOT would predict that in threat condition high scores would resist "my side" bias and thus do uniformly well in all conditions. There isn't much change in performance for high scorers in "affirmed" relative to the non-plitical condition (skin cream). But high scorers are 30 pct points less likely to get problem right when their idientity is threatened than they are when affirmed. The size of advantage over low socreing disappears. This is perfectly consistent with hypothesis that high AOT wouldn't block motivated reasoning, which is hypothezied mechanism for degraded performance in threat relative to affirmed conditions.

The improved performance of low AOT when affirmed is also plausibly result of motivated reasoning making them select right answer more often than in inert skin cream condition.

What am I missing?

October 30, 2017 | Registered CommenterDan Kahan

@Charles--There's always "another explanation" when one tests hypothesis empirically; results are always overdetermiend relative to theories.

But we can judge the alternaives to see if they are as plausible as tested hypothesis. I don't see why fatigue would be applying unevenly among high & low AOT scorers (remember too this is a between subjects design; they aren't solving multiple covariance problems).

But one person's "implausible hypothesis" is another's animating thesis in a test of his or her own that is designed to distiguish the two. What design would you propose that would do this? We've looked at latency & found, consistent w/ other studies, that high cognitive proficiency subjects don't take longer to respond when they get right answer, a findihng consistent w/ other studies

October 30, 2017 | Registered CommenterDan Kahan

link drop - not so politically motivated climate change reasoning:
https://phys.org/news/2017-10-political-views-limited-impact-climate.html

October 30, 2017 | Unregistered CommenterJonathan

@Dan

To clarify, my understanding is that the motivated reasoning account predicts that people are using reasoning to protect their identity. Thus, AOT/CRT/numeracy/IQ/etc. (things that facilitate reasoning) should be associated with increased rejection of ideologically threatening information (i.e., lower accuracy). This is the pattern of results for Republicans in the Nature Climate Change paper, for example.

As it stands, you're setting it up so that any pattern of results that fails to support the traditional reasoning account is taken as positive evidence for the motivated reasoning account.

Put differently, there may be other reasons for the null correlation with AOT in the identity-threatened context.

October 30, 2017 | Unregistered CommenterGordon Pennycook

@Gordon--

To clarify, my understanding is that the motivated reasoning account predicts that people are using reasoning to protect their identity. Thus, AOT/CRT/numeracy/IQ/etc. (things that facilitate reasoning) should be associated with increased rejection of ideologically threatening information (i.e., lower accuracy). This is the pattern of results for Republicans in the Nature Climate Change paper, for example..

That's exactly what the data show: relative to how they do in "identity affirmed" condition,high AOT siubjects in "identtity threatened" are 30 pct points less likely to get answer right. That's a pretty big drop in accuracy.

Maybe there is something about the way the data are reported that is confusing you?

October 31, 2017 | Registered CommenterDan Kahan

@Jonathan -- that's a great study. But as far as I can tell -- from having seen HJS et al. present these data in a couple of conferences -- they show only that there's less bias in updating of beliefs about recent weather. They aren't reporting whether beliefs in climate change (human caused or otherwise) are being updated in way that closes gap between partisans. NOte what they say here: "more work is needed to understand how this kind of updating affects broader beliefs about climate change...."

I could be wrong, not having yet read the paper, of course

October 31, 2017 | Registered CommenterDan Kahan

"That's exactly what the data show: relative to how they do in "identity affirmed" condition,high AOT siubjects in "identtity threatened" are 30 pct points less likely to get answer right. That's a pretty big drop in accuracy."

Sorry, what I meant is that the motivated reasoning account would predict that if high AOT subjects are 30 points less likely to get the answer right, low AOT subjects only should be 20 points (or some smaller number) less likely to get the answer right.

This would support the claim that the 30 point difference is a consequence of motivated reasoning per se, as opposed to some other process.

From the Kahan, Peters, et al. (2017) abstract: "This outcome supported ICT, which predicted that more Numerate subjects would use their quantitative-reasoning capacity selectively to conform their interpretation of the data to the result most consistent with their political outlooks."

What am I missing?

October 31, 2017 | Unregistered CommenterGordon Pennycook

OK, Hypothesis:

People put more effort into thinking about alternatives the more puzzled or surprised they are.

So, low suprise situation is when the obvious interpretation of what they're told is something they already know. The medium surprise situation is when they're told about something they have no idea about. The high surprise situation is when they're told something they previously believed to be untrue.

In the low surprise situation, they do no thinking, and go with the obvious answer which they consider to be obviously correct. There's no division between low and high AOT, because there's obviously no need for actively open minded thinking. It's obvious.

In the medium surprise situation, most people still go with the obvious answer, but high-AOT respond to the higher level of puzzlement and a higher proportion of them realise there's a more sophisticated, alternative interpretation giving the opposite answer.

In the high surprise situation, everyone is puzzled and activated to search for alternative explanations. A much higher proportion even of the low-AOT people realise there's an alternative interpretation (or they cheat and just give the obviously true answer without bothering to work out how and why), and the high-AOT people go above and beyond. Again , there's a split, because people are actively searching for explanations beyond the obvious, and high-AOT people are better at doing that.

So, the results are predicted by the "surprised people think harder" hypothesis. Does that mean the hypotheis is experimentally confirmed? Can researchers recognise the "confirming the consequent" fallacy more easily when it's apparently confirming a hypothesis they don't agree with?

October 31, 2017 | Unregistered CommenterNiV

@Gordon

Null effect? There's a 30% drop in probability of right answer on part of high AOT between "affirmed" & "threatened." That's no null effect!

I think you are conflating MS2R with buffer against motivated reasoning. Here there is motivated reasoning as a main effect-- people generally fit the results in 2x2 to their cultural presuppositions. AOT is supposed to counteract that bias. But it doesn't; high AOT rspts suffer same degradation in performance as low. End of story.

You don't see that high AOT do no better than low when one sees motivated reasoning in latter. Maybe that is b/c you have a picture of my & Stanovich's paper stuck in your mind? (It will be resubmitted soon, btw.)

November 1, 2017 | Registered CommenterDan Kahan

@NiV-- to me, your explanation sounds like ours, just more fine-grained or more attentive to "inside the head"

November 1, 2017 | Registered CommenterDan Kahan

"@NiV-- to me, your explanation sounds like ours, just more fine-grained or more attentive to "inside the head""

Mine doesn't involve cultural shibboleths, protection of threatened identities, political polarisation, or trying to fit in with your ideological peer group. In particular, it doesn't involve having to make any choice between what people believe and who they are. (Or between being honest and effective, for that matter.)

Those "in the head" aspects are a big part of the presentation of the explanation, and could potentially trigger unnecessary opposition from those who might be interested in the science, but whose "identities are threatened" by the suggestion that they're lying about their true beliefs just to fit in with their political peer group, when as far as they're concerned they're just telling the truth as they see it (and therefore see it as contradicting a high-confidence prior belief of theirs). Either way, it's a potentially "polluting" issue.

If you've got actual evidence that it *is* identity protection, rather than surprise at prior-belief-contradicting claims, then that's fair enough. I'd be interested in seeing it. But if there's no actual evidence for it - if it's just one of a number of equivalent hypotheses that the evidence doesn't distinguish between, that you've just picked as a representative or illustrative hypothesis - then the combination of the identity conflict with the lack of evidence presented for it could turn people off the bits you do have evidence for.

--
By the way, do you ever say things you don't truly believe just to fit in with your ideological peer group? I'm not judging - just curious. It's one of those popular truisms that, in the absence of specific information to the contrary, people tend to assume other people are a lot like themselves.

November 1, 2017 | Unregistered CommenterNiV

NiV,

I see similarities between this debate an this other one:
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3049660

Here, the presence of an issue that may induce identity-protective cognition is construed to be proof of identity-protective cognition when it induces bias, even though there are other potential ways it could induce bias. Similar to loss aversion vs. status quo or inaction-over-action biases in that other debate.

November 1, 2017 | Unregistered CommenterJonathan

Hi Dan,

I think the confusion is that I'm talking about MS2R (motivated System 2 reasoning) and not merely the "buffer against motivated reasoning". That is, I'm not conflating the two. I'm just focusing on the specific claims of the MS2R account (what I thought was ICT).

As noted (from the original motivated numeracy paper): "This outcome supported ICT, which predicted that more Numerate subjects would use their quantitative-reasoning capacity selectively to conform their interpretation of the data to the result most consistent with their political outlooks."

The relevant prediction for the present study would be that high AOT would be less accurate than low AOT in the identity threatened condition. Correct?

November 1, 2017 | Unregistered CommenterGordon Pennycook

@Gord

Answer to final question is: Nope.

Again, the only thing I'm trying to say here about AOT is that it doesn't reliably curb "myside bias”—at least not on the class of issues in connection with which factual positions become entangled with group identity. That's contrary to what one would expect based on the AOT construct; resistance to myside bias is the core AOT characteritic as presented by Baron, Stanovich & others. So either the construct needs to be modified to explain these results or the measure of AOT needs to be improved.

As for MS2R, it requires only that individuals high in the relevant critical-reasoning dispositon *more reliably* select the identity-congruent response than those who are low in that disposition. That doesn't necessarily demand that the former be shown to be more or less accurate than the latter. If, e.g., the high-disposition types are more accurate when the correct answer is identity-congruent but no more inaccurate than low when the correct answer is identity non-congruent, then the high-disposition types will more reliably express their identity than will low-dispositon types.

Remember that the whole point of MS2R is to *explain* survey results that show the most numerate, most cognitively reflective, most science comprehending etc. are the most polarized. Studies like “Motivated Numeracy” model how that could happen—by showing that in a controlled experimental environment those high in critical reasoning of one sort or another are “better” at engaging evidence in an identity-congruent fashion. (If one doesn’t observe that pattern in a lab experiment, btw, that doesn’t mean “it doesn’t happen,” “isn’t important” etc; it means that the study design wasn’t connecting with whatever real-world mechanisms generate this observable real-world state of affairs.)

November 2, 2017 | Registered CommenterDan Kahan

Thanks for the reply Dan. I'm mostly with you, but I don't think you're assessing the two hypotheses fairly.

One one hand, a case where AOT fails to lead to improved accuracy in identity threatened scenarios means that the typical theory needs to be reassessed (note that high AOT are still more accurate overall because they score higher in the identity consistent scenario - i.e., they are updating when they should be).

On the other hand, a lack of evidence for MS2R "doesn't mean it doesn't happen".

We agree on the latter point - motivated reasoning sometimes happens and, sometimes, more analytic people are more likely to engage in it.

Where we disagree is on the first point. I think the same is true for the classic AOT/CRT story. Sometimes more analytic people are more reasonable - and this even occurs in cases where they have a motivation to not be reasonable. Such as in our recent paper on fake news: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3023545

So perhaps it shouldn't be "classic story vs. MS2R", but instead a matter of figuring out when analytic thinking hurts, when it helps, and when it's irrelevant.

November 5, 2017 | Unregistered CommenterGordon Pennycook

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>