follow CCP

Recent blog entries
popular papers

Science Curiosity and Political Information Processing

What Is the "Science of Science Communication"?

Climate-Science Communication and the Measurement Problem

Ideology, Motivated Cognition, and Cognitive Reflection: An Experimental Study

'Ideology' or 'Situation Sense'? An Experimental Investigation of Motivated Reasoning and Professional Judgment

A Risky Science Communication Environment for Vaccines

Motivated Numeracy and Enlightened Self-Government

Making Climate Science Communication Evidence-based—All the Way Down 

Neutral Principles, Motivated Cognition, and Some Problems for Constitutional Law 

Cultural Cognition of Scientific Consensus
 

The Tragedy of the Risk-Perception Commons: Science Literacy and Climate Change

"They Saw a Protest": Cognitive Illiberalism and the Speech-Conduct Distinction 

Geoengineering and the Science Communication Environment: a Cross-Cultural Experiment

Fixing the Communications Failure

Why We Are Poles Apart on Climate Change

The Cognitively Illiberal State 

Who Fears the HPV Vaccine, Who Doesn't, and Why? An Experimental Study

Cultural Cognition of the Risks and Benefits of Nanotechnology

Whose Eyes Are You Going to Believe? An Empirical Examination of Scott v. Harris

Cultural Cognition and Public Policy

Culture, Cognition, and Consent: Who Perceives What, and Why, in "Acquaintance Rape" Cases

Culture and Identity-Protective Cognition: Explaining the White Male Effect

Fear of Democracy: A Cultural Evaluation of Sunstein on Risk

Cultural Cognition as a Conception of the Cultural Theory of Risk

« So is AOT measuring AOT? I think so; the steel cage results don't necessarily imply otherwise | Main | New NAS Report on Science Literacy, Cultural Values & Political Conflict on Policy-relevant Facts »
Monday
Aug152016

Still another cognitive-style steel cage match: CRT vs. AOT go "head to head" on belief in climate change & belief in evolution

Jonathan aka "Don King" CorbinThe carnage continues! SCS (aka "Science Curiosity Scale") is taking a rest after having bashed its way to the top of the open-mindedness rankings, but this week we bring you CRT vs. AOT in a match arranged by the Don King of the cognitive-style steel cage match world, Jonathan Corbin!

If You Open Your Mind Too Much Your Brain Might Fall Out, But At Least It’ll Have a Parachute to Soften the Landing

Jonathan Corbin

Frank Zappa said, “A mind is like a parachute. It doesn’t work if it is not open” (though Thomas Dewar may have coined the phrase).  This is the motivation behind the psychological scale measuring actively open-minded thinking (AOT; Baron, 2008).  AOT is a self-report scale meant to measure the tendency to which an individual seeks out information that conflicts with one’s own beliefs.  So, simply having an open mind is only half of what this scale tries to measure – the other half is the propensity to look for information that disagrees with your current beliefs.  At first look, AOT seems like a silver bullet in terms of understanding why some people seem so resistant to scientific information that threatens their beliefs. 

Recent work by Dan Kahan and colleagues has shown that another individual difference measure – Science Curiosity – has been shown to relate to increased acceptance of human-caused global warming regardless of political affiliation.  Whereas performance measures like the Cognitive Reflection Test (which measures some combination of impulse control/inhibition and mathematical ability) and measures of scientific knowledge predicted increased polarization on politically charged scientific issues like climate change, science curiosity predicted the opposite!  As soon as I saw this result, I was immediately curious about how the AOT would do in such a comparison.  The obvious prediction is that AOT should perform just like science curiosity – an increased predilection for seeking out information that disagrees with one’s beliefs should definitely predict increased acceptance of human-caused climate change!

Dan was nice enough to direct me to his publicly available dataset in which they measured climate change beliefs as well as AOT (along with CRT, science knowledge, and many other variables), allowing us to test the hypothesis that individuals higher in AOT should be more accepting of climate change regardless of political affiliation.  As you’ve probably guessed if you read Dan’s previous post, it turns out that AOT was more similar to performance measures like the CRT, showing greater polarization with higher scores on the scale.

So, unfortunately it appears to be the case that AOT is not the silver bullet that I once thought it could be.  Perhaps, rather than Zappa’s quote of the mind as a parachute, I should be looking to Tim Minchin, who said, “If you open your mind too much, your brain might fall out.”  To further explore this pattern, I looked at another contentious topic – evolution.  Rather than examining political identification, for this analysis, I relied on religiosity (given that there is also a reason for many highly religious individuals to deny evolution as an identity protective measure).  The other reason I looked at religiosity is that there is a lot of AOT research linking higher religiosity with lower AOT.  This is interpreted as evidence that greater religiosity is associated with a heavier reliance on associative intuition (or “going with your gut”) as opposed to deliberative thinking (Gervais & Norenzayan, 2012; Pennycooke et al., 2013).  Few (if any) other studies collect nationally representative samples with such a large number of participants, so Kahan’s ordinary science intelligence dataset allowed us to test whether greater AOT in religious individuals relates to increased acceptance of evolution. 

Results show a similar pattern to the climate change question, with CRT and AOT behaving similarly in that higher AOT failed to predict greater acceptance of evolution in the highly religious.

If there is any consolation, it is that we can say that higher AOT in the highly religious did not predict decreased belief in evolution.  However, this data certainly does not give hope for the prediction that belief should increase with greater AOT among the highly religious.  Similar to political identity and climate change, whereas the overall relationship between AOT and belief in Evolution remains positive, broken down by religiosity, the picture quickly becomes more complicated.

By now, you are probably asking yourself whether there is any real difference between the CRT and AOT.  Definitionally, they are distinct (though expected to share variance), however, so far I haven’t given you much data to encourage that belief.  Well first of all, there is other research out there to support a difference.  For example, Haran, Ritov, and Mellers (2013) examined both AOT and CRT scores in relation to forecasting accuracy and information acquisition (basically what predicts how much information you’re willing to take in as well as your accuracy in predicting an outcome related to such information).  They demonstrated that AOT predicted superior forecasting over and above any effect of CRT (and this was mediated by information acquisition). 

We can also look for differences in the ordinary science intelligence dataset that we previously examined.  Rather than looking at individualls' belief in evolution, I analyzed level of agreement with the statement ““From what you’ve read and heard, is there solid evidence that the average temperature on earth has been getting warmer over the past few decades, or not?”.  This question differed from the last, in that it does not ask about agreement with human-caused climate change – it only asks if there is solid evidence based on “what you’ve read and heard”.  The data showed that there was no main effect of CRT and no interaction between CRT and political affiliation (political affiliation did predict agreement with conservatives less likely to agree than liberals).  However, AOT did show a significant relationship, predicting greater agreement.

So, where does this leave us? It seems that although AOT is likely distinct from performance measures like the CRT, it falls into the same trap when it comes to science issues that generate conflicts with individuals’ identities.  Despite the fact that AOT is meant to measure one’s propensity toward seeking out belief-inconsistent information, it fails to predict higher levels of agreement with evidence-based claims that cue these identities. 

Given the final analysis reported here (and the literature as a whole), claiming that the result boils down to measurement error is probably incorrect.  It is more likely that one’s propensity to seek out information (particularly information that conflicts with one’s beliefs) is simply insufficient in countering the strength of cultural identity in swaying reasoning.  With regards to the evidence for human-caused climate change, there is an enormous amout of information available online.  Simply see the following website for a list of arguments in favor and against human-caused climate change.  This seems to be the perfect resource for someone high in AOT.  However, a lot of these arguments on both sides are technical, and it is possible that someone high in AOT may not be satisfied with trusting experts’ interpretation of the evidence, and would rather judge for themselves.  The need to judge for themselves mixed with the desire to come to conclusions that support one’s identity could very well increase polarization (or at the very least lead to no increase in support for those who’s identities support disagreement).  (It is worth the reminder that these are post-hoc explanations that require testing).

So, is Zappa correct in that an open mind a parachute or should we listen to Minchin who says that it is a recipe for losing one’s brain?  Well, the answer (because it is psychology) is--it depends!  When dealing with non-polluted science topics you should expect a positive relationship between AOT and agreement (maybe above and beyond performance measures like the CRT).  However, once you throw in the need to protect one’s identity, AOT is not going to be the solution.  So, why is science curiosity different from AOT?  Perhaps science curiosity is less about belief formation and more of a competing identity.  Whereas AOT is focused on how someone forms and changes beliefs, science curiosity is simply the need to consume scientific information.  Maybe instead of trying to throw information at people hoping that it’ll change their minds, we should start fostering a fascination with science. 

Refs

Baron, J. (2008). Thinking and Deciding. Cambridge University Press.

Gervais, W. M., & Norenzayan, A. (2012). Analytic thinking promotes religious disbelief. Science336(6080), 493-496.

Haran, U., Ritov, I., & Mellers, B. A. (2013). The role of actively open-minded thinking in information acquisition, accuracy, and calibration. Judgment and Decision Making8(3), 188.

Kahan, D. M. (2016). ‘Ordinary science intelligence’: a science-comprehension measure for study of risk and science communication, with notes on evolution and climate change. Journal of Risk Research, 1-22.

Pennycook, G., Cheyne, J. A., Barr, N., Koehler, D. J., & Fugelsang, J. A. (2014). Cognitive style and religiosity: The role of conflict detection. Memory & Cognition42(1), 1-10.

PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (5)

I'm curious how accurate the AOT measure is. In other words, can respondents achieve a high score by grading themselves highly on open-mindedness characteristics when in fact they are actually relatively closed-minded, at least in some realms? Has the AOT tool been validated against actual behavior?

August 16, 2016 | Unregistered CommenterMaine Wayne

@Maine--

Super good question... Most of the self-report measures of cognitive style -- like need for cogntion, need for clousre, etc. -- suck compared to objetive meaures like the Cogntive Refelction Test (Toplak, West & Stanovich 2011; Liberali, Reyna, Furlan & Pardo, 2011). It's also well established that people can't determine through introspection whether they are vulnerable to one or another bias associated with overreliance on heuristic reasoning (West, Mserve, Stanovich 2012). So one might wonder why any self-report measure of open-mindedness should be expected to work.

But the only thing one can do is test & see whether AOT predicts the sorts of things that evince open-mindedness. "Don King" Corbin cites some evidence that AOT does well; there's more out there.

I think the poor perfomance it might be dipslaying in regard to evoution & climate change, however, is not a good test. Even validated objective measures of crtitical thinking display the sort of anomalous magnfication of poltarization that we've seen in these "steel cage matches"-- so the performance of AOT in this rspect doesn't count against it any more than it counts against those.

I think the class of beliefs we are talking about here--ones the express identitity-- are ones where it makes sense to think people are *using* their reason to form identity-congrugent rather than truth-congruent beliefs. If so, then, higher cogntive proficiency will magnify polariztaion.

But those are special cases ...

See the anti- and penultimate paragraphs of our working paper on AOT & climate change polarization & also the new National Academy of Sciences Report on science literacy & science attitudes for more discussion.

Refs

Liberali, J.M., Reyna, V.F., Furlan, S., Stein, L.M. & Pardo, S.T. Individual Differences in Numeracy and Cognitive Reflection, with Implications for Biases and Fallacies in Probability Judgment. Journal of Behavioral Decision Making (2011).

Toplak, M., West, R. & Stanovich, K. The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks. Memory & Cognition 39, 1275-1289 (2011).

West, R.F., Meserve, R.J. & Stanovich, K.E. Cognitive sophistication does not attenuate the bias blind spot. Journal of Personality and Social Psychology 103, 506 (2012).

August 16, 2016 | Registered CommenterDan Kahan

Jonathan and Dan, I believe that one of the things that needs to be made more explicit in these “steel cage” analysis is the recognition that the steel cage is too small. Academic Social Science seems to be suffering from the fact that unlike entities like Google, Facebook, political consultants, marketing agencies, or corporate logistics, access to Big Data analytics is very limited. Understanding that these metrics are very subject to individual's identity protective mechanisms, and urges to simplify things that the individual might be able to judge for themselves ought to, in my opinion, lead to the conclusion that in a short question set there really isn't any silver bullet.

That recognition for the need for more data is acknowledged in references to Dan's work given in the NAS Science literacy report, but obviously limited by the willingness of the members of the public surveyed to answer more questions:
"More recently, Kahan (2015; 2016) suggests that the current Indicators items—including
both the factual knowledge questions and the process questions—could likely be combined into a
single measure but that the available questions are too easy (also see Pardo and Calvo, 2004). He
argues that the standard measures do a reasonable job at differentiating those individuals with
low levels of science knowledge from those individuals with medium levels of science
knowledge, they seem to be less useful in differentiating people with medium knowledge from
people with high knowledge. Kahan (2016) has thus proposed a new measure that includes some
of the more difficult NSF items and adds one item from a list of questions used by the Pew
Research Center (see, e.g., Funk and Rainie, 2015), as well as several questions from a short
form of a well-established numeracy scale (see, e.g., Weller et al., 2013)."

Results may be situational. Rand science analyst Deborah A. Cohen describes what she thinks is needed for the very limited special case regarding the impact of nutritional labeling on healthy food choices. See: http://www.rand.org/blog/2016/08/new-nutrition-labels-will-do-little-to-bring-quality.html. She notes that:

“Despite an increasing amount of research on decisionmaking and behavioral economics, the findings used to inform the design of the new Nutrient Facts label were limited and incomplete.”

She states that in the supermarket, shopping is a habitual behavior with most purchasing decisions made in less than a second.

“Even those dedicated to a healthy diet can often be undermined by point-of-purchase marketing intended to disrupt cognitive, thoughtful decisionmaking and promote instant gratification. Because people usually lack insight into how marketing practices interfere with their ability to make good food choices, regulators need to conduct careful research to learn how to protect consumers from such subliminal shopping hazards. “

Her solution seems a bit out of reach, and pales in comparison to analogous information is easily accessible by the relevant corporations, who collect massive amounts of such information with customer buying cards and other mechanisms:

“To accelerate improvements to the American diet, the U.S. government should invest in efforts aimed at doing research in authentic shopping environments. A series of experimental retail outlets could be created where scientists in many regions would have the opportunity to manipulate the various factors that influence what people buy. Supermarket chains already do this with an eye toward increasing profit, rather than improving health — and don't share the findings, saying they are proprietary. “

In my opinion, a better start would be in expanding what Big Food is required to disclose. In that regard, I think that the linkage of GMO labeling to an online database is a good start. This can be expanded to include much information on food origins and nutritional value. As noted above, detailed decisions are not going to be made at the point of purchase anyway. But such data can be mined, and then presented to consumers in a manner that would appeal to their interests and allow them greater discretion in utilizing their purchasing power.

Another example of new access to data leading to new advances is here: http://www.fastcompany.com/3062731/most-innovative-companies/23andmes-consumer-dna-data-goldmine-is-starting-to-pay-off

“Each of these studies used insights gathered from customers of 23andMe, the Google-backed company that makes a direct-to-consumer genetic test kit. Perhaps best-known for its battles with regulators over its consumer genetics test in 2013, 23andMe has quietly expanding its business to include brokered access to its database of over 1 million people’s DNA.”

August 22, 2016 | Unregistered CommenterGaythia Weis

@GAythia--

I agree! More data are necessary -- & also better understanding of the "situations" in which the dispositions in question operate as they do.

August 22, 2016 | Registered CommenterDan Kahan

But what I was hoping to do was to rally an academic social science army to mount an attack on corporate privately held Big Data! Information is power, and I believe more of it belongs in the public domain.

August 23, 2016 | Unregistered CommenterGaythia Weis

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>