Weekend update: modeling the impact of the "according to climate scientists prefix" on identity-expressive vs. science-knowledge revealing responses to climate science literacy items
Saturday, March 26, 2016 at 11:44AM
Dan Kahan

I did some analyses to help address issues that arose in an interesting discussion with @dypoon about how to interpret the locally weighted regression outputs featured in "yesterday's" post. 

Basically, the question is what to make of the respondents at the very highest levels of Ordinary Science Intelligence

When the prefix "according to climate scientists" is appended to the items, those individuals are the most likely to get the "correct" response, regardless of their political outlooks. That's clear enough.

It's also bright & clear that when the prefix is removed, subjects at all levels of OSI are more disposed to select the identity-expressive answer, whether right or wrong. 

What's more those highest in OSI seem even more disposed to select the identity-expressive "wrong" answer than those modest in that ability.  Insfar as they are the ones most capable of getting the right answer when the prefix is appended, they necessarily evince the strongest tendency to substitute the incorrect identity-expressive for the correct, science-knowledge-evincing response when the prefix is removed.

But are those who are at the very tippy top of the OSI hierarchy resisting the impulse (or the consciously perceived opportunity) to respond in an identity-protective manner--by selecting the incorrect but ideologically congenial answer-- when the prefix is removed?  Is that what the little little upward curls mean at the far right end of the dashed line for right-leaning subjects in "flooding" and for left-leaning ones in "nuclear"?

@Dypoon seems to think so; I don't.  He/she sensed signal; I caught the distinct scent of noise.

Well, one way to try to sort this out is by modeling the data.

The locally weighted regression just tells us the mean probabilities of "correct" answers at tiny little increments of OSI. A logistic regression model can show us how the precision of the estimated means--the information we need to try to ferret out signal from noise-- is affected by the number of observations, which necessarily get smaller as one approaches the upper end of the Ordinary Science Intelligence scale.

Here are a couple of ways to graphically display the models (nuclear & flooding). 

This one plots the predicted probability of correctly answering the items with and without the prefix for subjects with the specified political orientations as their OSI scores increase: 

 

This one illustrates, again in relation to OSI, how much more likely someone is to select the incorrect, identity-expressive response for the no-prefix version than he or she is to select the incorrect response for the prefix version:

The graphic shows us just how much the confounding of identity and knowledge in a survey item can distort measurement of how likely an individual is to know climate-science propositions that run contrary to his or her ideological predisposition on global warming.

I think the results are ... interesting.

What do you think?

To avoid discussion forking (the second leading cause of microcephaly in the Neterhlands Antilles), I'm closing off comments here.  Say your piece in the thread for "yesterday's" post.

Article originally appeared on cultural cognition project (http://www.culturalcognition.net/).
See website for complete article licensing information.