Replicate "Climate-Science Communication Measurement Problem"? No sweat (despite hottest yr on record), thanks to Pew Research Center! - Cultural Cognition of Health

One of the great things about Pew Research Center is that it posts all (or nearly all!) the data from its public opinion studies.  That makes it possible for curious & reflective people to do their own analyses and augment the insight contained in Pew’s own research reports.

I’ve been playing around with the “public” portion of the “public vs. scientists” study, which was issued last January (Pew 2015). Actually Pew hasn’t released the “scientist” (or more accurately, AAAS membership) portion of the data. I hope they do!

But one thing I thought it would be interesting to do for now would be to see if I could replicate the essential finding from “The Climate Science Communication Measurement Problem” (2015).

In that paper, I presented data suggesting, first, that neither “belief” in evolution nor “belief” in human-caused climate change were measures of general science literacy.  Rather both were better understood as measures of forms of “cultural identity” indicated, respectively, by items relating to religiosity and items relating to left-right political outlooks.

Second, and more importantly, I presented data suggesting hat there is no relationship between “belief” in human-caused climate change & climate science comprehension in particular. On the contrary, the higher individuals scored on a valid climate science comprehension measure (one specifically designed to avoid the confound between identity and knowledge that confounds most “climate science literacy” measures), the more polarized the respondents were on “belief” in AGW–which, again, is best understood as simply an indicator of “who one is,” culturally speaking.

Well, it turns out one can see the same patterns, very clearly, in the Pew data.

Patterned on the NSF Indicators “basic facts” science literacy test (indeed, “lasers” is an NSF item), the Pew battery consists of six items:

As I’ve explained before , I’m not a huge fan of the “basic facts” approach to measuring public science comprehension. In my view, items like these aren’t well-suited for measuring what a public science comprehension assessment ought to be measuring: a basic capacity to recognize and give proper effect to valid scientific evidence relevant to the things that ordinary people do in their ordinary lives as consumers, workforce members, and citizens.

One would expect a person with that capacity to have become familiar with certain basic scientific insights (earth goes round sun, etc.) certainly.  But certifying that she has stocked her “basic fact” inventory with any particular set of such propositions doesn’t give us much reason to believe that she possesses the reasoning proficiencies & dispositions needed to augment her store of knowledge and to appropriately use what she learns in her everyday life.

For that, I believe, a public science comprehension battery needs at least a modest complement of scientific-thinking measures, ones that attest to a respondent’s ability to tell the difference between valid and invalid forms of evidence and to draw sound inferences from the former.  The “Ordinary Science Intelligence” battery, used in the Measurement Problem paper, includes “cognitive reflection” and “numeracy”modules for this purpose.

Indeed, Pew has presented a research report on a more fulsome science comprehension battery that might be better in this regard, but it hasn’t released the underlying data for that one.

But anyway, the new items that Pew included in its battery are more current & subtle than the familiar Indicator items, & the six-member Pew group form a reasonably reliable (α = 0.67), one dimensional scale– suggesting they are indeed measuring some sort of science-related apptitude.

But the fun stuff starts when one examines how the resulting Pew science literacy scale relates to items on evolution, climate change, political outlooks, and religiosity.

For evolution, Pew used it’s two-part question , which first asks whether the respondent believes (1) “ Humans and other living things have evolved over time” or (2) “Humans and other living things have existed in their present form since the beginning of time.”

Subjects who pick (1) then are asked whether (3) “Humans and other living things have evolved due to natural processes such as natural selection” or (4) “A supreme being guided the evolution of living things for the purpose of creating humans and other life in the form it exists today.”

Basically, subjects who select (2) are “new earth creationists.” Subjects who select (4) are generally regarded as believing in “theistic evolution.”  Intelligent design isn’t the only variant of “theistic evolution,” but it is certainly one of the accounts that fit this account.

Only subjects who select (3)– “humans and other living things have evolved due to natural processes such as natural selection” — are the only ones furnishing the response that reflects science’s account of the natural history of humans.

So I created a variable, “evolution_c,” that reflects this answer, which was in fact selected by only 35% of the subjects in Pew’s U.S. general public sample.

On climate change, Pew assessed (using two items that tested for item order/structure effects that turned out not to matter) whether subjects believed (1) “the earth is getting warmer mostly because of natural patterns in the earth’s environment,” (2) “the earth is getting warmer mostly because of human activity such as burning fossil fuels,” or (3) “there is no solid evidence that the earth is getting warmer.”

About 50% of the respondents selected (2).  I created a variable, gw_c, to reflect whether respondents selected that response or one of the other two.