follow CCP

Recent blog entries
« The evolution debate isn't about science literacy either | Main | Who has a better comprehension of science--"skeptics" or "nonskeptics"? »
Friday
Jun012012

More on the statistical illiteracy/inanity of trying to figure out "yeah, but whose is bigger"

So I said in the last postthat the "57% vs. 56%" factoid can't be read to indicate that one side has better comprehension of science.

It was (quite reasonably) pointed out to me in a comment that our Nature Climate Change study reported statistically significant correlations between science literacy and numeracy (and the composite science comprehension scale that aggregated them), on the one hand, and climate-change risk perception, on the other. What bearing does that have on the issue of who has greater science comprehension-- the "yeah, but whose is bigger?" question--in the climate change debate?

To start, there's no tension between the statistical computations here. If one looks at the correlation between the continuous science comprehension (science literacy plus numeracy) scale and climate change, it's negative (r = -0.09) and significant (t-statistic = -3.35). But the difference in the mean scores of the most concerned and least concerned halves of the sample is not significant. That can certainly happen when one splits the sample and treats two continuous measures as categorical ones (the opposite can happen, too).

But that's particularly likely to happen when the correlation between the continuous variables is tiny. That's so here.

People who are numerate are likely to suspect that= -0.09 (r = -0.05 for science literacy by itself!) is small (which is how the paper characterizes this effect)--way too small to be responsible for the intensity of the climate change debate in our society.

But it's actually pretty bad craft to expect anyone to figure out whether an effect size is meaningful from bare correlation coefficients. Readers should be shown the effect in some way that conveys its practical importance.

The question under investigation in our study was what explains climate change conflict--differences in science comprehension or differences in cultural outlooks? One shouldn't really have to know statistics to see the answer in a figure like this:

 

I won't say anything more about the difference between statistical significance and practical significance because there's an excellent post that addresses it in the context of science comprehension and climate change at the Blackboard, in appreciation of & gratitude for which I am posting this:

 

But that leaves room to discuss another, and in my mind, more interesting point about statistical illiteracy that is reflected in obsessing over that effect: it’s meaningless in real-world terms.

The principal finding was that science comprehension interacts with cultural predispositions: individuals who are predisposed by their group values to skepticism become more skeptical, and those predisposed to concern become more concerned, as science comprehension increases.

So it is in fact misleading to characterize greater science comprehension as having any “main effect” toward either skepticism or acceptance. It has one or the other depending on other characteristics.

The only thing the “main effect” really conveys in these circumstances is the frequency of the two types (or maybe the intensity of the effect in one or the other, if it varies meaningfully) in one’s sample. Moreover, that’s true even if one’s “sample” is in fact a census: if the correlation when one looks at the population as a whole is negative, then there are simply more people out there predisposed (and/or more strongly predisposed) to fit the evidence to a "skeptical" conclusion; and if the correlation is positive — then the number predisposed (or predisposed more intensely) to see evidence as justifying concern is greater. End of story.

I talked to a researcher recently who tried to convince me that one should see a small positive correlation between science literacy & some other issue that had an interaction with ideology as meaning that when one “controls” for ideology, science literacy increases concern …” I kept trying to tell him to think about what it was he was actually modeling and how silly it is to describe the sample “mean” as the “effect controlling for” something that interacts with a characteristic that varies systematically in people in the real world.

I felt like I was arguing with the guy from spinal tap who kept saying, “mine goes to 11.”

PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (2)

I love the graph with the linear regression shown!

that when one “controls” for ideology, science literacy increases concern

Aside from the fact that this is a strange notion from the POV of statistics, it's also of little practical importance.

Suppose someone hopes that by educating the public they can increase the level for concern. But the public simply has whatever mix of ideology it has. You can't "correct" for ideology when trying to increase (or decrease) concern in the general public through attempts to improve (or degrade) their understanding of science and math.

You also can't do more fanciful things. No one is going to block access to science and math education to those individuals (often conservatives) for whom education appears to diminish concern in climate in the hope that one might thereby prevent an individual conservative's concern about climate from dropping even further when they learn what an atom is or how to compute the efficiency of a power cycle.

June 1, 2012 | Unregistered Commenterlucia

Lucia-- I agree! It's a mistake to have a model that purports to "control" for things when one can't imagine a real-world treatment in which you could manipulate the thing being controlled. Gelman & Hill have great discussion of this! The fact that so many people don't get this shows how lethal the world can get when someone w/ no conception of *modeling* can just walk into the corner software store & buy a multivariate regression package

June 1, 2012 | Registered CommenterDan Kahan

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>