follow CCP

Recent blog entries
« Why cultural predispositions matter & how to measure them: a fragment ... | Main | Who needs to know what from whom about climate science »
Friday
Nov292013

What does a valid climate-change risk-perception measure *look* like?


This graphic is a scatterplot of subjects from a nationally representative panel recruited last summer to be subjects in CCP studies.

The y-axis is an eight-point climate-change risk-perception measure. Subjects are "color-coded" consistent with the response they selected.

The x-axis arrays the subjects along a 1-dimensional measure of left-right political outlooks formed by aggregating their responses to a five-point "liberal-conservative" ideology measure and a seven-point party-identification one (α = 0.82).

I can tell you "r = -0.65, p < 0.01," but I think you'll get the point better if you can see it! (Here's a good guideline, actually: don't credit statistics-derived conclusions that you can't actually see in the data!)

BTW, you'll see exactly this same thing -- this same pattern -- if you ask people "has the temperature of the earth increased in recent decades," "has human activity caused the temperature of the earth to increase," "is the arctic ice melting," "will climate change have x, y, or z bad effect for people," etc.

Members of the general public have a general affective orientation toward climate change that shapes all of their more particular beliefs about it.  That's what most of the public's perceptions of the risks and benefits of any technology or form of behavior or public policy consist in -- if people actually have perceptions that it even makes sense to try to measure and analyze (they don't on things they haven't heard of, like nanotechnology, e.g.).

The affective logic of risk perception is what makes the industrial strength climate-change risk perception measure featured in this graphic so useful. Because ordinary peopole's answers to pretty much any question that they actually can understand will correlate very very strongly with their responses to this single item, administering the industrial-strength measure is a convenient way to collect data that can be reliably analyzed to assess sources of variance in the public's perceptions of climate change risks generally.

Indeed, if one asks a question the responses of which don't correlate with this item, then one is necessarily measuring something other than the generic affective orientation that informs (or just is) "public opinion" on climate change.  

Whatever it "literally" says or however a researcher might understand it (or suggest it be understood), an item that doesn't correlate with other valid indicators of the general risk orientation at issue is not a valid measure of it.

Consequently, any survey item administered to valid general public sample in today's America that doesn't generate the sort of partisan division reflected in this Figure is not "valid." Or in any case, it's necessarily measuring something different from what a large number of competent researchers, employing in a transparent and straightforward manner a battery of climate-change items that cohere with one another and correspond as one would expect to real-world phenomenon, have been measuring when they report (consistently, persistently) that there is partisan division on climate change risks.  

We'll know that partisan polarization is receding when the correlation between valid measures of political outlooks & like dispositions, on the one hand, and the set of validated indicators of climate change risk, on the other, abates. Or when a researcher collects data using a single validated indicator of a high-degree of discernment like the industrial strength measure and no longer observes the pretty-- and hideous-- picture displayed in the Figure above.

But if you don't want to wait for that to happen before declaring that the impasse has been broken-- well, then it's really quite easy to present "survey data" that make it seem like the "public" believes all kinds of things that it doesn't.  Because most people haven't ever heard of, much less formed views on, specific policy issues, the answers they give to specific questions on them will be noise.  So ask a bunch of questions that don't genuinely mean anything to the respondents and then report the random results on whichever ones seem to reflect the claim you'd like to make!

Bad pollsters do this. Good social scientists don't.

PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (20)

Yet another great graphic. The problem is that none are so blind as those who will not see.

I wonder what that graph would look like if the dependent axis variable were people's view on the degree of influence that a supernatural being has on evolution of the human species (low end = a supernatural being is 100% responsible; high end = a supernatural being has no involvement at all). My guess is that the patterns wouldn't be as distinct, but still fairly similar.

November 29, 2013 | Unregistered CommenterJoshua

@Joshua:

Thought experiments have a known type 1 error bias associated with their exclusive reliance on preconceived views as a data source.

But do you want to play "WSMD? JA!"?

There's no measure like the one you have in mind. But there are ones relating to beliefs in evolution, including whether independently of or under the direction of God. The items don't readily lend themselves to a continuous measure akin to Conserv_Repub, but if the point is just that you think belief in evolution will "explain" as much variance as Conserv_Repub, that's easily tested.

But why should we do the test? What's nature of the conjecture you are making? What is it an alternative to etc?

I don't think it is necessarily an alternative to the most plausible conjecture for why the graphic I put up shows the pattern that it does.

Basically, political outlooks are an indicator of a latent disposition associated with one or another cultural style characteristic of membership in the sorts of affinity groups in which people form their undertanding of what science knows.

So are cultural outlooks. So is religiosity, etc.

Belief in evolution is an indicator of such a style too. So we should expect any of these thigns to bear relationships similar to the one illustrated.

But how vividly, as it were, they display that pattern will depend on how discerning the characteristic in questin is as an indicator of the disposition.

I've discussed this point several times (I think 97 times, even), including just a few posts ago in connection with the religiosity/science comprehension interaction & also a few before that in connection with the tea party & climate change.

My general view is that "cultural outlooks" are best, although it would be even better if we could combine cultural outlooks, politics, religiosity, race, gender, etc. in a theoretically satisfying and psychometrically valid and tractable latent-variable measure.... I work on this about 14 hrs a day, actually, & only take breaks now again to post things to this blog or go running or listen to Feynman lectures on physics or watch old richard nixon speeches on youtube.

So ... We can do a WSMD? JA! here. But you have to say, in addition to how we might test your conjecture w/ data on hand, what the hypothesis is & why it is an interesting one in relation to whatever the dynamic is that accounts for the graphic that you only sort of like above.

November 29, 2013 | Unregistered Commenterdmk38

My conjecture is that like views on climate change, views on the role of a supernatural force in evolution are likely associated with cultural outlook (although not as clearly associated), and for that reason would also show up (albeit more indirectly) when graphed against the spectrum of liberal -->> conservative ideological spectrum.

So the reason for asking would be to gain more information about that conjecture. In case you were wondering, the reason is not because I believe it would help show that "Republicans hate science." My guess is that Republicans/conservatives and Democrats/liberals are equally likely to "hate" and "love" science - depending on the overlay of political ideology onto any given science-related issue.

If there were a pattern similar to what you show with climate change, what could be the explanation? Some might think degree of science-loving - but I don't think so. So then what would explain why religious Republicans might see a stronger supernatural hand in evolution than religious Democrats?

If a pattern appeared, would the results describe some difference in the religious believes of Repubs and Dems?

November 29, 2013 | Unregistered CommenterJoshua

Here's more conjecture - completely speculative and completely lacking any grounding in actual evidence.

I would bet that orthodox Jews would, on average, not tend to attribute the same roll of a supernatural force in evolution as would highly religious Christians. And the reason? It could be because of different doctrine w/r/t the role of the supernatural in creation of the universe. But I don't think that would pan out. Similarly, I think that Orthodox Jews would likely lean more liberal than highly religious Christians. But I don't think that the causality there is that political ideology drives views on evolution, but that political ideology is a mediator on the relationship between affinity group and reasoning in the face of uncertainty.

November 29, 2013 | Unregistered CommenterJoshua

Even more speculation that the roll of a supernatural force would be a dinner roll.

November 29, 2013 | Unregistered CommenterJoshua

People, scientists included, have a strong tendency to try to prove hypotheses by looking for confirming instances. While it can sometimes turn up surprises by showing that an expected effect isn't there, it's not very reliable or systematic.

The more scientific approach is to prove hypotheses by trying to disprove them, and failing. We have our theory, that cultural outlook causes climate risk perception, and we have our expected correlation. What *alternative* theories could explain the same observations? How do they differ in their predictions? Constructing the alternative hypotheses directs our attention to those critical tests most likely to reveal something new, or by not doing, to strengthen our confidence that our primary hypothesis is the best available.

If you can't think of any alternative hypotheses that can explain the observations you have so far, that gives you a bit more confidence (although not much, since an inability to think of alternatives to their favourite theory is endemic to humans). If you can think of lots of alternatives but no way of testing them, because the sort of observations that would distinguish them are not available or accessible, then your confidence ought to drop. You really don't know if it's true. And confirming instances don't help, since they're confirming instances of the alternatives, too.

So suppose we played this game, and found that the correlation of belief in divine creation with climate change risk perception was completely the *opposite* of what we expected. Suppose we saw that the most strongly YEC types were at the same time the most worried about climate change. What would it mean? What theories would we construct to try to explain it?

For example, do people think climate change is a sign of the end times? Are they more likely to believe it is a judgement on man's sinful ways, a punishment from on high? Are rising sea levels like Noah's flood?

Do people believe in climate change risk because they think the world is about to end? That modern civilisation is immoral and sinful - greed and pride leading us to pillage the Earth for our own decadent pleasures; burning down the forests, poisoning the streams and rivers, ripping up the ground, and then concreting it over with ugly cities full of grime and noise, poverty and suffering. Do millenarian cults arise from our alienation from modern society, our anger at the way it behaves making us want to smash it, and when we don't have the power, to make us invent and take pleasure in the idea of greater forces doing it for us?

Do you feel that the dirty, noisy, nature-raping, capitalist industrial system deserves to be smashed?

Creationism might then be a proxy for End Times millenarianism (since I doubt Dan asked a question about that), which could conceivably give rise to the same sort of anger at modern society as their more secular brothers on the other end of the spectrum. It's noticeable from the graphs that there are a few outliers - people at the liberal end with low risk perception, or the conservative end with high risk perception. What are their other characteristics, and do they give any other clues as to possible alternatives?

I personally think it's pretty unlikely that Creationists are especially Green, or especially likely to go for this particular end-of-the-world scenario. The Hand of God isn't obvious enough in it for their taste (and the theology doesn't fit, eiither). But the parallels between the modern belief in environmental catastrophe and past religious tales of apocalypses and divine punishments is striking and has been noted many times before, so I wouldn't rule it out that people with that tendency might think alike.

On the other hand, if you don't like that hypothesis, and you can't think of another one that would explain why Creationists would have a high risk perception, then it probably isn't going to be an informative or interesting test. :-)

http://www.e-n.org.uk/p-1129-Climate-change-and-the-Christian.htm

November 30, 2013 | Unregistered CommenterNiV

@NiV:

1. Best design is one geared to generate observations that will make one interesting, plausible hypothesis more or less likely than an alternative interesting, plausible hypothesis. The design is a good one if one can say with confidence that if we observe x, we have more reason than previously to believe H1 than H2; if we don't observe x, then we have more reason to believe H2 than H1.

2. This is alternative to classlic "null hypothesis testing," which is highly likely to be confounded (rejecting "null" is often as consistent with H2 as H1) and even not still far *too* likely to generate no net gain in information (even if "rejection of null" makes H1 "more likely" than before, it doesn't follow that "failure to reject null" makes it any less likely-- type 2 error is ignored by this testeig construct).

Pretty much all CCP studies aspire to be like 1 (e.g., study of CC thesis vs. SC thesis in our study of science comprehension, cultural cogniton & climate change risk perceptions). WSMD? JA! entries should do the same

November 30, 2013 | Registered CommenterDan Kahan

@Joshua:

The "dinner roll" item, while usually included in CCP subject-pool screening surveys, was accidentially left out of this data set. Darn.

But hep me out generally: what's the *dispositon" that motivates climate skepticism, on your view? The one I generally attribute it to in this blog, & on the basis of our studies? Are you saying just that you expect disbelief in evolution to predict climate change b/c that belief is an "indicator" of that disposition? Actually, I'm pretyy sure that I've already presented evidence on this blog that shows that there is healthy correlation between beief in evolution & beief in climate change; one would expect that if you are right. But this seems too easy, too predictable a thing to simply re-confirm.

But if you tell me why someone would think that there is a connection betwen beliefs in evolution & climate change risk that *differs* from the theory I've just given -- that belief/disbelief in evolution is an indicator of the same identity being measured (even more discerningly) by cultural outoooks--that wold be very interesting. Building on my comment to NiV, give me a test we can perform where that view is H2 and the one I've asserted H1; what would we expect to observe if H2 is right but not if H1 is, and not observe if in fact H1 is true.

NiV suggests that maybe beliefe in "divine control" over everything could see climate change risks as *high* but oppose efforts to resist it b/c we are being punished for being sinful etc. I can think of ways to test that. But actually, it is already a hyptothesis that is in trouble given what we kmnow about evolution beliefs & climate change risk perceptions: viz., that people who think "God is controlling evolution" or "God created everyitng as is" tend to be climate skpetics; they don't believe that the risks are high, so they presumably don't think we are being punished (they might see a particular extreme weather event that way, of course). In additoin, people who tend to believe in climate change *do* tend to believe that all manner of things could help to mitigate it. Indeed, they tend to believe that strengthing the Clean Air Act would do that -- it clearly wouldn't (sulfur emissions cool the atmosphere!). That's b/c they have a general affective orientation toward "being nice to the environment" that then shapes their answer to anything one asks them about climtae change. Again, though, that seems contrary to NiV's conjecture (which I don't think he meant to endorse --jjust to toiss out there as an example of a claim that one could test).

But probably too there are more fine-grained sets of outlooks that combine things that are indicators of climate change attitudes in ways that don't fit this view. They are part of the "noise" in the measures we are discussing. And of course one man or woman's "noise" is another's opportunity to improve our measurement apparatuses

November 30, 2013 | Unregistered Commenterdmk38

Dan -

But hep me out generally: what's the *dispositon" that motivates climate skepticism, on your view? The one I generally attribute it to in this blog, & on the basis of our studies?

Yes, pretty much. It seems to me that your attribution is well-backed by data, and I have seen no compelling alternative explanations. Still, I'm not quite sold on the confidence that you have in your matrix as a descriptor of "world views" -- as I think that "world views," like views on climate change or gun control, etc., mostly overlay on top of a dominant tribal orientation. I have seen situations where strongly held beliefs on questions such as government vs. individual freedom can flip 180 on pretty much the drop of a hat (e.g., the attitude of "conservatives" towards an individual mandate, the view of SCOTUS on states' rights). So my feeling is that world views, also, is an indicator of a more general disposition: a disposition that mostly comprises tribal orientation and an identity protection, with some pattern-seeking cognition mixed in for good measure.

But if you tell me why someone would think that there is a connection betwen beliefs in evolution & climate change risk that *differs* from the theory I've just given -- that belief/disbelief in evolution is an indicator of the same identity being measured (even more discerningly) by cultural outoooks--that wold be very interesting.


I agree with you. Clearly there are many, who would argue (from both sides) that your conjecture is not accurate - but not I.

December 1, 2013 | Unregistered CommenterJoshua

My eyes are drawn to the people who didn't get the tribal memo and are highly conservative/strong Republican yet assess high risk from global warming and the highly liberal/strong Democrat who assess low risk from global warming.

These are the people I would be interested in talking with.

Isabel

December 1, 2013 | Unregistered CommenterIsabel Penraeth

Isabel, then take a look at Geoff Chambers's blog,
http://geoffchambers.wordpress.com/2013/11/30/lives-of-the-climate-bloggers-3-robert-e-phelan/

Dan gets a mention too.

December 1, 2013 | Unregistered CommenterPaul Matthews

@Isabel:

Yes, you can *see* them too.

r = 0.65 isn't r = 1.0. And one person's statistical noise is another's opportunity to figure something out... So what do you think explains my "noise"? Tell me & maybe we can play WSMD? JA!

Actually, I have literally 10^3's of observations of these "cultural contrarians" in dozens of different studies. All I need are some ideas about ways to test hypotheses about why they are the way they are w/ other variables in those data sets

December 1, 2013 | Registered CommenterDan Kahan

Paul -

How do you see that link as following from Isabel's comment?

December 1, 2013 | Unregistered CommenterJoshua

Geoff is very much a lefty.

December 2, 2013 | Unregistered CommenterPaul Matthews

Paul - "pretty much a lefty" (even if it were true, which seems dubious from what I read), doesn't follow from Isabel's comment.

December 2, 2013 | Unregistered CommenterJoshua

Isabel said she would like to talk to lefty climate sceptics.
So I suggested Geoff, who is a sceptic and labour party supporter.
Not sure why you are having difficulty understanding.

December 2, 2013 | Unregistered CommenterPaul Matthews

Paul -

She said, "highly liberal/strong Democrat." You offered someone who is, perhaps, mildly liberal.

December 2, 2013 | Unregistered CommenterJoshua

I see the Republican and Democratic parties as coalitions of special interest groups who may have conflicts with each other, or may not even care about each other's agendas, but they "join" the party that is least destructive of their agenda. The parties are dedicated to the acquisition of political power, which is then shared by the groups. For example, economic libertarians have little in common with the religious right, except their opposition to a powerful centralized government, yet they both tend to vote Republican. These groups tend to give support to each other's agendas, almost unconsciously. I think a lot of the correlation between belief in a supernatural being and low-risk assesment of climate change is of this nature. There is also a need on the part of any "affinity group" to cast their opponents as monolithic in their beliefs, it makes passionate opposition so much easier. E.g. the belief in monolithic communism which walked us into the Viet Nam war. People on the "left" should be wary of trying to imagine the reasons for the correlation as coming from a coherent ideology that they project onto Republicans, just as people on the "right" must be wary of trying to imagine that a correlation between atheism and high-risk assessment of climate change is part of a coherent ideology of Democrats.

December 3, 2013 | Unregistered CommenterFrankL

For the political grid version of the graph, where did the "indiv-communitarian" and "egalitarian-hierarchy" axis data values come from? I've read the original post - can only see data on a liberal to conservative scale. Apologies if I've missed something.

December 9, 2013 | Unregistered CommenterDan Olner

@DanO:

Scales here: http://www.sjdm.org/dmidi/Cultural_Cognition_Worldview_Scales.html

Good discussion of how relate to right-left here: http://www.culturalcognition.net/blog/2012/6/21/politically-nonpartisan-folks-are-culturally-polarized-on-cl.html

December 9, 2013 | Registered CommenterDan Kahan

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>