follow CCP

Recent blog entries
popular papers

Science Curiosity and Political Information Processing

What Is the "Science of Science Communication"?

Climate-Science Communication and the Measurement Problem

Ideology, Motivated Cognition, and Cognitive Reflection: An Experimental Study

'Ideology' or 'Situation Sense'? An Experimental Investigation of Motivated Reasoning and Professional Judgment

A Risky Science Communication Environment for Vaccines

Motivated Numeracy and Enlightened Self-Government

Making Climate Science Communication Evidence-based—All the Way Down 

Neutral Principles, Motivated Cognition, and Some Problems for Constitutional Law 

Cultural Cognition of Scientific Consensus
 

The Tragedy of the Risk-Perception Commons: Science Literacy and Climate Change

"They Saw a Protest": Cognitive Illiberalism and the Speech-Conduct Distinction 

Geoengineering and the Science Communication Environment: a Cross-Cultural Experiment

Fixing the Communications Failure

Why We Are Poles Apart on Climate Change

The Cognitively Illiberal State 

Who Fears the HPV Vaccine, Who Doesn't, and Why? An Experimental Study

Cultural Cognition of the Risks and Benefits of Nanotechnology

Whose Eyes Are You Going to Believe? An Empirical Examination of Scott v. Harris

Cultural Cognition and Public Policy

Culture, Cognition, and Consent: Who Perceives What, and Why, in "Acquaintance Rape" Cases

Culture and Identity-Protective Cognition: Explaining the White Male Effect

Fear of Democracy: A Cultural Evaluation of Sunstein on Risk

Cultural Cognition as a Conception of the Cultural Theory of Risk

« Science curiosity research program | Main | Science literacy, science curiosity, and education »
Monday
Nov122018

Guest post: Some weird things in measuring belief in human-caused climate change

From an honest-to-god real expert--a guest post by Matt Motta, a post doctoral fellow associated with the Cultural Cognition Project and Annenberg Public Policy Center. Matt discusses his recent paper, An Experimental Examination of Measurement Disparities in Public Climate Change Beliefs.

 Do Americans Really Believe in Human-Caused Climate Change? 

Matt Motta (@matt_motta)

Image result for matt motta minnesotaDo most Americans believe that climate change is caused by human activities? And what should we make of recent reports (e.g., Van Boven & Sherman 2018) suggesting that self-identified Republicans largely believe in climate change?

Surprisingly, given the impressive amount of public opinion research focused on assessing public attitudes about climate change (see: Capstick et al., 2014 for an excellent review), the number of Americans (and especially Republicans) who believe that climate change is human caused actually a source of popular and academic disagreement.

For example, scholars at the Pew Research Center have found that less than half of all Americans, and less than a quarter of Republicans, believe that climate change is caused by human activity (Funk & Kennedy 2016). In contrast, a team of academic researchers recently penned an op-ed in the New York Times (Van Boven & Sherman 2018; based on Van Boven, Ehret, & Sherman 2018) suggesting that most Americans, and even most Republicans, believe in climate change – including the possibility that it is human caused.

In a working paper, my coauthors (Daniel Chapman, Dominik Stecula, Kathryn Haglin and Dan Kahan) and I offer a novel framework for making sense of why researchers disagree about the number of Americans (and especially Republicans) who believe in human caused climate change. We argue that commonplace and seemingly minor decisions scholars make when asking the public questions about anthropogenic climate change can have a major impact on the proportion of the public who appears to believe in it.

Specifically, we focus on three common methodological choices researchers must make when asking these questions. First, scholars must decide whether they want to offer “discrete choice” or Likert style response options. Discrete choice responses force respondents to choose between alternative stances; e.g., whether climate change is human caused, or caused by natural factors. Likert-style response formats instead ask respondents to assess their levels of agreement or disagreement with a particular argument; e.g., whether one agrees or disagrees that climate change is human caused.

Likert-style response can be subject to “acquiescence bias,” which occurs when respondents simply agree with statements, potentially to avoid thinking carefully about the question being asked. Discrete choice response formats can reduce acquiescence bias, but allow for less granularity in expressing opinions about an issue. Whereas the Pew Study mentioned earlier made use of discrete style response options, the aforementioned op-ed made use of Likert style responses (and found comparatively higher levels of belief in anthropogenic climate change).

Second, researchers must choose whether or not to offer a hard or soft “don’t know” (DK) response option. Hard DK options expressly give respondents the opportunity to report that they do not know how they feel about a certain question. Soft DK responses, on the other hand, allow respondents to skip a question, but do not expressly advertise their ability to not answer it.

Hard DKs have the benefit of giving those who truly have no opinion about a particular prompt to say so; rather than either guess randomly, or – especially when Likert style questions – simply agree with the prompt. However, expressly offering a DK option risks that respondents will simply indicate that they “don’t know” rather than engage more effortfully with the survey. Again drawing on the two examples described earlier, the comparatively pessimistic Pew study offered respondents a hard DK, whereas the work summarized in the New York Times op-ed did not.

Third, researchers have the ability to offer text that provides basic background information about complex concepts; including (potentially) anthropogenic climate change. This approach has the benefit of making sure that respondents have a common level of understanding about an issue, before answering questions about it. However, scholars must choose the words provided in these short “explainers” very carefully – as information presented there may influence how respondents interpret the question.

For example, the research summarized in the New York Times op-ed described climate change as being caused by “increasing concentrations of greenhouse gasses.” Although this text does not attribute greenhouse gas emissions to any particular human source, it is important to keep in mind that skeptics may see climate change as the result of factors having nothing to do with gas emissions (e.g., that the sun itself is responsible for increased temperatures). Consequently, this text could lead respondents toward providing an answer that better matches scientific consensus on anthropogenic climate change.

We test the impact of these three decisions on the measurement of anthropogenic climate change attitudes, in a large demographically online survey of American adults (N = 7,019). Respondents were randomly assigned to answer one of eight questions about their belief in anthropogenic climate change; each varying one of the methodological decisions described above, and holding all other factors constant.

The results are summarized in the figure below. Hollow circles are number of respondents in each condition who purport to believe in human-caused climate change, with 95% confidence intervals extending outward from each one. The left-hand pane plots these quantities for the full sample, and the right-hand pane does the same for just self-identified Republicans. The elements varied in each experimental condition are listed in the text just below the figure.

Generally, the results suggest that minor differences in how we ask questions about anthropogenic climate change can increase the number of Americans (especially Republicans) who appear to believe in it.  For example, Likert style response options (conditions 5–8) always produce higher estimates of the number of Americans and Republicans than discrete choice style questions (conditions 1–4).

At times, these differences are quite dramatic. For example, Condition 1 mimics the way Pew (i.e., Funk & Kennedy 2016) ask questions about anthropogenic climate change;  using discrete-choice questions that offer a hard DK option with no “explainer text.” This method  suggests that 50% of Americans, and just 29% of Republicans, believe that climate change is caused by human activities.

Condition 8, on the other hand, mimics method used in the piece reported in the aforementioned op-ed; featuring Likert-style response options, text that explains that climate change is caused by the greenhouse effect, and no explicit DK option. In sharp contrast, this method finds that 71% of Americans and 61% of Republicans believe that climate change is human caused. This means that the methods used in Condition 8 more than double the number of Republicans who appear to believe in human caused climate change.

We think that these results offer readers a useful framework for making sense  public opinion about anthropogenic climate change. Our research urges readers to pay careful attention to the way in which public opinion researchers ask questions about anthropogenic climate change, and to consider how those decisions might increase (or decrease) the number of Americans who appear to believe in anthropogenic climate change. Of course, we do not propose a single measurement strategy as a “gold standard” for assessing opinion about anthropogenic climate change. Instead, we hope that these results can readers be better consumers of public opinion about climate change.

References

Capstick, S., Whitmarsh, L., Poortinga, W., Pidgeon, N., & Upham, P. International trends in public perceptions of climate change over the past quarter century. Wiley Interdisciplinary Reviews: Climate Change , 6(1), 35-61. (2015).

Ehret, P. J., Van Boven, L., & Sherman, D. K. (2018). Partisan Barriers to Bipartisanship: Understanding Climate Policy Polarization. Social Psychological and Personality Science, 1948550618758709.

Funk, C., & Kennedy, B. The politics of climate. Pew Research Center. Retrieved from: http://www.pewinternet.org/2016/10/04/the-politics-of-climate/ (2016, Oct 4)

Van Boven, L. & Sherman D. Actually, Republicans Do Believe in Climate Change. New York Times  (2018, July 28)

Van Boven, L., Ehret, P. J., & Sherman, D. K. Psychological barriers to bipartisan public support for climate policy. Perspectives on Psychological Science , 13(4), 492-507. (2018).

 

PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (28)

Likert-style response can be subject to “acquiescence bias,” which occurs when respondents simply agree with statements, potentially to avoid thinking carefully about the question being asked.

But, the Likert scales used have as their center choice "Neither agree nor diagree". Why wouldn't acquiescent subjects answer with that instead of with a more positive response if they are really trying "to avoid thinking carefully about the question being asked"?

I'm not arguing that something isn't happening, just that "acquiescence bias" as described doesn't seem IMHO to be an adequate explanation.

November 12, 2018 | Unregistered CommenterJonathan

Matt -

This approach has the benefit of making sure that respondents have a common level of understanding about an issue, ...

It seems to me that text can specify context so respondents have a common level of understanding of the question, as much as provide background so prespondents have a common level of understanding about the issue.

In this case, it seems to me that specifying anthropgenic CO2 emissions, or even better, anthropogenic greenhouse gas emissions, is critical context for making responses meaningful.

However, scholars must choose the words provided in these short “explainers” very carefully – as information presented there may influence how respondents interpret the question.

Well sure, but again, are they "explainers" for background, or explainers for establishing the meaning of the question?

For example, the research summarized in the New York Times op-ed described climate change as being caused by “increasing concentrations of greenhouse gasses.”

It seems to me that to make responses meaningful, it is critical to establish how respondents view attribution for changes in the climate. Specifying concentrations of greenhouse emissions as a cause for changes, as distinguished from other causes, is critical. Questions that lump them together (e.g., "Is the climate warming") are rather useless unless they are combined with additional questions related to attribution (as Dan often does).

As far as concentrations of greenhouse gases as a cause of climate change is concerned, a bit more specificity would be useful, as there is a (rather small) group of people who think that greenhouse gas concentrations are increasing, but question the anthropogenic nature of those increases (Salby and his followers - and even Judith Curry rather infamously described Salby's work as "interesting.")

But here's where I get confused by your post:

Although this text does not attribute greenhouse gas emissions to any particular human source, it is important to keep in mind that skeptics may see climate change as the result of factors having nothing to do with gas emissions (e.g., that the sun itself is responsible for increased temperatures).

I don't see the match between your "although" clause, and the dependent clause that follows.

The source of greenhouse gas emissions is not directly relevant to the view among "skeptics" that it is "The sun what done it." Yes, many "skeptics" kind of vaguely hand-wave to "the sun" as a cause of recent, anomalous warming, but they generally don't bother to go into any detail related causal mechanism there (hence, IMO, a major problem with their arguments).

I think that there are probably very few "skeptics" who think that "the sun" causes an increase in concentration of greenhouse gases, and thus asking about greenhouse gas concentrations seems to me, again, to be a critical "explainer" for setting up the context of the question (although I think that attribution for the increased concentrations should probably be specified as well).

My point being, to be really useful, questions should, definitely, specify whether people feel that climate change is attributable to anthropogenic greenhouse gas emissions. A set of questions without that specification seem necessarily less comprehensive.

That is the way to measure belief in AGW. That isn't to say that it isn't also useful to measure whether people think that warming is taking place due to other causes (i.e, handwaves to "the sun" because "the climate has always changed"), and also whether people think that no warming is taking place: those are all useful measures.

November 12, 2018 | Unregistered CommenterJoshua

Are you are going to include similar graphs for Democrats similar to the ones you made for Republicans? Should be easy enough to add them and it would give the paper a more balanced appearance.

Also, It would be nice to know the sample sizes for Reps, Dems, and their leaners.

November 12, 2018 | Unregistered CommenterBob Koss

Joshua,

I agree, but the distinction between explainers vs. no explainers isn't nearly as dramatic as the distinction between Likert vs dichotomous.

But, if the responses with explainers are so wording sensitive, as suggested, then that can be tested as well with multiple different explainers.

The explainer they use is:

Global warming refers to the recent and ongoing rise in global average temperature near the Earth’s surface. Increasing concentrations of greenhouse gasses is the primary cause of global warming. Global warming, in turn, is causing climate patterns to change. Climate change includes major changes in temperature, precipitation, wind patterns, or other effects that occur over several decades or longer.

which, IMHO, as other issues. Note that the second and third statements are not "explainers" of semantics at all - they are instead declarations worded as scientific facts that would be in accordance with answering in a particular way (in favor of belief in anthropomorphic climate change). It could very well be the case that the slight negative bias introduced by these "explainers" is little more than a reaction against such "in your face" questioning.

November 12, 2018 | Unregistered CommenterJonathan

Jonathan -

How did you access Boven et al.?

November 12, 2018 | Unregistered CommenterJoshua

Joshua,

I wasn't referring to Van Boven et al. The explainer is in the supplemental material of Motta et al.

However, here's a link to Van Boven et al.

November 12, 2018 | Unregistered CommenterJonathan

W/r/t wording of questions, I note the following:

Which of these three statements about the Earth’s temperature comes closest to your view?

1) The Earth is getting warmer mostly because of human activity such as burning fossil fuels

2) The Earth is getting warmer mostly because of natural patterns in the Earth’s environment

3) There is no solid evidence that the Earth is getting warmer

The third question has a different syntactical structure than the other two. Why the change? IOW, why not word the first two in the following way?:

1) There is solid evidence that the earth is getting warmer mostly because of human...

And

2) There is solid evidence that the earth is getting warmer mostly because of natural...

....so as to be consistent with the 3rd question?

November 12, 2018 | Unregistered CommenterJoshua

I'll also note that, IMO (related to a point that has been mentioned elsewhere) :

Which of these three statements about the Earth’s temperature comes closest to your view?

1. The Earth is getting warmer mostly because of human activity such as burning fossil fuels

2. The Earth is getting warmer mostly because of natural patterns in the Earth’s environment

3. There is no solid evidence that the Earth is getting warmer

... might be a little problematic because it kind of sets up 1 and 2 to be distinguished by "mostly"

... which may be a bit of a subtle pivot point. Presumably, there are people who think there are both both anthropogenic and "natural" causes for recent warming, but who don't have a particularly strong opinion (or any opinion at all) about which is "mostly" causal and yet also think that there is solid evidence of warming.

How would such a person respond to that item?

November 12, 2018 | Unregistered CommenterJoshua

Jonathan -

Thanks. I gotta say, that explainer is pretty strangely leading.

November 12, 2018 | Unregistered CommenterJoshua

Joshua,

Just to avoid confusion - even though I got that explainer from Motta et al., it is the same as the one used in Van Boven et al., as shown in their supplemental materials:

http://journals.sagepub.com/doi/suppl/10.1177/1745691617748966

November 12, 2018 | Unregistered CommenterJonathan

Jonathan -

Thanks. I was wondering why that would be used in Motta et al. if it wasn't used in Boven et al., as it would set up a bit of a straw man if Motta had done that.

Either way, I think using that "explainer" resembles almost, in effect, including a primer to achieve a particular result (I'm not implying that intent)- not something I would expect to see in a straight up opinion poll.

November 12, 2018 | Unregistered CommenterJoshua

I didn't like the Van Boven et al paper, but not (just) because of the blatantly-leading explainer, nor the strangely controversial(?) use of a 5-point Likert scale to measure AGW agreement - it's how the second part of the paper tries to distill the subjective narratives of 4(!) politicos to uncover the reality behind the mysterious partisan death spiral. This second part of their paper was more appropriate for a NYT op-ed than a peer-reviewed psyc journal, but didn't even make the cut there as the NYT editors apparently had better judgement than the journal editors of Perspectives on Psychological Science.

Also, this from the NYT op-ed:

Fortunately, there is some cause for optimism. Our studies revealed a consistent, if somewhat surprising, pattern: Political disagreement was substantially smaller when it came to Republican-backed policies.

This seems to be referring specifically to the revenue-neutral carbon tax - and their data shows that Ds and Rs support it roughly equivalently when both knew it was R-backed. However, they neglected to analyze how the support would change once Rs got word that Ds supported it even though the Ds knew it was R-backed. Does the phrase individual mandate ring a bell?

November 12, 2018 | Unregistered CommenterJonathan

link drop:

https://digest.bps.org.uk/2018/11/13/people-are-consistently-inconsistent-in-how-they-reason-about-controversial-scientific-topics/

cannot find a non-paywall version.

November 13, 2018 | Unregistered CommenterJonathan

Prof. van Boven:
"....Correcting misperceived norms of opposition and decoupling policy evaluation from identity concerns would help overcome these seemingly insurmountable barriers to bipartisan support for climate policy."
---------------------------------------------------------------------------------------------------------------------------
It's often useful to move a "cultural" model to another field altogether, see how it fits. From today's NYTimes, article on cultural Marxism.
https://www.nytimes.com/2018/11/13/opinion/cultural-marxism-anti-semitism.html
Prof. Moyn tries to tie the alt-right to century-old allegedly antisemitic opposition to cultural Marxism using Breivik as an example. But Breivik never set out to kill any Jews; his stated - and undoubted - aim was to kill leftists.

Is the good professor suggesting the two groups are identical? Is he trying to convert readers to the original Marxism shorn of its "cultural" appendages? I've no idea and hope someone would articulate whatever unspoken assumptions underlie his article.

The same confusion is evident in the van Boven article. How about a test? Washington State (overwhelmingly Democratic) just held a referendum on a carbon tax. It was rejected 60-40. The idea a Republican majority would ever vote in favor of such a tax - no matter where it originated - is as absurd as implying Breivik (or the alt-right) is antisemitic. I don't personally object to such absurdities being bruited about, just wondering when the weight of contrary evidence will finally convince the true believers. My guess is, never.

November 13, 2018 | Unregistered CommenterEcoute Sauvage

P.S. On the off chance anyone here wants to read the book cited by prof. Moyn, William Lind's "Victoria", please note you don't have to buy it. My brothers and sisters on the alt-right believe in free speech, made freely available.

If anyone here knows prof. Moyn, please tell him that worrying about Breivik is whistling in the dark while walking through a graveyard. Particularly recommended is the concluding paragraph of the final chapter, #35.

https://www.traditionalright.com/victoria/

November 13, 2018 | Unregistered CommenterEcoute Sauvage

link drop:

https://phys.org/news/2018-11-social-relationships-important-hard-evidence.html

November 13, 2018 | Unregistered CommenterJonathan

Ecoute -

Considering your deep concerns about political correctness, freedom of speech, and censorship, what are your feelings about the Trump administration's pulling Acosta's press credentials?

Are you joining with the rest of the alt-right in a unified stance of support for Acosta?

November 13, 2018 | Unregistered CommenterJoshua

Don't forget the NRA's latest "stay in their lane" tweet. I'm confident that the alt-right's free speech brigade is all over that one.

November 13, 2018 | Unregistered CommenterJonathan

Jonathan -

I'm confident that the alt-right's free speech brigade is all over that one.


Take that to the bank.

After all remember the entire alt-right going to the mat en masse to defend Bron in that whole Ingraham "Shut up and dribble" thingy? My guess is that Ecoute reached for her firearm in case someone over at Fox News resisted with inappropriate force..

November 13, 2018 | Unregistered CommenterJoshua

Jonathan -

From that article:

By joining like-minded groups, individuals also prevent the psychological stress, or "cognitive dissonance," of considering opinions that do not match their own.

Does this joke only work for Jews?

https://www.youtube.com/watch?v=lDxE4r5ixD0

November 13, 2018 | Unregistered CommenterJoshua

You mean the Groucho Marx classic at the end of that vid? I guess that could explain why some found their way into the alt-right - although once there, then what?

BTW, I did recently see this paper at SSRN: https://papers.ssrn.com/abstract=3234745

However, it mostly reminded me of the Emo Phillips classic, which I regard as a very good refutation of the ultimate consequences of Haidt's binding moral foundations.

November 13, 2018 | Unregistered CommenterJonathan

Yes, the Marx joke.

November 13, 2018 | Unregistered CommenterJoshua

Ah yes. Pious ignoramus. Good thing I've never come across any of those!

November 13, 2018 | Unregistered CommenterJoshua

Jonathan -

However, they neglected to analyze how the support would change once Rs got word that Ds supported it even though the Ds knew it was R-backed.

As per the ballot measure that Ecoute referenced, not only whether demz back it becomes relevant in real world context. In the Oregon ballot measure, the FF industry spend 10s of millions to influence views in a carbon ta
(it would be interesting to see if any of their ads described the carbin tax as a commielubrulwackyleftymob plot), and of course there was that CO ballot measure for setbacks for cracking after some 41 spent by the FF industry to lobby against it (coincidence, I'm sure,)

So many of these research projects seem to be based in the notion that what they find in experimental models with controlled input variables can generalize to the zero sum scorched earth real world of ideological warfare. Needless to say, I'm generally dubious.

November 13, 2018 | Unregistered CommenterJoshua

"So many of these research projects seem to be based in the notion that what they find in experimental models with controlled input variables can generalize to the zero sum scorched earth real world of ideological warfare."

I sometimes imagine that electrons have about as much luck studying particle physics as humans do studying cognitive psych. They might even be just as negative about their prospects for success.

November 13, 2018 | Unregistered CommenterJonathan

Joshua,

"Are you joining with the rest of the alt-right in a unified stance of support for Acosta?"

https://www.foxnews.com/entertainment/fox-news-stands-with-cnn-passes-for-working-white-house-journalists-should-never-be-weaponized

If the speech impediment hits close enough to home...

November 14, 2018 | Unregistered CommenterJonathan

Saw that.

I can't say that I'm not surprised. I don't expect that kind of consistency. Of course, I would be even more surprised if Breitbart or other alt-righters support Acosta, also.

What do you say, Ecoute? Are you on board?

November 14, 2018 | Unregistered CommenterJoshua

The optional response that needs to be added in many cases is "We don't know." This is the position of many skeptics and it is very different from "I don't know."

Beyond that this article seems to suffer from the mistaken belief that the scientific debate has been settled in favor of the Consensus faction.

December 1, 2018 | Unregistered CommenterDavid Wojick

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>