follow CCP

Recent blog entries

"Messaging" scientific consensus: ruminations on the external validity of climate-science-communication studies, part 2

This is the second installment of a set on "external validity" problems in climate-science communication studies.

"Internal validity" refers to qualities of the design that support drawing inferences about what is happening in the study. "External vality" refers to qualities of the design that support drawing inferences from the study to the real-world dynamics it is supposed to be modeling.

The exernal validity problems I want to highlight don't affect only the quality of studies. They affect the quality of the practice of climate-science communication, too, because communicators are relying on externally invalid studies for guidance.

The last entry concerned the use of surveys to measure public opinion on climate change.

This one addresses experimental and other evidence used to ground "social marketing campaigns" that feature scientific consensus.  It is also only the first of two on "messaging" scientific consensus; the next, which I'll post "tomorrow," will examine real-world "messaging" that purports to implement these study findings.

This post, like the last, is from a paper that I'm working on and will post soon (one with some interesting new data, of course!)

* * *

5. “Messaging” scientific consensus

a. The “external validity” question. On May 16, 2013, the journal Environmental Research Letters published an article entitled “Quantifying the consensus on anthropogenic global warming in the scientific literature.” In it, the authors reported that they had reviewed the abstracts of 12,000 articles published in peer-reviewed science journals between 1991 and 2011 and found that “among abstracts expressing a position on AGW, 97.1% endorsed the consensus position that humans are causing global warming” (Cook et al. 2013).

“This is significant,” the lead author was quoted as saying in a press statement issued by his university, “because when people understand that scientists agree on global warming, they’re more likely to support policies that take action on it.” “Making the results of our paper more widely-known,” he continued, “is an important step toward closing the consensus gap”—between scientists who agree with one another about global warming and ordinary citizens who don’t—“and increasing public support for meaningful climate action” (Univ. Queensland 2013).

The proposition that disseminating the results of ERL study would reduce public conflict over climate change was an empirical claim not itself tested by the authors of the ERL paper.  What sorts of evidence might one use (or have used) to assess it?

Opinion surveys are certainly relevant.  They show, to start, that members of the U.S. general public— Republican and Democrat, religious and nonreligious, white and black, rich and poor—express strongly pro-science attitudes and hold scientists in high regard (National Science Foundation 2014, ch. 7; Pew 2009). In addition, no recognizable cultural or political group of consequence in American political life professes to disagree with, or otherwise dismiss the significance of, what scientists have to say about policy-relevant facts. On the contrary, on myriad disputed policy issues—from the safety of nuclear power  to the effectiveness of gun control—members of the public in the U.S. (and other liberal democratic nations, too) indicate that the position that predominates in their political or cultural group is the one consistent with scientific consensus (Kahan, Jenkins-Smith & Braman 2011; Lewendowsky, Gignac & Vaugh 2012).

Same thing for climate change. As the ERL authors noted, surveys show a substantial proportion of the U.S. general public rejects the proposition that there is “scientific consensus” on the existence and causes of climate change. Indeed, the proportion that believes there is no such consensus consists of exactly the same proportion that says it does not “believe in” human-caused global warming (Kahan et al. 2011).

So, the logic goes, all one has to do is correct the misimpression of that portion of the public. Members of the public very sensibly treat as the best available evidence what science understands to be the best available evidence on facts of policy significance. Thus, “when people understand that scientists agree on global warming, they’re more likely to support policies that take action on it” (Univ. Queensland 2013).

But there is still more evidence, of a type that any conscientious adviser to climate-science communicators would want them to consider carefully. That evidence bears directly on the public-opinion impact of “[m]aking the results” of studies like the ERL one “more widely-known” (Univ. Queensland 2013).

The ERL study was not the first one to “[q]uantify[]the consensus on anthropogenic global warming”; it was at least the sixth, the first one of which was published in Science in 2004 (Oreskes 2004; Lichter 2008; Doran & Zimmerman 2009; Anderegg et al. 2010; Powell 2012).  Appearing on average once every 18 months thereafter, these studies, using a variety of methodologies, all reached conclusions equivalent to the one reported in ERL paper.

Like the ERL paper, moreover, each of these earlier studies was accompanied by a high degree of media attention. 

Indeed, the “scientific consensus” message figured prominently in the $300 million social marketing campaign by Alliance for Climate Protection, the advocacy group headed by former Vice President Al Gore, whose “Inconvenient Truth” documentary film and book both prominently featured the 2004 “97% consensus” study published in Science (which was characterized by Gore as finding that "0%" of peer-reviewed climate science articles disputed the human contribution to global warming). 

An electronic search of major news sources indicates finds over 6,000 references to “scientific consensus” and “global warming” or “climate change” in the period from 2005 to May 1, 2013.

There is thus a straightfroward way to assess the prediction that “[m]aking the results” of the ERL study “more widely-known” can be expected to influence public opinion.  It is to examine how opinion varied in relation to efforts to publicize these earlier “scientific consensus” studies. 

Figure 9 plots the proportion of the U.S. general public who selected “human activities” as opposed to “natural changes in the environment” as the main cause of “increases in the Earth’s temperature over the last century” over the period 2003 to 2013 (in this Gallup item, there is no option to indicate rejection of the premise that the earth’s temperature has increased, a position a majority or near majority of Republicans tend to selection when it is available). The year in which “scientific consensus” studies appeared is indicated on the x-axis, as is the year in which “Inconvenient Truth” was released.   

Nothing happened.

Or, in truth, a lot happened.  Many additional important scientific studies corroborating human-caused global warming were published during this time.  Many syntheses of the data were issued by high-profile institutions in the scientific community, including the U.S. National Academy of Sciences, the Royal Society, and the IPCC, all of which concluded that human activity is heating the planet. High-profile, and massively funded campaigns to dispute and discredit these sources were conducted too.  People endured devastating heat waves, wild fires, and hurricanes, punctuated by long periods of weather normality.  The Boston Red Sox won their first World Series title in over eight decades.

It would surely be impossible to disentangle all of these and myriad other potential influences on U.S. public opinion on global warming.  But one doesn’t need to do that to see that whatever the earlier scientific-consensus "messaging" campaigns added did not “clos[e] the consensus gap” (Univ. Queensland 2013). 

Why, then, would any reflective, realistic person counsel communicators to spend millions of dollars to repeat exactly that sort of “messaging” campaign? 

The answer could be laboratory studies. One (Lewendowsky et al. 2012), published in Nature Climate Change, reported that the mean level of agreement with the proposition “CO2 emissions cause climate change” was higher among subjects exposed to a “97% scientific consensus” message than among subjects in a control condition (4.4 vs. 4.0 on a 5-point Likert scale).  After being advised that “97% of scientists” accept  CO2 emissions increase global temperatures, those subjects also formed a higher estimate of the proportion of scientists who believe that (88% vs. 67%).

Is it possible to reconcile this result with the real-world data on the failure of previous “scientific consensus” messaging campaigns to influence U.S. public opinion?  The most straightforward explanation would be that the NCC experiment was not externally valid—i.e., it didn’t realistically model the real-world dynamics of opinion-formation relevant to the climate change dispute. 

The problem is not the sample (90 individuals interviewed face-to-face in Perth, Australia). If researchers were to replicate this result using a U.S. general population sample, the inference of external invalidity would be exactly the same. 

For “97% consensus” messaging experiments to justify a social marketing campaign featuring studies like the ERL one, it would have to be reasonable to believe that what investigators are observing in laboratory conditions—ones created specifically for the purpose of measuring opinion—tell us what is likely to happen when communicators emphasize the “97% consensus” message in the real world. 

Such a strategy has already been tried in the real world.  It didn’t work.

There are, to be sure, many more things going on in the world, including counter-messaging,  than are going on in a “97% consensus” messaging experiment.  But if those additional things account for the difference in the results, then that is exactly why that form experiment must be regarded as externally invalid: it is omitting real-world dynamics that we have reason to believe, based on real-world evidence, actually matter in the real world.

On this account, the question to be investigated is not whether a “97% consensus” messaging campaign will influence public opinion but why it hasn’t over a 10-year trial.  The answer, presumably, is not that members of the public are divided on whether they should give weight to the conclusions scientists have reached in studying risks and other policy relevant facts. Those on both sides of the climate change believe that the other side’s position is the one in consistent with scientific consensus. 

The ERL authors’ own recommendation to publicize their study results presupposes public consensus in the U.S. in support of using the best available scientific evidence in policymaking.  The advice of those who continue to champion “97% consensus” social marketing campaigns does, too. 

So why have all the previous highly funded efforts to make “people understand that scientists agree on global warming” so manifestly failed to “close the consensus gap” (Univ. Queensland 2013)?

There are studies that seek to answer exactly that question as well.  They find that culturally biased assimilation—the tendency of people to fit their perceptions of disputed facts to ones that predominate in their cultural group—applies to their assessment of evidence of scientific consensus just as it does to their assessment of all other manner of evidence relating to climate change (Corner, Whitmarsh & Dimitrios 2012; Kahan et al. 2011). 

When people are shown evidence relating to what scientists believe about a culturally disputed policy-relevant fact (e.g., is the earth heating up? is it safe to store nuclear wastes deep underground? does allowing people to carry hand guns in public increase the risk of crime—or decrease it?), they selectively credit or dismiss that evidence depending on whether it is consistent with or inconsistent with their cultural group’s position. As a result, they form polarized perceptions of scientific consensus even when they rely on the same sources of evidence.

These studies imply misinformation is not a decisive source of public controversy over climate change.  People in these studies are misinforming themselves by opportunistically adjusting the weight they give to evidence based on what they are already committed to believing.  This form of motivated reasoning occurs, this work suggests, not just in the climate change debate but in numerous others in which these same cultural groups trade places being out of line with the National Academy of Sciences’ assessments of what “expert consensus” is.

To accept that this dynamic explains persistent public disagreement over scientific consensus on climate change, one has to be confident that these experimental studies are externally valid.  Real world communicators should definitely think carefully about that.  But because these experiments are testing alternative explanations for something we clearly observe in the real world (deep public division on climate change), they don’t suffer from the obvious defects of studies that predict we should already live in world we don’t see.

Part 3


Anderegg, W.R., Prall, J.W., Harold, J. & Schneider, S.H. Expert credibility in climate change. Proceedings of the National Academy of Sciences 107, 12107-12109 (2010).

Cook, J., Nuccitelli, D., Green, S.A., Richardson, M., Winkler, B., Painting, R., Way, R., Jacobs, P. & Skuce, A. Quantifying the consensus on anthropogenic global warming in the scientific literature. Environmental Research Letters 8, 024024 (2013).

Corner, A., Whitmarsh, L. & Xenias, D. Uncertainty, scepticism and attitudes towards climate change: biased assimilation and attitude polarisation. Climatic Change 114, 463-478 (2012).

Doran, P.T. & Zimmerman, M.K. Examining the Scientific Consensus on Climate Change. Eos, Transactions American Geophysical Union 90, 22-23 (2009).

Farnsworth, S.J. & Lichter, S.R. Scientific assessments of climate change information in news and entertainment media. Science Communication 34, 435-459 (2012).

Kahan, D.M., Jenkins-Smith, H. & Braman, D. Cultural Cognition of Scientific Consensus. J. Risk Res. 14, 147-174 (2011).

Lewandowsky, S., Gignac, G.E. & Vaughan, S. The pivotal role of perceived scientific consensus in acceptance of science. Nature Climate Change 3, 399-404 (2012).

Lichter, S. Robert. Climate Scientists Agree on Warming, Disagree on Dangers, and Don't Trust the Media's Coverage of Climate Change. Statistical Assessment Service, George Mason University (2008).

National Science Foundation. Science and Engineering Indicators (Wash. D.C. 2014), available at

Oreskes, N. The scientific consensus on climate change. Science 306, 1686-1686 (2004).

Pew Research Center for the People & the Press. Public praises science; scientists fault public, media (Pew Research Center, Washington D.C., 2009).

Powell, J. Why Climate Deniers Have No Scientific Credibility - In One Pie Chart. (2012).

Univ. Queensland. Study shows scientists agree humans cause global-warming (2013). Available at


External validity of climate-science-communication studies: ruminations part 1

The following is an excerpt from a paper I'm writing.  One of the paper's central themes is external validity.

Roughly, "internal validity" refers to the quality of a design that warrants drawing inferences from the results to what is going on in the study. "External validity" refers to the quality of a design that warrants drawing infernces from the results of a study to the real-world phenomenon the study is supposed to be engaging or modeling.  

I'm convinced that the study and practice of climate-science communication both reflect insufficient attention to external validity issues, and that this disregard is significantly dissipating the effectiveness of--wasting the resources committed to--communicating climate science. 

I'll post the paper in the near future. It has some cool data in it!

But in the meantime, I'll post a few bits -- somewhere between 2 and 17-- as blog posts.

* * *

3. What does “belief in” global warming measure?

Just as we can use empirical methods to determine that “belief in evolution” measures “who one is” rather than “what one knows,” so we can use these methods to assess what “belief in global warming” measures. An illuminating way to start is by seeing what a valid measure of “belief in global warming” looks like.

Figure 3 presents a scatter plot of the responses to a survey item that asked respondents (1800 members of a nationally representative sample) to rate “how much risk … global warming poses to human health, safety, or prosperity” in “our society.” The item, which I’ll call the “Industrial Strength Measure” (ISM), used an eight-point response scale, running form “none at all” to “extremely high risk,” with each point in between assigned a descriptive label.  The survey participants are arrayed along the y-axis in relation to their score on “Left_Right,” a reliable (α = 0.78) composite scale formed by aggregating their responses to a seven-point “party self-identification” measure (“Strong Republican” to “Strong Democrat”) and a five-point “ideology” one (“Very liberal” to “Very conservative). The color-coding of the observations—orange to red for higher risk ratings, yellow for middling ones, and green to blue for lower ones—helps to reveal the strength of the correlation between the global-warming risk ISM and left-right political outlooks.

Exactly how “strong,” though, is that correlation?  An “r” of  “- 0.65” might intuitively seem pretty big, but determining its practical significance requires a meaningful benchmark.  As it turns out, subjects’ responses to the party self-identification and liberal-conservative ideology items are correlated to almost exactly the same degree (r = 0.64, p <  0.01). So in this nationally representative sample, perceptions of the risk of global warming are as strongly associated with respondents’ right-left political outlooks as the indicators of their political outlooks are with one another. 

We could thus combine the global-warming ISM with the party-identification and liberal-conservative ideology items to create an even more reliable political outlook scale (α = 0.81), one with which we could predict with even greater accuracy people's positions on issues like Obamacare and Roe v. Wade.  From a psychometric perspective, all three of these items are measuring the same thing—a latent (unobserved) disposition that causes different groups of people to adopt coherent sets of opposing stances on political matters (DeVellis 2012).

The global-warming ISM has another interesting property, one it shares with ISMs for other putative hazards: it coheres very strongly with professed beliefs about the facts relevant to assessing the specified risk source (Dohmen eta al. 2011; Ganzach et al. 2008; Weber et al. 2002). “Cronbach’s α” is a conventional measure of scale reliability that ranges from 0.0 to 1.0; a score of 0.70 is generally regarded as signifying that a set of indicators  display the requisite degree of intercorrelation necessary to qualify as measure of some underlying latent variable. When global-warming ISM is combined with items measuring whether people believe that “average global temperatures are increasing,” that “[h]human activity is causing global temperatures to rise,” and that global warming will result in various “bad consequences for human beings” if not “counteracted,” the resulting scale has a Cronbach’s α of 0.95.   These “belief” items, then, can also be viewed as measuring the same thing as the “risk seriousness” item—viz., a latent disposition to form coherent sets of beliefs about the facts and consequences of the climate change.

Not surprisingly—indeed, as a matter of simple logic—there is a comparably high degree of coherence between “belief in climate change” and political outlooks. In this sample, some 75% of the individuals whose scores placed them to the “left” of the mean on the political outlook scale indicated that they believe human activity is the primary source of global warming. Only 22% of those who scores placed them to the “right” of the mean indicated that they believed that, and 58% of them indicated that they did not believe there was “solid evidence that the average temperature on earth has been getting warmer over the past few decades.” These figures are in accord with ones consistently reported by scholars and public opinion research centers for over a decade.

Nevertheless, advocacy groups regularly report polls that paint a very different picture. “A new study,” their press releases announce, show that “an overwhelming majority of Americans”—“Blue State and Red ones alike,” enough “to swing” an upcoming presidential election etc.— “support taking action” immediately to combat global warming. Disturbingly, the producers of such polls do not always release information about the survey’s wording or the (mysteriously characterized) methods used to analyze them. But when they do, informed observers point out that the questions posed were likely to confuse, mislead, or herd the survey respondents toward desired answers (Kohut 2010). 

Given the source of these surveys, one could infer that they reflect an advocacy strategy aimed at fostering “overwhelming majority support” for “action on climate change” by insisting that such support already exists. If so, the continued generation of these surveys itself displays determined inattention to over a decade’s worth of real-world evidence showing that advocacy polls of this sort have failed to dissipate the deep partisan conflict measured by various straightforward items relating to global warming.

Indeed, that is the key point: items that show “an overwhelming majority of Americans” believe or support one thing or another relating to climate change are necessarily not measuring the same thing as items that cohere with ISM. The question, then, is simply which items—ones that cohere with one another and ISM and that attest to polarization over climate change, or ones that do not cohere with anything in particular and that report a deep bipartisan consensus in favor of “taking action”—are more meaningfully tracking the real-world phenomena of interest. Unless one is prepared to conclude that the latent or unobserved disposition that causes coherent responses to political outlook and various global warming “belief” and risk perception items are irrelevant for making sense of the public opinion over climate change in the United States, it follows that survey questions that do not cohere with those ones are.

Serious opinion scholars know that when public-policy survey items are administered to a general population sample, it is a mistake to treat the responses as valid and reliable measures of the particular positions or arguments those items express.  One can never be sure that an item is being understood as one intended. In addition, if, as is so for most concrete policy issues, the items relate to an issue that members of the general population have not heard of or formed opinions on, then the responses are not modeling anything that people in the general population are thinking in their everyday world; rather they are modeling only how such people would respond in the strange, artificial environment they are transported into when a pollster asks them to express positions not meaningfully connected to their lives (Bishop 2005; Shuman 1998).

Of course many public policy issues are ones on which people have reflected and adopted stances of meaning and consequence to them.  But even in that case, responses to survey items relating to those issues are not equivalent to statements or arguments being asserted by a participant in political debate.  The items were drafted by someone else and thrust in front of the survey participants; their responses consist of manifestations of a pro- or con- attitude, registered on a coarse, contrived metric.

Because the response to any particular item is at best only a noisy indicator of that attitude, the appropriate way to confirm that an item is genuinely measuring anything, and to draw inferences about what that is, is to show that responses to it cohere with other things (responses to other items, behavior, performance on objective tests, and so forth) the meaning of which is already reasonably understood. Whatever thatitem does measure, moreover, can be measured more precisely when that item is appropriately combined into a scale with others that measure that same thng (Bishop 2005; Zaller 1992; Berinsky & Druckman 2007; Gliem & Gliem 2003).

The striking convergence of items measuring perceptions of global warming risk and like facts, on the one hand, and ones measuring political outlooks, on the other, suggests they are all indicators of a single latent variable.  The established status of political outlooks as indicators of cultural identity supports the inference that that is exactly what that latent variable is. Indeed, the inference can be made even stronger by replacing or fortifying political outlooks with even more discerning cultural identity indicators, such as cultural worldviews and their interaction with demographic characteristics such as race and gender: the resulting latent measures of identity will be even more strongly correlated with climate change risk perceptions and related attitudes (McCright & Dunlap 2012; Kahan et al. 2012).  In sum, whether people “believe in” climate change, like whether they “believe in” evolution, expresses who they are

Part 2

Part 3


Berinsky, A.J. & Druckman, J.N. The Polls—Review: Public Opinion Research and Support for the Iraq War. Public Opin Quart 71, 126-141 (2007).

Bishop, G.F. The Illusion of Public Opinion : Fact and Artifact in American Public Opinion Polls (Rowman & Littlefield, Lanham, MD, 2005).

DeVellis, R.F. Scale development : theory and applications (SAGE, Thousand Oaks, Calif., 2012).

Dohmen, T., Falk, A., Huffman, D., Sunde, U., Schupp, J. & Wagner, G.G. Individual risk attitudes: Measurement, determinants, and behavioral consequences. Journal of the European Economic Association 9, 522-550 (2011).

Ganzach, Y., Ellis, S., Pazy, A. & Ricci-Siag, T. On the perception and operationalization of risk perception. Judgment and Decision Making 3, 317-324 (2008).

Gliem, J.A. & Gliem, R.R. Calculating, interpreting, and reporting Cronbach’s alpha reliability coefficient for Likert-type scales. (Midwest Research-to-Practice Conference in Adult, Continuing, and Community Education, The Ohio State University, Columbus, OH, 2003). Available at

Kahan, D.M., Peters, E., Wittlin, M., Slovic, P., Ouellette, L.L., Braman, D. & Mandel, G. The polarizing impact of science literacy and numeracy on perceived climate change risks. Nature Climate Change 2, 732-735 (2012).

Kohut, A. Views on climate change: What the polls show. N.Y. Times A22 (June 13, 2010), available at 

McCright, A.M. & Dunlap, R.E. Bringing ideology in: the conservative white male effect on worry about environmental problems in the USA. J Risk Res, 1-16 (2012).
Shuman, H. Interpreting the Poll Results Better. Public Perspective 1, 87-88 (1998).

Weber, E.U., Blais, A.-R. & Betz, N.E. A Domain-specific Risk-attitude Scale: Measuring Risk Perceptions and Risk Behaviors. Journal of Behavioral Decision Making 15, 263-290 (2002).

Zaller, J.R. The Nature and Origins of Mass Opinion (Cambridge Univ. Press, Cambridge, England, 1992).


"Resolved: Climate change is not a 'crisis'": Using cultural cognition research in high school ecology class

[from Dan Kahan: The following is a guest post on a super important topic: teaching secondary-school students climate science in a polluted science communication environment. In today's society, opposing stances on climate change have taken on the character of badges of membership in, and loyalty to, competing cultural groups. No one should have to choose between knowing what's known to science and being who they are; certainly kids can't be expected to learn effectively when put in that position. But talented, dedicated science educators have faced the challenge of dispelling this conflict and have overcome it in other settings. It won't be easy to do here, but I'm confident they'll succeed--and that all of us will learn something in the process about how to disentangle the toxic knot between cultural identity and positions on climate change.  Read this--a report form a reflective and passionate science educator on his encounter with this dilemma--& you'll see why I'm so optimistic!]

By Peter Buckland

What do you do when you get an email from a parent who’s worried your teaching climate alarmism?

Peter Buckland, displaying the sense of wonder that he is dedicated to enabling his students to experienceIn my second year as Director of Sustainability at Kiski, I was tasked with teaching two sections of Ecology. I designed the course to merge ecological concept mastery, major human-environmental issues, a campus arboretum and organic gardens, and opportunities for reflection.  Given the world as it is, I had to do a fairly in-depth unit on the science of the climate and climate change.

When I was hired, I told my interviewers that I was likely to encounter some resistance to scientifically-based climate education. The national politics and the personal convictions of a sizable swath of conservative Americans and their vociferousness indicated we’d get a phone call, email, or some grousing. At Kiski, many of my students come from white middle- to upper-class conservative families whose political alliances virtually guarantee they will doubt anthropogenic global warming or outright deny it as liberal garbage. I knew my audience and the potential resistance and I also knew I had administrators and a science department chair who backed me up.

I focused on the scientific consensus and how it has been achieved. We did labs and activities using radiative forcing data from NOAA and historical regional land and ocean temperature maps from the Australian Bureau of Meteorology. They read the most recent IPCC AR 5 “Headline Statement” and other current materials. One section got to Skype with Dr. Michael E. Mann, Director of Penn State’s Earth Systems Science Center and author of The Hockey Stick and the Climate Wars who spoke about current findings in climate science and not much on politics or lawsuits. All in all, the unit was shaping up well.

Then, at the end of the unit, I received an email from a concerned parent. He was concerned that I was being imbalanced in my teaching and courting some kind of climate alarmism. As a geologist, he had done some personal research and discerned that our climate was changing, that there was some anthropogenic forcing, but that climate change was not as bad as some people were making it out to be and that it certainly wasn’t a catastrophe. He offered to come to my class to balance the scales with a presentation of his own.

I admit, I was initially insulted and started a keyboard barrage. Yes! Bludgeon him with scientific data, authoritative scientific organizations, and self-righteous ire. After a few minutes though, I realized my strategy would backfire. My awareness of research about motivated reasoning and identity protection overrode my impulses. The emailing parent seemed in the “Doubtful” or maybe “Dismissive” camp of the Yale Six Americas study. Working from Lewandowsky’s and Cook’s The Debunking Handbook, I knew I should avoid emphasizing falsehoods, prevent an overkill backfire from a barrage of information (so hard), and do what I can to stop a worldview backfire, Dan Kahan’s focus at Cultural Cognition. Caution was in order. This was an opportunity.

When I wrote back I thanked the parent for being interested in his son’s education, his interest in the topic, and then explained my course’s logic. First, I work to represent current science accurately. Second, I am not an arbiter of my students’ values. While I am pretty alarmed about climate change’s scale and pace, it is not my place to indoctrinate my students into a political or emotional faction but to invite them to reflect on the state of the world and their own lives and values (how American of me). Third, and most importantly, I would provide them with the opportunity to develop their own views on the matter by taking positions in a mock UNFCCC deliberation where they could determine their thresholds for risk or whether or not climate change is a catastrophe. I would end up changing that format, though, because I wanted to get around my students’ motivated reasoning and develop their scientific literacy, their moral literacy, and their communication and analytical skills.

I split my classes into small groups and had them take positions on the following proposition: “Climate change is not a crisis.” We followed the format of Intelligence Squared that airs on National Public Radio. I instructed my students’ to incorporate well-grounded scientific information that reflects current understanding and make a clear argument of definition about what does or does not constitute a crisis. I chose this format from among three options for a few strategic reasons. First, the debate was not “Climate change is real: yes or no.” We could not deal in disproven or junk science. Second, by debating the proposition in the negative – “not a crisis” – I avoided an alarmist’s position. Third, equal numbers of students would have to be on one side or the other in groups I created deliberately.  This way I could put very concerned or alarmed students for the proposition and dismissive or doubtful students against it, thereby inviting them to reason in ways that could counter their own motivated reasoning. Fourth, at no point did I tell my students to be “objective,” “unbiased,” “rational,” or “open-minded.” While I might like them to do that, entreating them to be so could backfire as research has indicated it does. We like to believe we are open-minded and those people over there are the close-minded unrealistic ones. My set-up could avert some of that problem.

This model has at least two potential flaws. Someone could accuse me of a certain kind of censorship or choice editing. I have to edit my students’ choices in class. All teachers are, to some extent, editors. To master trophic relationships in the soil I would not tell my students that it is okay to entertain the notion that the soil food web is a hoax because the first law of thermodynamics is wrong. Similarly, any responsible understanding of the science of climate change at this point will not entertain hoax arguments.

A second problem may be more insidious. Simply inviting group deliberation could entrench people even further. At Cultural Cognition, Dan Kahan writes, “Far from counteracting this effect, deliberation among diverse groups is likely to accentuate polarization.  By revealing the correlation between one or another position and one or another cultural style, public debate intensifies identity-protective pressure on individuals to conform to the views dominant within their group.” Earlier I wrote that I had initially conceived of this project using a mock UNFCCC framework. I decided not to use that format to skirt worldview-threatening messages. As Kahan et al show in some of their research, hierarchical individualists (who map fairly well onto American conservatives) doubt climate change science more if it’s couched in terms of carbon regulations. Because the UNFCCC deliberations focus so heavily on regulation of markets and perceived United Nations’ interference, I tried to dodge that landmine.

How did they debate? They crafted arguments around what constitutes a crisis. One group that agreed with the proposition – climate change is not a crisis – said that on the scale of crises there is only so much room for big crises; global poverty, AIDS, and wars held up that space. Climate change may be a problem but it is so slow that right now it is not urgent. But another group argued it is a crisis because of the enormous costs to disaster-prone areas, the cost to insure them, and the costs to reinsurance. Using MunichRe and SwissRe as sources they made a powerful argument against the proposition. Yet another used techno-optimism a la Ray Kurzweil to predict that solar power will eclipse fossil fuels in the next couple of decades, thereby eliminating the largest emissions sources. But those against the proposition showed that threats to the carbon cycle were so severe already that major disruptions in ecosystem services and extinction were real, present, and harming people. Sadly, one group did not follow directions and used thoroughly debunked and scientifically invalid arguments. A teacher can’t control everything.

The emailing parent’s son was a star too. He was placed on a team that argued that climate change is a crisis and he spoke for the group (each group had a designated speaker). During the question and answer session that followed each teams’ statement, he answered questions clearly and asked his opponents intelligent and pointed questions. At one point, after pointing out the denier groups’ inaccuracies, their leader looked red in the face and asked, Do you think it’s a crisis?

He said something to the effect of, No. But that’s not the point. I’m arguing a position and doing it the best I can. But there are facts and we shouldn’t be afraid of facts. I don’t have to think this is a crisis to believe it’s real.

I felt pretty satisfied as a teacher at that moment. He and his group had formulated a scientifically-informed and sensible argument with which he did not agree. And in so doing, he showed that he could master scientific information he might have rejected were it presented in a way that it would have threatened his and his family’s worldview.

Just last week, he graduated and I had the chance to talk to his dad, the emailing parent. He and his son had talked about the debate. I told him I had created it in part because of his email. And he was pleased with that and the experience it gave his son. He agreed that whatever we might think climate change’s status as crisis or not, that we should all master sound information and concepts and learn to place them into coherent messages. It was, once again, very satisfying to have some evidence that developing the assignment using my understanding of identity protection and motivated reasoning had worked with at least one student.

It seems to me that I might have made some educational errors had I not known and thoughtfully strategized from the cultural cognition and related research. I encourage others to craft similar strategies to develop their students and the public’s/publics’ climate literacy. By attending to who we are, we have better chances at dealing with reality together. With the world’s climate changing as rapidly as it is, we need to use the best strategies we can so that we can be better ecological citizens.

Peter Buckland is completing his two-year term as Director of Sustainability at the Kiski School. He has worked on energy, waste, land, and educational projects ranging from gardens and an arboretum to a comprehensive energy strategy and teacher development. He sees his purpose as making possibilities for all people to become better ecological citizens, people who “recognizes the importance and interconnectivity of all living beings, human and non-human…[who] understands that she or he is responsible to all beings and actively seeks sustainable futures for them” (Kissling and Barton, 2013). He is finishing his doctorate in Educational Theory and Policy at Penn State University.


Got facts? The boring, ignorant, anti-liberal, science-communication-environment polluting "who is more anti-science" game

I haven’t really been paying that much attention but I gather that some attention-seeking talking head—or maybe multiple of them—have decided that their own best available current strategy for getting people to pay attention to them instead of some other attention-seeking, know-nothing talking head who also has nothing important to say is to recycle the evidence-free assertion that the “left” is anti-science b/c of “its” supposed view that vaccines cause autism.

I guess the sheer tedium of shooting down such nonsense is too self-indulgent a reason not to keep shooting down such views given how much harm this sad feature of our political discourse can inflict.

It's important to clear such bullshit from the pathways of collective opinion formation first, b/c our society really does need an accurate understanding of why perceptions of risk and similar beliefs about policy-relevant facts sometimes (but very very infrequently) become entangled in antagonistic cultural meanings, and second, b/c the false assertion that one or another cultural group is “unreasoning,” “anti-science” etc. is almost certainly one of the mechanisms by which such entanglements get perpetuated.


No, it’s not the case that that "liberals" are more likely to be anti-vaccine than "conservatives."

from CCP Risk Perception Studies, No. 17

And guess what? People on the “right” are  not meaningfully “anti-vaccine” either, as you can see (if you can't see that in the figure, what does that mean?). 

And --despite the boring boring boring "growing anti-science sensibility" trope to the contrary-- there’s no meaningful correlation between climate-change skepticism and disbelief in evolution and the perception that childhood vaccines endanger public health! 

So next time you want to “sound smart” without knowing anything, please don’t make that claim either.  Becaues the truth-- in case that really matters to you & isn't something you just say you care about for effect--is that there is widspread cultural consensus in the US that universal vaccination is a very valuable thing. 

Also, while we are on the topic: No, there’s no meaningful correlation between holding “liberal”—or “conservative”—views and being concerned about GM food risks!  In fact, the vast majority of ordinary people don’t have any particular opinion on GM foods whatsoever!

And in case you were even thinking of going there, don’t:

There's no meaningful correlation between believing raw milk is healthy and conventional political outlooks in the general population! 

Same for risks of high-voltage power lines and cellphones (and flouridation of water and medical x-rays, etc.), so if you think holding forth on those w/o knowing anything will make you sound smart, just don't. Okay?

Yes, there are small groups of people who believe absurd, unscientific things about vaccines, GM foods, pasteurization of milk & all these other things.

If you are genuinely worried about people spreading misinformation about consequential matters-- & you should be on vaccines, GM foods etc.-- great! By all means call them out on it

Just don't say that that what weird unrepresentative groups are doing is evidence of a "creeping anti-science" sensibility in the public or of the hostility toward science of large communities of Americans who hold completely ordinary political or cultural outlooks.

We live in a nation with 3.2x10^8 people—so you can find large, in absolute terms, numbers of people who believe anything (e.g., that “contrails” are some sort of mind-controlling gas being sprayed by the CIA etc.).

People who believe those weird-ass things might have similar views on politics (I have no idea, frankly, whether people who are anti-vax hold similar political outlooks).

But it is a logical fallacy to infer that therefore all other people who hold those views on politics generally believe in whatever weird things these small fringe groups believe in.

But the illogic of the talking heads who play the “the other side is anti-science!” game bothers me less than two other things.

The first is their conviction that their casual impressions (I doubt they are even first-hand; they likely consist in reading of what other empirically uninformed bull shitters are saying) support empirical characterizations about public opinion.

Just because things “feel” a certain way to you based on your very limited, very skewed exposure to public opinion doesn’t mean that that’s the truth.

The second is the contribution of these know-it-all-know-nothings are making to poisoning our science communication environment.

Ordinary members of the public, of all cultural and political outlooks in the US, are extraordinarily PRO-science.

If you don’t believe this, you are misinformed, likely as a result of the sad vulnerability of people of all cultural and political outlooks to believe that people who don’t agree with them on (admittedly important!) questions about how to live are “stupid,” “closed-mined” etc.

And by dragging science into your illiberal status competition with people whose cultural identity is different from yours, you are making it harder for all of us to converge on the best available evidence on matters that are critical to our collective decisinomaking.

So please just shut up already. 


Critical thinking about public opinion on climate change

A Washington Post Poll found that "70%" of Americans support regulation of green house gases.

The first thing to do, always, is take a close look and see if one accepts that a survey item of this sort is indeed worded in the manner that supports how its results are being characterized.

After that, the question to ask is, "What is the survey item actually measuring?"  

The answer is usually "nothing," or nothing of consequence.

If the item refers to a policy that members of the public don't know or think about--something that doesn't figure in their everyday interactions with other ordinary people--then the survey is not modeling anything going on in the world we live in.

Consider: About half the respondents in a general population survey won't know-- or even have good enough luck to guess-- the answer to the multiple-choice question "how long is the term of a U.S. Senator?"

Are we really supposed to base an inference about what people with that level of political engagement are thinking from the responses of the 1,000 who get weirdly transported out of their everyday worlds and asked (by aliens, it must seem) to indicate whether they "approve" of, say, "the NSA's metadata collection policy?" ("hell, I'm for it-- college athletes are there to be educated!")

The Washington Post tells us that an "overwhelming majority of Americans" support regulating CO2 emissions.

But given that only 58% (+/- 3%) of a general population telephone sample know that "carbon dioxide" rather "hydrogen," "helium," or "radon" is the "gas ... most scientists believe causes temperatures in the atmosphere to rise," what exactly did the "overwhelming majority" of the weird-lottery winners called by "Langer Research Associates" understand "greenhouse gases" to be?

There's zero correlation between responses to the "which gas" question and party affiliation.  That's how one knows that it isn't measuring the same thing as a survey item asking ordinary people whether they believe in human-caused global warming.

That's a question fraught w/ meaning -- not as a matter of policy or science, necessarily -- but as an element of their social world.  

It's one of those things -- along with abortion and gun control -- that separate "us" from "them." 

Indeed, the issue has split people right down the middle -- about 50% have answered "yes" & 50% "no" to the "human-caused global warming" item --for many years. And no, that has not changed recently: 48%, in a nationally representative Cultural Cognition Project survey conducted last month.

What are we supposed to think when told that an "overwhelming majority" of Americans said they "support" a policy that regulates human activities responsible for climate change even though Americans are divided 50-50 on whether human behavior is even causing global warming?

But even that takes responses to the the Washington Post survey item waaaaaaaaaay too seriously.

What valid public-policy survey items measure is an affective orientation -- a feeling that is either positive or negative, strong or weak.

Such orientations can be of extreme importance.  

They can propel people to disregard serious health risks (e.g., lung cancer from smoking) & shrink in terror from non-existent ones (autism from vaccines).

They can determine who they vote for for President (or for the House of Representatives or for anything else)--or who they marry or try to kill.

Often, such affective sensibilities are an expression of a vital element of the respondent's self-conception, one more convincingly seen as a cause than as a consequence of how that person makes sense of all manner of evidence and information, from raw data to brute sense impressions.

But that's what the response is--a register of an affective orientation. It's not an argument or an idea -- not anything like what you would make of a statement that a person in a conversation uttered on his or her own accord.

Same for hypothetical "willingness to pay" measures: if they are genuinely connected to something meaningful for people, responses to them express an attitude-- but are not a valid predictor of the willingness to pay for anything.

Psychometrically speaking, survey responses are "indicators" or indirect measures of a "latent" or unobserved "variable" or influence.  They need to be validated -- i.e., shown by independent means (including coherence with other survey items) to be measuring what one thinks they are measuring--and even then must be regarded as only noisy or imprecise approximations.

Or practically speaking, someone who is genuinely motivated to understand public opinion treats responses to any particular public-policy survey item as one of many ambiguous pieces of evidence.  

When connected to other pieces of evidence including responses to other items that cohere with one another, the responses often support inferences -- solid ones -- on the basis of which we can predict behavior and explain states of affairs.  

But a particular item by itself supports no reliable inference.

Likewise, if an outlier survey item (or even a collection of them) invites an interpretation contrary to the ones borne by other, conventional ones known to be valid and to support reliable inferences, the "hey look" item more than likely isn't measuring the affective sensibility that genuinely motivates the real-world attitudes the pollster is purporting to model.

As I've explained before, the "industrial strength" risk-perception item --the rating of a putative risk source on a multi-point scale -- elicits a straightforward pro- or con- expression of attitude that can vary within a meaningful but relatively constrained range.  At least where the putative risk is something members of the general population have had experience with in the world, responses to ISM can be expected to correlate highly with particular perceptions or factual beliefs relating to that same object.  It can be expected to be correlated with forms of behavior that fit or express the sensibility that it elicits.

But note the emphasis on correlation.  

By itself, responses to ISM are meaningless; the scale is arbitrary ("OMG, the public's perception of the risk of 'private ownership of guns' is only 3.4!").  

Its interpretive utility lies in its covariance with other characteristics -- e.g., w/ "cognitive reflection" or "numeracy" in the case of some risk perception hypothesized to be connected to over-reliance on heuristic reasoning. 

Or w/ "ideology." Here is the ISM for "global warming" in relation to political outlooks:

The correlation displayed is quite strong. Indeed, the correlation between the respondents' ISM rating and their right-left political outlooks is as high as the correlation between the two items -- partisan self-identification (on a 7-point measure) and liberal-conservative ideology (5-point) --that were combined to form the left-right outlook scale itself  (r = - 0.64, p < 0.01).

Well, believing in human caused global warming, unsurprisingly (the two together support the inference that each measures what it seems to).

It also correlates with positions on very familiar, very strongly contested issues like gun control and abortion.

ISM also correlates with individual characteristics, like being white & male, and hierarchical and individualistic, that are known to be indicators, too, of a group disposition that generates strong negative reactions to the issue of global warming (or "climate change"; changing the labe doesn't have any material effect -- which is to say, "believe in climate change?" and "believe in global warming?" are both valid, if noisy, indicators of the same latent affective disposition).  

Look: The affective sensibility that motivates cultural polarization on climate is real.

It can't be exorcised by magic words.

It won't abate if people rely on lab experiments to justify "messaging campaigns" that have been shown decisively not to work by over a decade of real-world evidence.

Yrs of advocacy polls designed to create "overwhelming majority" support for "action on climate change" by insisting it already exists have proven their valuelessness too.

The only way to ameliorate the destructive impact that the climate conflict is having on our capacity for enlightened self-government is to extricate the scientific issues it turns on from the ugly, illiberal form of  status competition that now engulfs them.

And we won't figure out with wishful thinking & meaningless measures.




Weekend update: You'd have to be science illiterate to think "belief in evolution" measures science literacy

It's been soooo long -- at least 3 weeks! -- since I last did a post on the relationship between "belief in evolution" & "science literacy."

That's just not right!  Plus I have some cool new data on this issue.

But let's start with a reprise of the basics -- because one can never overstate how aggressively ignored they are by those who flip out & let loose with a toxic stream of ignorance & cultural zealotry every time a polling organization announces the "startling" news that nearly 50% of the US public continues (as it has for decades) to say "no" when asked whether they believe in evolution (in addition, if one asks how many of the "believers" subscribe to a "naturalistic" or Darwinian view as opposed to a "theistic" variant, the proportion plummets down all the more-- for "Democrats" as well as "Republicans" blah blah blah).

First, there is zero correlation between saying one "believes" in evolution & understanding the rudiments of modern evolutionary science.

Those who say they do "believe" are no more likely to be able to be able to give a high-school-exam passing account of natural selection, genetic variance, and random mutation -- the basic elements of the modern synthesis -- than than those who say they "don't" believe.

In fact, neither is very likely to be able to, which means that those who "believe" in evolution are professing their assent to something they don't understand.

That's really nothing to be embarrassed about: if one wants to live a decent life -- or just live, really --one has to accept much more as known by science than one can comprehend to any meaningful degree.

What is embarrassing, though, is for those who don't understand something to claim that their "belief" in it demonstrates that they have a greater comprehension of science than someone who says he or she "doesn't" believe it.

Second, "disbelief" in evolution poses absolutely no barrier to comprehension of basic evolutionary science.

Fantastic empirical research shows that it is very very possible for a dedicated science educator to teach the modern synthesis to a secondary school student who says he or she "doesn't believe" in evolution.  

The way to do it is to do the same thing that one should do for the secondary school student who says he or she does believe in evolution & who, in all likelihood, doesn't understand it: by focusing on correcting various naive misconceptions that have little to do with belief in the supernatural, etc., & everything to do with the ingrained attraction of people to functionalist sorts of accounts of how natural beings adapt to their environments.

The thing is, though, even after acquiring knowledge of the modern synthesis-- likely the most awe-inspiring & elegant, not to mention astonishingly useful, collection of insights that human reason has ever pried loose from nature--the bright kid who before said "no" when asked if he or she "believes" in evolution is not any more likely to say that he or she now "believes" it

Indeed, confusing "comprehension" with profession of "belief" is a very good way to assure that those kids who are disposed to say they "don't believe" won't learn these momentous insights.

As Lawson & Worsnop observed in the conclusion of their classic study (the one that presented such amazingly cool evidence on how to teach evolution in a way that excited kids of all cultural outlooks to want to learn it), 

[E]very teacher who has addressed the issue of special creation and evolution in the classroom already knows that highly religious students are not likely to change their belief in special creation as a consequence of relative brief lessons on evolution. Our suggestion is that it is best not to try to do so, not directly at least. Rather, our experience and results suggest to us that a more prudent plan would be to utilize instruction time, much as we did, to explore the alternatives, their predicted consequences, and the evidence in a hypothetico-deductive way in an effort to provoke argumentation and the use of reflective thought. Thus, the primary aims of the lesson should not be to convince students of one belief or another, but, instead, to help students (a) gain a better understanding of how scientists compare alternative hypotheses, their predicated consequences, and the evidence to arrive at belief and (b) acquire skill in the use of this important reasoning pattern-a pattern that appears to be necessary for independent learning and critical thought.

There are actually some who say in response, "Not good enough; it is essential not merely to impart knowledge but also to extract a profession of belief too!"

When someone says that, he or she helps us to see that there are actually illiberal sectarians on both sides of the "evolution in education" controversy in this society.

Third -- and here we are getting to the point where the new data come in! -- profession of "belief" in evolution is simply not a valid measure of science comprehension.

This is very much related to what I have already recounted but is in fact a separate point.

Because imparting basic comprehension of science  in citizens is so critical to enlightened democracy, it is essential that we develop valid measures of it, so that we can assess and improve the profession of teaching science to people.

What should be measured, in my view, is a quality of  ordinary science intelligence -- not some inventory of facts ("earth goes 'round the sun, not other way 'round-- check!") but rather an ability to to distinguish valid from invalid claims to scientific insight and a disposition to use in one's own decisions science's signature style of inference from observation.

The National Science Foundation has been engaged in the project of trying to formulate and promote such a measure for quite some time. A few years ago it came to the conclusion that the item "human beings, as we know them today, developed from earlier species of animals," shouldn't be included when computing "science literacy."

The reason was simple: the answer people give to this question doesn't measure their comprehension of science. People who score at or near the top on the remaining portions of the test aren't any more likely to get this item "correct" than those who do poorly on the remaining portions.

What the NSF's evolution item does measure, researchers have concluded, is test takers' cultural identities, and in particular the centrality of religion in their lives.

Predictably, the NSF was forced to back off this position by a crescendo of objections from those who either couldn't get or didn't care about the distinction between measuring science comprehension and administering a cultural orthodoxy test. The NSF regularly notes the controversy but prudently distances itself from what the significance of it.

But those of us who don't have to worry about whether taking a stance will affect our research budgets, who genuinely care about science, and who recognize the challenge of propagating widespread comprehension and simple enjoyment of science in a culturally pluralistic society (which is, ironically, the type of political regime most conducive to the advance of scientific discovery!) shouldn't equivocate.

We should insist that science comprehension be measured scientifically and point out the mistakes -- myriads of them -- being made by those who continue to insist that professions of "belief" in evolution are any sort of indicator of that.

I've reported some evidence before in this blog that reinforces the conclusion that "belief" in evolution is a measure of who people are and not what they know.

Well, here's some more.

Following up on a super interesting tidbit from the 2014 NSF Science Indicators, I included alternate versions of the conventional NSF Indicator "evolution" item in a science comprehension battery that I administered to a large (N = 2000) nationally representative sample earlier this month.

One was the conventional "true-false" statement, "Human beings, as we know them today, developed from earlier species of animals.”

The second simply added to thist sentence the introductory clause, "According to the theory of evolution, ..."

The NSF had reported on a General Social Science module from a few years ago that found that the latter version elicits a much higher percentage of "true" responses.

Well, sure enough.   

As the Figure at the top of the post shows, the proportion who selected "true" jumped from 55% on the NSF item to 81% on the GSS one!

Wow!  Who would have thought it would be so easy to improve the "science literacy" of benighted Americans (who leaving aside the "evolution" and related "big bang" origin-of-the-universe items already tend to score better on the NSF battery than members of other industrialized nations).

Seriously: as a measure of what test takers know about science, there's absolutely no less content in the GSS version than the NSF.  Indeed, if anyone who was asked to give an explanation for why "true" is the correct response to the NSF version failed to connect the answer to  "evidence consisntent with the theory of evolution  ..." would be revealed to have no idea what he or she is talking about.

The only thing the NSF item does that the GSS item doesn't is entangle the "knowledge" component of the "evolution" item (as paltry as it is) in the identity-expressive significance of "positions" on evolution.  

Want some more evidence? Here you go:

This figure shows the relationship between the probability of a "true" response to the respective versions of the question conditional "religiosity" & "science comprehension." (The figure graphically reports the results of a regression model. If you want to see the raw Click me--I will make you more science literate, I swear!data, click on the inset to the left!)

The former was measured by aggregating into a scale responses to items on self-reported frequency of church attendance, frequency of prayer, and importance of God (α = 0.87).

The latter was formed by combining the NSF's science indicator battery (excluding the "evolution" one, to avoid circularity) with a set of Numeracy and Critical Reflection Test items.  The NSF indicators, a collection of "true-false" items,  can be seen as comprising knowledge of elementary facts; the additional items assess the sorts of reasoning skills--including, in particular, the disposition and ability to make valid inferences from quantitative and other forms of information--that a person needs in order reliably to acquire scientific knowledge. 

The items cohere nicely, forming a highly reliable unidimensional scale (α = 0.84), which I scored with an item response theory model. 

Indeed, the main reason for collecting data on the GSS and NSF variants of the evolution item was to see what the frequency of "true" responses to them would reveal about the item's relative connection to religious identity and science comprehension.

These data answer that question.

The panel on the left confirms that the NSF item does indeed measure religious identity, not scientific knowledgeable.  

Or maybe one can see it as indicating science comprehension for relatively secular folks, since in them one sees what one would expect if that were the case--namely, that the probability of answering "true" goes up as people become progressively more comprehending of science.

But the probability of answering "true" doesn't go up--if anything it goes down--as individuals who are above average in religiosity become more science comprehending.  That's manifestly inconsistent with any inference that the answer to the question indicates the science comprehension of people with a more religious identity. (In case you were wondering -- and it's perfectly reasonable to -- there was a fairly minor negative correlation-- r = - 0.17, p < 0.01-- between religiosity and science comprehension.)  

Now behold the panel on the right!

Here we do see exactly what one would expect of an item that indicates (i.e., correlates, because it's presumably caused by) science comprehension--an increasing probability of answering "true" -- for both non-religious and religious individuals!

By adding the introductory clause, "According to the theory of evolution," the GSS question disentangles ("unconfounds" in psychology-speak) the "science knowledge" component and the "identity expressive" components of the item.

Gee, Americans aren't that dumb after all!

Or maybe they are; this is too easy a question if one wants to figure out whether Americans or anyone else really knows anything about science: some 80% of the respondents answer it correctly -- a figure that rapidly approaches 100% among those of even middling science comprehension.

So ditch this question & substitute for it one more probative of genuine science comprehension -- like whether the test taker actually gets natural selection, random mutation, and genetic variance, which are of course the fundamental mechanisms of evolution and which kids with a religious identity can be taught just as readily as anyone else.

Or actually, how about this.

Instruct the test taker to reflect on the graph above and then respond to the item, 

"'Belief in evolution' is a valid measure of a person's science literacy," true or false?


What is to be done? Let's start with why ... a fragment

From still another thing I'm working on that is distracting me from my main job--writing blog entries:

You asked me to describe what I want to do.  I think I’m more likely to convey that if I start with an account of why.

Two things concern me.  The first is the failure of professions that exist to enlarge, disseminate, and exploit the insights of valid empirical inquiry to use those methods to improve their own proficiency in enlarging, disseminating, and exploiting scientific knowledge.  Call this the “meta-empiricism spectacle” (MS).

Call the second problem “Popper’s revenge” (PR). Cultural pluralism makes liberal democratic societies uniquely congenial to the advancement of scientific inquiry; at the same time, however, it multiplies the occasions for polarizing forms of status conflict between the cultural groups within which diverse citizens necessarily come to know what’s known.  This dynamic puts at risk citizens’ enjoyment of both the promise of tolerance and the enormity of knowledge that are the hallmarks of liberal democratic societies.

MS and PR interact.  As a result of their failure to apply empirical methods to themselves, the professions that traffic in empirical knowledge—from conservation advocacy groups to government regulatory agencies, from science journalists to public health professionals, from educators to judges—fail to negotiate the forms of illiberal status competition that impede public recognition of what’s known to science.

I want to help address these problems....

But in any case, you now have a sense of why; so here is what I want to do.

I am intent on stimulating and being a party to the creation of as many projects as possible aimed at creating “evidence-based practices” within the professions most responsible for assuring reliable recognition of what science knows by the culturally diverse individuals and groups whose welfare such knowledge can enhance....


More on public "trust of scientists": *You* tell *me* what it means!

Okay, so I've done a good number of posts on "trust" in science/scientists. The basic gist of them is that I think  it's pretty ridiculous to think that any significant portion of the US public distrusts the authority of science -- epistemic, cultural, political, etc. -- or that partisan divisions in regard to trust in science/scientists can plausibly explain polarization over particular risks or other policy-relevant facts that admit of scientific inquiry (vice versa is a closer call but even there I'm not persuaded).

So here's some more data on the subject.

It comes from a large (N = 2000) nationally representative survey administered as part of an ongoing collaborative research project by the Annenberg Public Policy Center and CCP (it's a super cool project on reasoning & political polarization; I've been meaning to do a post on it -- & will, tomorrow"!).

The survey asked respondents to indicate on a 6-point "agree-disagree" Likert measure whether they "think scientists who work" (or in one case, "do research for") in a particular institutional setting "can be trusted to tell the public the truth."

The institutions in questions were NASA, the CDC, the National Academy of Sciences, the EPA, "Industry," the military, and "universities."

We had each subject evaluate the trustworthiness of only one such group of scientists.

Often researchers and pollsters ask respondents to asses the trustworthiness of multiple groups of scientists, or of scientists generally in relation to multiple other groups.

One problem with that method is that it introduces a "beauty pageant" element in which respondents rank the institutions.  If that's what their doing, one might conclude that the public "trusts" a group of scientists or scientists generally more than they actually do simply because they trust the others even less.

So what did we find?

I'll tell you (just hold on, be patient).  

But I won't tell you what I make of the findings. 

Do they support the widespread lament of a creeping "anti-science" sensibility in the U.S.?  

Or the claim that Republicans/conservatives in particular are anti-science or less trusting in science than they were in the past.

Or do they show "the left" is in fact "anti-science" -- as much so or more than "the right" etc.

You tell me!

Actually, I'm sure everyone will come to exactly the same conclusion on these questions.  Here as elsewhere, the facts speak for themselves!


The "generalizability problem" -- a fragment

From something I'm working on (one of many things distracting me from this blog; I've experienced a curious inversion recently in proscrastination diversions....)

One of the major challenges confronting the science of science communication is generalizability.  This problem is obvious when researchers engage in  lab experiments. By quieting the cacophony of uncontrollable real-world influences, such experiments enable the researcher to isolate and manipulate mechanisms of interest, and thus draw confident inferences about their significance, or lack thereof. But how, then, can one know whether the effects observed in these artificially tranquil conditions will hold up in the chaotic real-life environment from which the researcher sought refuge in the lab? 

It would be a mistake, though, to think that this difficulty reflects some fatal defect in laboratory methods.  And not just because such methods do indeed play an indispensable role in the formation of communication strategies that can subsequently be tested outside the lab. For any empirical testing that occurs in the field must also confront the question of generalizability: how is one to know that what worked in one distinctively messy real-world setting will work in another distinctively messy one?

The generalizability problem is central to the motivation for our proposal.  Disturbingly, a large fraction of researchers offering counsel to conservation advocates and policymakers simply ignore this issue altogether. 

But just as bad, a large fraction of the remainder try to address it in the wrong way.  They believe that the goal of empirical research is to identify a fixed set of universally effective “techniques” or “best practices” that can, with the benefit maybe of cartoon-illustrated instruction manuals, be confidently and more-or-less thoughtlessly applied by communicator "consumers." 

But in fact, the only technique of the science of science communication that generalizes—the sole valid “best practice” it has to offer—is its method. Successful lab experiments and field studies alike do enlarge understandings of how the world works. But how the insights they generate can be brought successfully to bear on any new problem will always be a question that those promoting science-informed conservation policymaking will have to answer for themselves.  The only way they can reliably do so, moreover, is by using empirical methods to adapt what the science of science communication knows to the distinctive circumstances at hand.  

Perfecting knowledge of how to use empirical methods in the everyday practice of conservation-science communication—so that the generalizability issue will always be confronted and confronted effectively—is the whole point of the proposed ....


"Energy future 2030" talk (slides, video)


Some "pathological" public risk perceptions & a whole bunch of "normal" ones

From slides in tak about to give at a biotech conference in Syracuse.  Political differences (or lack thereof) in top slide & "science comprehension" magnification of the same (or lack thereof) in bottom.

More later -- but if anyone wants to offer their own views in the meantime, freel free!


So much for that theory . . . (fracking freaks me out  #2)


So having been freaked out to discover how pervasively polarized members of the public appear to be about fracking despite knowing nothing about it, I resolved to do a little experiment.

In the previous data collection, I had measured perceptions of fracking risks using the "industrial strength measure," which solicits a rating of how "serious" a societal risk some activity poses to "human health, safety, or prosperity."

My thought was that maybe what had generated such a strong degree of polarization might be the wording of the item, which asked subjects to supply such a rating for "fracking (extraction of natural gas by hydraulic fracturing)."

I figured maybe this language--the sort of "dirty" sounding word "fracking" and the references to "extraction" (sounds like a painful and invasive procedure to subject mother Nature to) &  "natural gas" ("boo" if you have an egalitarian, "game over, capitalists!" sensibility; yay, if you have an individualist, "yes we can, forever & ever & ever!" one) would be sufficient to alert  the ordinary Americans who made up the sample (most of whom likely wouldn't have been able to define fracking without this clue) that this was an "environmental" issue. That would be enough to enable most of them to locate the issue's position on the "cultural theory of risk" map, particularly if they were above-average in science comprehension and thus especially skilled at fitting information to their cultural identities.

So I thought I'd try an experiment.  Administer the same measure but vary the description of the putative risk source: in one condition, it would be called simply "fracking"; in another, it would be referred to as "shale oil gas production"; and in a third, the risk source would be identified as it was in the earlier survey-- "fracking (extraction of natural gas by hydraulic fracturing.)"

I figured that relative to the third group, those in the first (plain old "fracking") would be less polarized, and those in the second ("shale oil gas production"; sounds harmless!) would be the least agitated of all.

Actually, I was modeling this experiment loosely on  Sinaceur, M., Heath, C. & Cole, S. Emotional and deliberative reactions to a public crisis mad cow disease in France, Psychol Sci 16, 247-254 (2005)), a great study in which the investigators showed that lab subjects formed affect- or emotion-pervaded judgments when evaluating risk information relating to "Mad Cow disease" but formed more analytical, calculative ones when the information referred to either "bovine spongiform encephalopathy (BSE)" or "a variant of Creutzfeldt-Jakob disease (CJD)" instead.

Well, here's what I found:


Click on the image for a closer inspection, but basically, the difference in effect associated with the variation in wording, while "in the direction" hypothesized, was way too small for anyone to think it was practically meaningful.

Same thing for the influence of the wording on the interaction between political outlooks (measured with a right-left scale) and science comprehension (measured with a cool composite of substantive knowledge & critical reasoning measures; more on that "tomorrow"): 

So much for that theory.

But I have another one!  

All this agitation about fracking, I'm convinced, is really a battle between those who do & those who don't recognize the supreme value of local democratic decisionmaking!




What to think about how "How You Say It" — an empirical study of aporetic judicial reasoning

D. Evans, atop "Aporia," before this year's Kentucky Derby

A "CCP journal club!" report from D. Evans:

"Aporia" is a mode of reasoning that shows the author comprehends “an issue’s intractable complexity.” 

Too often, judicial opinions addressing complex value questions are anything but aporetic. While the public is deeply divided over the issue, judicial opinions often “effect a posture of unqualified, untroubled confidence” in the outcome. This “[h]yperbolic certitude” might undermine the legitimacy of the opinion with the losing side, making it seem as though the decisionmaker was biased or unwilling to recognize the strength of arguments supporting the losing side’s position.

In addressing how courts can assure citizens of the law's neutrality, my CCP colleagues and I have conjectured that judicial decisions might reduce cultural polarization and garner acceptance from the losing side by abandoning the norm of reasoning as if the answer is obvious, indisputable, and certain.

Instead, if a court were to recognize (a) the difficulty (even intractability) of the problem, and (b) the strength of the losing side’s case, perhaps the losers would be more likely to perceive the opinion as a legitimate one; one that took their concerns and arguments deeply into account. If the losing side sees its concerns and arguments were thoroughly considered in the decision, it might also be more open to accepting the arguments that prevailed in the outcome. I have long thought about testing this hypothesis that aporetic reasoning would reduce cultural polarization over a controversial ruling.

So I was really excited to read Rob Robinson’s empirical study on exactly this point: It’s How You Say It – Ameliorating Cultural Cognition of Judicial Rulings Through Aporetic Reasoning.

Robinson's study follows a few others with promising implications for the aporia hypothesis: Tom Tyler's research, described here, finds that public views about the legitimacy of legal authority are influenced by the procedural justice and by the distributive justice of the outcomes, but less affected by the favorability of the outcome. Dan Simon and Nicholas Scurich, Lay Judgmentsof Judicial Decision-Making, have found that people tend to agree more with decisions recognizing good reasons support either side of the case than decisions that only recognize the value of one side's position. They also find that an opinion giving no reasons is more persuasive than one including a single, curt reason. (Simon and Schurich's findings rebuffed a preexisting hypothesis called ‘placebic reasoning’ – that people are more likely to credit decisions or actions when backed by reasons, even if those reasons are entirely redundant (i.e., asking to cut in line for a copy machine was less credible than asking to cut in line for a copy machine and providing a redundant reason, “because I have to make copies.”)).

While these studies support the aporia hypothesis, Robinson is the first (to my knowledge) to frame his testing in terms of aporia, specifically.

Robinson conducted an experiment designed to test how members of the public would react to more and less aporetic versions of a judicial decision contrary to their own position on gay marriage.

The study subjects, 619 individuals representing a mix of university students Amazon’s MTurk workers, were assigned to one of three mirror-image conditions. 

In the “control” condition, subjects read a newspaper article describing a judicial decision that examined whether homosexuality should be recognized as an “immutable” (i.e., unchosen, and unalterable) trait. The story reported the court’s conclusion—either “no,” if subjects said they supported gay marriage; or  “yes,” if they said they did—and nothing more.

In the “monolithic” condition, the article includes a quote from the court’s opinion in which the court defends its reasoning by remarking that that an “objective reading of the evidence leads to no other conclusion.”  The court explains that it is obliged to reject the position supported by the study subject—either that homosexuality  is “ immutable,” in the version of the article shown to gay-marriage supporters; or that it is not, in the version shown to gay-marriage opponents—on the ground that there is “no clear scientific consensus” in favor of that view.

In the “aporetic” condition,  the news story quotes language from the opinion evincing a more nuanced stance.  The quoted language chides one side or the other—either  “those who believe homosexuality is a choice” for “often ignor[ing] evidence [to the contrary]” or  “those who argue sexual orientation is fixed or unchanging” for “often overstat[ing] their case.”  The court nevertheless justifies a ruling in favor of the scolded side on the ground that a court is powerless to deem matters otherwise in the face of uncertain evidence.

Robinson reports that subjects found the court’s reasoning more persuasive in both the “monolithic” and “aporetic” conditions  than in the control. In other words, the subjects were least disappointed by the decision when they were told the court had given an explanation for rejecting their position. 

In the view of the subjects who oppose gay marriage, the aporetic opinion was even more persuasive than the monolithic one.

But for those who support gay marriage, the persuasiveness of the decision did not differ significantly among those assigned to “aporetic” and “monolithic” conditions, respectively.

The mean opponents of same-sex marriage rated their disagreement with three forms of the pro-same-sex marriage decision on a scale of 1 ("extremely agree") to 6 ("extremely disagree"): Control 4.16; Monolothic 4.10; Aporetic 3.53. For opponents of same sex marriage, the monolithic opinion was about .06 less disagreeable than the control, the aporetic one was .6 less disagreeable. The mean supporters of same-sex marriage rated their three forms of the anti-same-sex marriage decision follows: Control 4.58; Monolithic  4.46; Aporetic 4.36. Among supporters of same-sex marriage, the monolithic opinion was about .02 less disagreeable than the control, and the aporetic one was .12 less disagreeable than the control. 

This is a super valuable study!

I particularly liked the way in which Robinson distilled the aporetic reasoning into a few quotes set within the framework of a newspaper article. There is much innovative about his deisgn, and his study makes me eager to design a follow up study along these lines. In thinking about how to do so, I have been pondering several questions about the design of this study:

  • One puzzling aspect of his findings is that supporters of same-sex marriage were overall more negative about all three forms of the opinion ruling against it, and they found the aporetic version only slightly less disagreeable. While the aporetic opinion significantly reduced the extent that opponents of same-sex marriage disagreed with a pro-same-sex marriage decision. (The effect of the aporetic treatment on anti-same sex marriage group's disagreement was -0.592, while the effect of the aporetic treatment on pro-same-sex marriage group's disagreement was only -0.150.)

    Why were supporters of same-sex marriage overall more resistant to crediting the contrary opinion, and why was their disagreement less mitigated by aporia? Robinson states this might be caused by the sample of those who favor same-sex marriage being larger (N pro-same-sex marriage=496, N anti-same-sex= 161). (But the larger sample should supply the more significant result if the phenomenon exists, not the less significant one.) He also posits that the difference in reaction may result from "those who favor gay marriage simply having a stronger reaction to empirical claims regarding immutability than those who are opposed." P. 18.

    It could be the case that supporters of same-sex marriage are categorically more rigid in their position, and less willing to credit a contrary ruling regardless of its reasoning.

    But I'd posit another possible explanation. Perhaps the pro-same-sex marriage group's rigid disagreement relates to their views on the relevance of whether homosexuality is immutable, as opposed to an extra-strong belief that same-sex marriage should be allowed. It seems that there may be many egalitarian individuals like me who think that same-sex marriage should be allowed regardless of whether it is immutable. I think any constitutionally protected individual liberty should be an impermissible basis for discrimination, regardless of whether it is immutable. (Indeed I'm offended by the notion that protection is limited to traits that are predetermined rather than chosen pursuant to constitutionally guaranteed autonomy.). I would be much more persuaded to support regulation of same-sex relationships if it were shown that they caused harm to public welfare: the stability of marriage or childrearing.

    Hence, I wonder whether the extra-strong disagreement with the opinion finding homosexuals are not a protected class may represent disdain of the idea that immutability determines the degree of constitutional protection. This is frustration with the legal standard as opposed to ideology-based cognitive rigidity. For this reason, one of my overarching questions about Robinson's study is whether immutability is the best empirical issue for measuring cultural effects in the same sex marriage debate. I would be inclined to focus on welfare-related empirical questions, such as how same-sex marriage impacts childrearing, a question on which strong cultural effects have been observed.

    Furthermore, because these welfare concerns seem to be more often cited in the public debate as a reason for prohibiting same sex marriage, it seems cultural identity may be more strongly tied to one’s beliefs about these questions than one’s belief about immutability. (While certainly part of the debate about the morality of homosexuality, immutability seems to be cited less often as the public reason for prohibiting same sex marriage.) It seems some might oppose same sex marriage for purported public welfare consequences, regardless of whether sexual orientation is immutable. And as I have described above, some proponents of same-sex marriage might be particularly resentful of a decision based on immutability, as they do not believe this should be a relevant factor. This group might also, while cognitively motivated to support a pro-same-sex marriage ruling, be disinclined to support a ruling that homosexuality is immutable.

My other questions pertain to specific elements of the study's design:

  • Asking for views about same-sex marriage: I wonder whether first asking subjects about their stance on same-sex marriage makes them less susceptible to being persuaded by the aporetic reasoning we are testing. Because people don’t want to be inconsistent—either internally or be perceived as such by those conducting the survey—they might resist crediting the ruling after reporting disagreement with its conclusion at the outset of the study, regardless of whether the find the aporetic or monolithic reasoning persuasive. It seems the cultural measures provide enough information to predict a subject’s likely orientation on same-sex marriage, and it is unnecessary to ask subjects about the issue being studied.        
  • Assignment to conditions with which subjects are inclined to disagree: I also question the decision to only show subjects opinions with which they are inclined to disagree. It seems to me that a study of this nature should measure the reasoning’s persuasiveness to both those inclined to disagree with it and those inclined to agree with it. It may be that an aporetic opinion is more persuasive to those inclined to disagree, it is less persuasive to those inclined to agree. It seems this, too, would be a noteworthy finding. The question should be whether opposing cultural groups converge on the persuasiveness of an aporetic opinion more than they do on a monolithic one.        
  • Focus on whether the opinion is persuasive rather than correct: I would not focus on asking subjects whether the court’s conclusion is correct or accurately reflects scientific findings, but whether they find the opinion persuasive. Subjects might agree with the court’s conclusion or believe that it accurately states scientific research, but find its reasoning unpersuasive. Or to the contrary, they might disagree with the court’s scientific conclusion, but find the reasoning persuasive.        
  • More detailed reasoning: I might consider including a few more sentences so that the court’s reasoning more clearly pronounces three elements that I associate with aporia: (a) noting that this is a difficult, perhaps intractable, question, on which there may be no correct answer; (b) saying the evidence is unclear, and presents the strongest points in favor of each side; and (c) gives reasons for crediting one side’s position despite this empirical uncertainty. (I think this last point is the most contentious aspect of aporia – a court must justify its conclusion after admitting that it is uncertain as to the evidence – and it would be particularly interesting to test.). The monolithic condition would do the opposite--e.g., (a) state that the question is simple with a clear right answer; (b) say the evidence is clear or unequivocal; and (c) hold that there's no way one could reach a different result based on the evidence before the court.         
  • Singling out one side: The aporetic versions in Robinson's study single out one side (The unprotected class version begins: “Those who believe homosexuality is a choice often ignore evidence [to the contrary]”; and protected class begins: “Those who argue sexual orientation is fixed or unchanging often overstate their case.”). In contrast, the monolithic condition does not single out one side in this way, but states: “There is no scientific consensus. . . .” I wonder whether statements that the winning parties “overstate” their case or “ignore” evidence are necessary to the aporetic reasoning. It seems that, for the sake of maintaining the highest degree of similarity between conditions, the aporetic opinion should simply say “The evidence is uncertain as to whether. . . .” Aside from uniformity, I am concerned is that these words might be read as accusing the prevailing side of being disingenuous. One party overstating its case has nothing to do with the court’s aporetic reasoning, but it could heighten the losing side’s suspicion for the winning side’s claims.  
  • Explaining what’s at stake before the aporia manipulation: The prompt in this survey tells subjects that immutability determines the degree of constitutional protection afforded same-sex couples, but it does not explicitly say that the degree of constitutional protection determines whether laws prohibiting same-sex marriage are constitutional. It seems this connection—immutability effectively determines the constitutionality of laws prohibiting same-sex marriage—should be made explicit before the aporetic statement about immutability. It seems that priming readers with the cultural significance of the court’s reasoning about immutability would enhance the tendency to engage in motivated reasoning, and this would increase the effects we’d expect to see.        

In raising these questions, I do not mean to undermine the value of Robinson’s study. To the contrary, I find it very valuable. Not only is it encouraging in that it suggests this question is worth studying further, it also supplies an inspiring baseline for designing another study on this subject.


The fractal nature of the "knowledge deficit" hypothesis: Biases & heuristics, system 1 & 2, and cultural cognition

I often get asked—in correspondence, in Q&A after talks, in chance encounters with strangers while using one or another mode of public transportation—what the connection is between “cultural cognition” and “all that heuristics and biases stuff” or some equivalent characterization of the work, most prominently associated with Nobelist Daniel Kahneman, on the contribution that automatic, largely unconscious mechanisms of cognition make to risk perception.  

This excerpt, from Kahan, D., Braman, D., Cohen, G., Gastil, J. & Slovic, P. Who Fears the HPV Vaccine, Who Doesn’t, and Why? An Experimental Study of the Mechanisms of Cultural Cognition,  Law & Human Behavior 34, 501-516,  (2010), furnishes half the answer.  

The basic idea is that cultural cognition is not an alternative to the “heuristics and biases” position but a supplement that helps explain how one and the same mechanism—“the availability effect,” “biased assimilation,” “probability neglect” etc.—can generate systematically opposing risk perceptions in identifiable groups of people. 

But as I said, this is only half the answer. At the time that CCP researchers did this study, they were carrying out a research project to examine how cultural cognition interacts with heuristic or “System 1” information processing, which as I indicated features automatic, unconscious mechanisms of cognition. 

In a project that we started thereafter, we’ve been examining the connection between cultural cognition and “System 2” reasoning, which involves conscious, analytic forms of information processing.  In particular, we’ve been empirically testing the popular conjecture that disputes over climate change and other politically contested risks reflects the public’s over-reliance on heuristic reasoning

Not so. Cultural cognition captures and redirects conscious, analytical reasoning, too

Tragically, people use their quantitative and critical-reasoning dispositions to fit empirical data and other technically complex forms of evidence to the position that affirm their identities.  As a result, those who are most disposed to use System 2 reasoning are the most polarized

If you are wandering the internet preaching that the climate change controversy is a consequence of public’s over-reliance on “emotion” or “fast, intuitive heuristics” etc etc you are ignoring evidence. It was a very reasonable hypothesis, but you need to update your understanding of what’s going on as new evidence emerges—just as climate scientists do! 

Sometimes I think this account—that the climate change controversy is a consequence of “public irrationality”—is a kind of pernicious story-telling virus that is impervious to treatment with evidence. 

Makes me realize, too, the irony that I am implicitly affirming my adherence to the “knowledge deficit” hypothesis by continually trying to overcome a version of it by simply bombarding propagators of the "System 1 vs. system 2" (or "bounded rationality," "experiential reasoning," "public irrationality" etc.) explanation of conflict over climate change with more and more and more and more empirical evidence that their account is way too simple. 

Life is weird. And interesting.


Theoretical Background: Heuristics, Culture, and Risk

The study of risk perception addresses a puzzle. How do people—particularly ordinary citizens who lack not only experience with myriad hazards but also the time and expertise necessary to make sense of complex technical data—form positions on the dangers they face and what they should do about them?

Social psychology has made well-known progress toward answering this question. People (not just lay persons, but quite often experts too) rely on heuristic reasoning to deal with risk and uncertainty generally. They thus employ a range of “mental shortcuts”: when gauging the danger of a putatively hazardous activity (the possession, say, of a handgun, or the use of nuclear power generation), they consult a mental inventory of recalled instances of misfortunes involving it, give special weight to perceived authorities, and steer clear of options that could improve their situation but that also involve the potential to make them worse off than they are at present (“better safe, than sorry”) (Kahneman, Slovic, & Tversky, 1982; Slovic, 2000; Margolis, 1996). They also employ faculties and styles of reasoning—most conspicuously affective ones informed by feelings such as hope and dread, admiration and disgust—that make it possible for them to respond rapidly to perceived exigency (Slovic, Finucane, Peters & MacGregor, 2004).

To be sure, heuristic reasoning of this sort can lead to mistakes, particularly when they crowd out more considered, systematic forms of reasoning (Sunstein, 2005). But they are adaptive in the main (Slovic et al, 2004).

As much as this account has enlarged our knowledge, it remains incomplete. In particular, a theory that focuses only on heuristic reasoning fails to supply a cogent account of the nature of political conflict over risk (Kahan, Slovic, Braman & Gastil, 2006). Citizens disagree, intensely, over a wide range of personal and societal hazards. If the imprecision of heuristic reasoning accounted for such variance, we might expect such disagreements to be randomly distributed across the population or correlated with personal characteristics (education, income, community type, exposure to news of particular hazards, and the like) that either plausibly related to one or another heuristic or that made the need for heuristic reasoning less necessary altogether. By and large, however, this is not the case. Instead, a large portion of the variance in risk perception coheres with membership in groups integral to personal identity, such as race, gender, political party membership, and religious affiliation (e.g. Slovic, 2000, p. 390; Kahan & Braman, 2006). Whether the planet is overheating; whether nuclear wastes can be safely disposed of; whether genetically modified foods are bad for human health—these are cultural issues in American society every bit as much as whether women should be allowed to have abortions and men should be allowed to marry other men (Kahan, 2007). Indeed, as unmistakably cultural in nature as these latter disputes are, public debate over them often features competing claims about societal risks and benefits, and not merely competing values (e.g. Siegel, 2007; Pollock, 2005).

This is the part of the risk-perception puzzle that the cultural theory of risk is distinctively concerned with (Douglass & Wildavsky, 1982). According to that theory, individuals conform their perceptions of risk to their cultural evaluations of putatively dangerous activities and the policies for regulating them. Thus, persons who subscribe to an “individualist” worldview react dismissively to claims of environmental and technological risks, societal recognition of which would threaten markets and other forms of private ordering. Persons attracted to “egalitarian” and “communitarian” worldviews, in contrast, readily credit claims of environmental risk: they find it congenial to believe that commerce and industry, activities they associate inequity and selfishness, cause societal harm. Precisely because the assertion that such activities cause harm impugns the authority of social elites, individuals of a “hierarchical worldview” are (in this case, like individualists) risk skeptical (Rayner, 1992).

Researchers have furnished a considerable body of empirical support for these patterns of risk perception (Dake, 1991; Jenkins-Smith, 2001; Ellis & Thompson, 1997; Peters & Slovic, 1996; Peters, Burriston & Mertz, 2004; Kahan, Braman, Gastil, Slovic & Mertz, 2007). Such studies have found that cultural worldviews explain variance more powerfully than myriad other characteristics, including socio-economic status, education, and political ideology, and can interact with and reinforce the effect of related sources of identity such as race and gender.

Although one could see a rivalry between culture theory and the heuristic model (Marris, Langford, O’Riordan 1998; Douglas, 1997), it is unnecessary to view them as mutually exclusive. Indeed, one conception of the cultural theory—which we will call the cultural cognition thesis ((Kahan, Braman, Monahan, Callahan & Peters, in press; Kahan, Slovic, Braman & Gastil, 2006)—seeks to integrate them. Culture theorists have had relatively little to say about exactly how culture shapes perceptions of risk.[i] Cultural cognition posits that the connection is supplied by conventional heuristic processes, or at least some subset of them (DiMaggio, 1997). On this account, heuristic mechanisms interact with cultural values: People notice, assign significance to, and recall the instances of misfortune that fit their values; they trust the experts whose cultural outlooks match their own; they define the contingencies that make them worse off, or count as losses, with reference to culturally valued states of affairs; they react affectively toward risk on the basis of emotions that are themselves conditioned by cultural appraisals—and so forth. By supplying this account of the mechanisms through which culture shapes risk perceptions, cultural cognition not only helps to fill a lacuna in the cultural theory of risk. It also helps to complete the heuristic model by showing how one and the same heuristic process (whether availability, credibility, loss aversion, or affect) can generate different perceptions of risk in people with opposing outlooks.

The proposition that moral evaluations of conduct shape the perceived consequences of such conduct is not unique to the cultural cognition thesis. Experimental study, for example, shows that negative affective responses mediate between moral condemnation of “taboo” behaviors and perceptions that those behaviors are harmful (Gutierrez & Giner-Sorolla, 2007). The same conclusion is also supported by a number of correlational studies (Horvath & Giner-Sorolla, 2007; Haidt & Hersh, 2001). The point of contact that the cultural cognition thesis, if demonstrated, would establish between cultural theory and these other works in morally motivated cognition would also lend strength to the psychological foundation of the former’s account of the origins of risk perceptions.



[i] For functionalist accounts, in which individuals are seen as forming risk perceptions congenial to their ways of life precisely because holding those beliefs about risk cohere with and promote their ways of life, see Douglas (1986) and Thompson, Ellis & Wildavsky (1990).


Science and public policy: Who distrusts whom about what?

More or less what I said at really great NSF-sponsored "trust" workshop at University of Nebraska this weekend. Slides here

1.  What public distrust of science?

I want to address the relationship of trust to the science communication problem.

As I use the term, “the science communication problem” refers to the failure of valid, compelling, and widely accessible scientific evidence to dispel persistent cultural conflict over risks or other policy-relevant facts to which that evidence directly speaks. 

The climate change debate is the most spectacular current example, but it is not the only instance of the science communication problem. Historically, public controversy over the safety of nuclear power fit this description. Another contemporary example is the political dispute over the risks and benefits of the HPV vaccine.

Distrust of science is a common explanation for the science communication problem. The authority of science, it is asserted, is in decline, particularly among individuals of a relatively “conservative” political outlook.

This is an empirical claim.  What evidence is there for believing that the public trusts scientists or scientific knowledge less today than it once did? 

The NSF, which is sponsoring this very informative conference, has been compiling evidence on public attitudes toward science for quite some time as part of its annual Science Indicators series.

One measure of how the pubic regards science is its expressed support for federal funding of scientific research.  In 1985, the public supported federal science funding by a margin of about 80% to 20%. Today the margin in the same—as it was at every point between then and now.

Back in 1981, the proportion of the public who thought that the government was spending too little to support scientific research outnumbered the proportion who thought that the government was spending too much by a margin of 3:2. 

Today around four times as many people say the government is spending too little on scientific research than say it is spending too much.

Yes, there is mounting congressional resistance to funding science in the U.S.--but that's not because of any creeping "anti-science" sensibility in the U.S. public. 

Still aren't sure about that?

Well, how would you feel if your child told you he or she was marrying a scientist? About 70% of the public in 1983 said that would make them happy.  The proportion who said that grew to 80% by 2001, and grew another 5% or so in the last decade.

Are “scientists … helping to solve challenging problems”? Are they “dedicated people who work for the good of humanity”?

About 90% of Americans say yes.

Do you think you can squeeze the 75% of Republicans who say they “don’t believe in human-caused climate change” from the remainder? Better double check your math.

In sum, there isn’t any evidence that creeping distrust in science explains the science communication problem, because there’s no evidence either that Americans don’t trust scientists or that fewer of them trust them now than in the past.

Of course, if you like, you can treat the science communication problem itself as proof of such distrust.  Necessarily, you might say, the public distrusts scientists if members of the public are in conflict over matters on which scientists aren’t.

But then the “public distrust in science” explanation becomes analytic rather than empirical.  It becomes, in other words, not an explanation for the science communication problem but a restatement of it.

If we want to identify the source of the science communication problem, simply defining the problem as a form of “public distrust” in science—on top of being a weird thing to do, given the abundant evidence that the American public reveres science and scientists—necessarily fails to tell us what we are interested in figuring out, and confuses a lot of people who want to make things better.

2. The impact of cultural distrust on perceptions of what scientists believe

So rather than define the science communication problem as evincing “public distrust in science,” I’m going to offer an evidence-based assessment of its cause.

A premise of this explanation, in fact, is that the public does trust science.

As reflected in the sorts of attitudinal items in the NSF indicators and other sources, members of the public in the U.S. overwhelmingly recognize the authority of science and agree that the individual and collective decisionmaking should be informed by the best available scientific evidence.

But diverse members of the public, I’ll argue, distrust one another when they perceive that the status of the cultural groups they belong to are being adjudicated by the state’s adoption of a policy or law premised on a disputed risk or comparable fact.

When risks and other facts that admit of scientific investigation become the focus of cultural status competition, members of opposing groups will be unconsciously motivated to construe all manner of evidence in a manner that reinforces their commitment to the positions that predominate within their respective groups.

One source of evidence—indeed, the most important one—will be the weight of opinion among expert scientists.

As a result, culturally diverse people, all of whom trust scientists but who disagree with one another’s intentions on policy issues that have come to symbolize clashing worldviews, will end up culturally polarized over what scientists believe about the factual presuppositions of each other's position.

That is the science communication problem.

I will present evidence from two (NSF-funded!) studies that support this account.

3.  Cultural cognition of scientific consensus

The first was an experiment on how cultural cognition influences perceptions of scientific consensus on climate change, nuclear waste disposal, and the effect of “concealed carry” laws.

The cultural cognition thesis holds that individuals can be expected to form perceptions of risk and like facts that reflect and reinforce their commitment to identity-defining affinity groups.

For the most part, Individuals have a bigger stake in forming identity-congruent beliefs on societal risks than they have in forming best-evidence-congruent ones. If a person makes a mistake about the best evidence on climate change, for example, that won’t affect the risk that that individual or anyone he or she cares about faces: as a solitary individual, that person’s behavior (as consumer, voter, etc.) is too inconsequential to have an impact.

But if that person makes a “mistake” in relation to the view that dominates in his or her affinity group, the consequences could be quite dire indeed.  Given what climate change beliefs now signify about one’s group membership and loyalties, someone who forms a culturally non-conformity view risks estrangement from those on whose good opinion that person’s welfare—material and emotional—depends.

It is perfectly rational, in these circumstances, for individuals to engage information in a manner that reliably connects their beliefs to their cultural identities than to the best scientific evidence. Indeed, experimental evidence suggests that the more proficient that person’s critical reasoning capacities, the more successful he or she will be in fitting all manner of evidence to the position that expresses his or her group identity.

What most scientists in a particular field believe is one such form of evidence.  So we hypothesized that culturally diverse individuals would construe evidence of what experts believe in a biased fashion supportive of the position that predominates in their respective groups.

In the experiment, we showed study subjects the pictures and resumes of three highly credentialed scientists and asked whether they were “experts” (as one could reasonably have inferred from their training and academic posts) in the domains of climate change, nuclear power, and gun control.

Half the subjects were shown a book expert in which the featured scientist took the “high risk” position on the relevant issue (“scientific consensus that humans are causing climate change”; “deep geologic isolation of nuclear wastes is extremely hazardous”; “permitting citizens to carry concealed guns in public increases crime”), and half a book excerpt in which the same scientist too the “low risk” position (“evidence on climate change inconclusive”; “deep geologic isolation of nuclear wastes poses no serious hazards”; “allowing citizens to carry concealed guns reduces crime”).

If the featured scientist’s view matched the one dominant in a subject’s cultural group, the subject was highly likely to deem that scientists an “expert” whose views a reasonable citizen would take into account. 

But if that same scientist was depicted as taking the position contrary to the one that was dominant in the subject’s group, then she was highly likely to perceive that the scientist lacked expertise on the subject in question.

This result was consistent with our hypotheses.

If individuals in the real-world selectively credit or discredit evidence on “what experts believe” in this manner, then individuals of diverse cultural outlooks will end up polarized on what scientific consensus is.

And this is exactly the case.  In an observational component of the study, we found that the vast majority of subjects perceived “scientific consensus” to be consistent with the position that was dominant among members of their respective cultural groups.

Judged in relation to National Academy of Sciences “expert consensus” reports, moreover, all of the opposing cultural groups turned out to be equally bad in discerning what the weight of scientific opinion was across these three issues.

In sum, they all agreed that policy should be informed by the weight of expert scientific opinion. 

But because policies in question turned on disputed facts symbolically associated with membership in opposing groups, they were motivated by identity-protective cognition to assess evidence of what scientists believe in a biased fashion.

4.  The cultural credibility heuristic

The second study involved perceptions of the risks and benefits of the HPV vaccine.

The CDC’s 2006 recommendation that the vaccine be added to the schedule of immunizations required as a condition of middle school enrollment, although only for girls, provoked intense political controversy across the U.S. in the years immediately thereafter.

In our study, we found that there was very mild cultural polarization on the safety of the HPV vaccine among subjects’ whose views were solicited in a survey.

The degree of cultural polarization was substantially more pronounced, however, among subjects who were first supplied with balanced information on the vaccines’ potential risks and expected benefits.  Consistent with the cultural cognition thesis, the subjects were selectively crediting and discrediting the information we supplied in patterns that reflected their stake in forming identity-supportive beliefs.

But still another group of subjects assessed the risks and benefits of the HPV vaccine after being furnished the same information from debating “public health experts.” These “experts” were ones whose appearances and backgrounds, a separate pretest had shown, would induce study subjects to competing cultural identities to them.

In this experiment condition, subjects’ assessments of the risks and benefits of the HPV vaccine turned decisively on the degrees of affinity between the perceived cultural identities of the experts and the study subjects’ own identities.

If subjects observed the position that they were culturally predisposed to accept being advanced by the “expert” they were likely to perceive as having values akin to theirs, and the position they were predisposed to reject being advanced by the “expert” they were likely to perceive as having values alien to their own, then polarization was amplified all the more.

But where subjects saw the expert they were likely to perceive as sharing their values advancing the position they were predisposed to reject, and the expert they were likely to perceive as holding alien values advancing the position they were predisposed to accept, subjects of diverse cultural identities flipped positions entirely.

The subjects, then, trusted the scientific experts.

Indeed, polarization disappeared when experts whom culturally diverse subjects trusted told them the position they were predisposed to accept was wrong.

But the subjects remained predisposed to construe information in a manner protective of their cultural identities.

As a result, when they were furnished tacit cues that opposing positions on the HPV vaccine risks corresponded to membership in competing cultural groups, they credited the expert whose values they tacitly perceived as closest to their own—a result that intensified polarization when subjects' predispositions were reinforced by those cues.

5.  A prescription

The practical upshot of these studies is straightforward.

To translate public trust in science into convergence on science-informed policy, it is essential to protect decision-relevant science from entanglement in culturally antagonistic meanings.

No risk issue is necessarily constrained to take on such meanings

There was nothing inevitable, for example, about the HPV vaccine becoming a focus of cultural status conflict.  It could easily, instead, have been assimilated uneventfully into public health practice in the same manner as the HBV vaccine.  Like the HPV vaccine, the HBV vaccine immunizes recipients against a sexually transmitted disease (hepatitis-b), was recommended for universal adolescent vaccination by the CDC, and thereafter was added to the school-enrollment schedules of nearly every state.

The HBV vaccine had uptake rates of over 90% during the years in which the safety of the HPV vaccine was a matter of intense, and intensely polarizing, political controversy in the U.S.

The reason HPV ended up becoming suffused with antagonistic cultural meanings had to do with ill-advised decisions, pushed for by the vaccine’s manufacturer and acquiesced in without protest by the FDA, that made it certain that members of the public would learn about the vaccine for the first time not from their pediatricians, as they had with the HBV vaccine, but from news reports on the controversy occasioned by a high-profile, nationwide campaign to secure legislative enactments of a “girls’ only STD shot” as a condition of school enrollment.

The risks associated with introducing the HPV vaccine in this manner were not only foreseeable but foreseen and even empirically studied at the time.

Warnings about this danger were not so much rejected as never considered—because there is no mechanism in place in the regulatory process for assessing how science-informed policymaking interacts with cultural meanings.

The U.S. is a pro-science culture to its core.

But it lacks a commitment to evidence-based methods and procedures for assuring that what is known to science becomes known to those whose decisions, individual and collective, it can profitably inform.

The “declining trust in science” trope is itself a manifestation of our evidence-free science communication culture.

Those who want to solve the science communication problem should resist this & all the other just-so stories that are offered as explanations of it.

They should also steer clear of those drawn to the playground-quality political discourse that features competing tallies of whose “side” is “more anti-science.”

And they should instead combine their energies to the development of a new political science of science communication that reflects an appropriately evidence-based orientation toward the challenge of enabling the members of a pluralistic liberal society to reliably recognize what’s known by science.




Still more evidence of my preternatural ability to change people's minds: my refutation of Krugman's critique of Klein's article convinces Klein that Krugman's critique was right

That's not Harmon Killebrew, is it?! Nah...Huh.

Well, I actually agree 70% w/ what Klein says; once I explain why, I predict Klein will thoughtfully disagree -- and end up more-or-less where I was in my post on Krugman's "symmetry proof."

But I don't have time to go into this now (am busy w/ field experiments aimed at counteracting the motivated reasoning of cultural anti-cat zealots).  Will write something on this "tomorrow." 

In meantime, maybe someone else will explain why I was 100% right (everyone who commented on the Krugman post definitely felt that way).


What you "believe" about climate change doesn't reflect what you know; it expresses *who you are*

More or less the remarks I delivered yesterday at Earthday "Climate teach in/out" at Yale University:

I study risk perception and science communication.

I’m going to tell you what I regard as the single most consequential insight you can learn from empirical research in these fields if your goal is to promote constructive public engagement with climate science in American society. 

It's this:

What people “believe” about global warming doesn’t reflect what they know; it expresses who they are.

Accordingly, if you want to promote constructive public engagement with the best available evidence, you have to change the meaning of the climate change.

You have to disentangle positions on it from opposing cultural identities, so that people aren't put to a choice between freely appraising the evidence and being loyal to their defining commitments.

I’ll elaborate, but for a second just forget climate change, and consider another culturally polarizing science issue: evolution.

About every two years, a major polling organization like Gallup issues a public opinion survey showing that approximately 50% of Americans “don’t believe in evolution.” 

Pollsters issue these surveys at two-year intervals because apparently that’s how long it takes people to forget that they’ve already been told this dozens of times.  Or in any case, every time such a poll is released, the media and blogosphere is filled with expressions of shock, incomprehension, and dismay.

“What the hell is wrong with our society’s science education system?,” the hand-wringing, hair-pulling commentators ask.

Well, no doubt a lot.

But if you think the proportion of survey respondents who say they “believe in evolution” is an indicator of the quality of the science education that people are receiving in the U.S., you are misinformed.

Do you know what the correlation is between saying “I believe in evolution” and possessing even a basic understanding of “natural selection,” “random mutation,” and “genetic variance”—the core elements of the modern synthesis in evolutionary science?


Those who say they “do believe” are no more likely to be able to give a high-school biology-exam-quality account of how evolution works than those who say they “don’t.”

In a controversial decision in 2010, the National Science Foundation in fact proposed removing from its standard science-literacy test the true-false question “human beings developed from an earlier species of animals.”

The reason is that giving the correct answer to that question doesn’t cohere with giving the right answer to the other questions in NSF’s science-literacy inventory.

What that tells you, if you understand test-question validity, is that the evolution item isn’t measuring the same thing as the other science-literacy items.

Answers to those other questions do cohere with one another, which is how one can be confident they are all validly and reliably measuring how much science knowledge that person has acquired.

But what the NSF “evolution” item is measuring, researchers have concluded, is test takers’ cultural identities, and in particular the significance of religiosity in their lives.

What’s more, the impact of science literacy on the likelihood that people will say they “believe in evolution” is in fact highly conditional on their identity: as their level of science comprehension increases, individuals with a highly secular identity become more likely to say “they believe” in evolution; but as those with a highly religious identity become more science literate, in contrast, they become even more likely to say they don’t.

What you “believe” about evolution, in sum, does not reflect what you know about science—in general, or in regard to the natural history of human beings.

Rather it expresses who you are.

Okay, well, exactly the same thing is true on climate change.

You’ve all seen the polls, I’m sure, showing the astonishing degree of political polarization on “belief” “human-caused” global warming.

Well, a Pew Poll last spring asked a nationally represented sample, “What gas do most scientists believe causes temperatures in the atmosphere to rise? Is it carbon dioxide, hydrogen, helium, or radon?”

Approximately 60% got the right answer to that question.

And there was zero correlation between getting it right and being a Democrat or Republican.

The percentage of Democrats who say they “believe” in global warming is substantially higher than 65%: it’s over 80%, which means that a good number of Democrats who say they “believe” in global warming don’t understand the most basic of all facts known to climate science.

The percentage of Republicans who say they don’t believe in global warming is a lot lower than 65%. Only about 25% say they believe human beings have caused global temperatures to rise in recent decades, according to Pew and other researchers. 

That means that a large fraction of the Republicans who tell pollsters they “don’t believe” in human-caused global warming do in fact know the most important thing there is to understand about climate change: that adding carbon to the atmosphere causes the temperature of the earth to increase.

Do you know what the correlation is between science literacy and “belief” in human-caused global warming?

You get half credit for saying zero.

That’s the right answer for a nationally representative sample as a whole.

But it’s a mistake to answer the question without dividing the sample up along cultural or comparable lines: as their score on one or another measure of science comprehension goes up, Democrats become more likely, and Republicans less, to say they “believe” in human-caused global warming.

Like saying “I do/don’t believe in evolution,” saying I “do/don’t believe in climate change” doesn’t convey what you know about science—generally, or in relation to the climate.

It expresses who you are.

Al Gore has described the climate change debate as a “struggle for the soul of America.

He’s right.

But that’s exactly the problem.  Because in “battles for the soul” of America, the stake that culturally diverse individual have in forming beliefs consistent with their group identity dominates the stake they have in forming beliefs that fit the best available evidence.

In saying that, moreover, I’m not talking about whatever interest people have in securing comfortable accommodations in the afterlife. I’m focused entirely on the here and now.

Look: What an ordinary individual believes about the “facts” on climate change has no impact on the climate.

What he or she does as a consumer, as a voter, or as a participant in public debate is just too inconsequential to have an impact.

No mistake that individual makes about the science on climate change, then, is going to affect the risk posed by global warming for him or her or for anyone else that person cares about.

But if he or she takes the “wrong” position in relation to his or her cultural group, the result could be devastating for her, given what climate change now signifies about one’s membership in and loyalty to opposing cultural groups.

It could drive a wedge—material, emotional, and psychological—between individual the people whose support are indispensable to his or her well-being.

In these circumstances, we should expect a rational person to engage information in a manner geared to forming and persisting in positions that are dominant within their cultural groups. And the better they are at making sense of complex information—the more science comprehending they are –the better they’ll do at that. 

That’s what we see in lab experiments.  And it’s why we see polarization on global warming intensifying in step with science literacy in the real world.

But while that’s the rational way for people to engage information as individuals, given what climate change signifies about their cultural identities, it’s a disaster for them collectively.  Because if everyone does this at the same time, members of a culturally diverse democratic society are less likely to converge on scientific evidence that is crucial to the welfare of all of them.

And yet that by itself doesn’t make it any less rational for individuals to attend to information in a manner that reliably connects them to the position that is dominant in their group.

This is a tragedy of the commons problem—a tragedy of the science communications commons.

If we want to overcome it, then we must disentangle competing positions on climate change from opposing cultural identities, so that culturally pluralistic citizens aren’t put in the position of having to choose between knowing what’s known to science and being who they are.

Only that will dissolve the conflict citizens now face between their personal incentive to form identity-consistent beliefs and the collective one they have in recognizing and giving effect to the best available evidence.

Science educators, by the way, have already figured this out about evolution. They’ve shown you can in fact teach the elements of the modern synthesis-- random mutation, genetic variance and natural selection—just as readily to students whose identities cohere with saying they “don’t believe” in evolution as you can to students whose identities cohere with saying they do. You just can’t expect the former to “I believe in evolution” after.

Indeed, you must take pains not to confuse understanding evolutionary science with the “pledge of cultural allegiance” that “I believe in evolution” has become.

You must remove from the education environment the toxic cultural meanings that make answers to that question badges of membership in and loyalty to one’s cultural group.  The meanings that fuel the pathetic spectacle of hand-wringing and hair-pulling that occurs every time Gallup or another organization issues its “do you believe in evolution” survey results.

All the diverse groups that make up our pluralistic democracy are amply stocked with science knowledge.

They are amply stocked with public spirit too. 

That means you, as a science communicator, can enable these citizens to converge on the best available evidence on climate change.

But to do it, you must banish from the science communication environment the culturally antagonistic meanings with which positions on that issue have become entangled—so that citizens can think and reason for themselves free of the distorting impact of identity-protective cognition.

If you want to know what that sort of science communication environment looks like, I can tell you where you can see it: in Florida, where all 7 members of the Monroe County Board of Commissioners -- 4 Democrats, 3 Republicans -- voted unanimously to join Broward County (predominantly Democratic), Monroe County (predominantly Republican), and Miami-Dade County (predominantly Republican) in approving the Southeast Climate Compact Action plan, which, I quote from the Palm Beach County Board summary, “includes 110 adaptation and mitigation strategies for addressing seal-level risk and other climate issues within the region.”

I’ll tell you another thing about what you’ll see if you make this trip: the culturally pluralistic, and effective form of science communication happening in southeast Florida doesn’t look anything  like the culturally assaultive "us-vs-them" YouTube videos and prefabricated internet comments with which Climate Reality and Organizing for American are flooding national discourse.

And if you want to improve public engagement with climate science in the United States, the fact that advocates as high profile and as highly funded as that still haven’t figured out the single most important lesson to be learned from the science of science communication should make you very sad.


No, I don't think "cultural cognition is a bad thing"; I think a *polluted science communication environment* is & we should be using genuine evidence-based field communication to address the problem

Stenton Benjamin Danielson has a characteristically thoughtful post, 95% of which I agree with, on cultural cognition, "public opinion," and promoting constructive public engagement with climate science.  But of course the 5%-- which has to do with whether I think "cultural cognition" is a "bad thing" that is to be overcome rather than a dynamic to be deployed to promote such engagement -- sticks in my craw!  Maybe this response will get us closer to 100% agreement--if not by moving him a full 5% in my direction, then maybe by  provoking him to elaborate & thereby move me some fraction of the remainder toward his point of view.

 So read what he says.  Then read this:

Part of the problem, I'm sure, is that I'm an imperfect communicator.

Another is the infeasibility of saying everything one believes every time one says anything.

But it is simply not the case that I view

cultural cognition as unreservedly bad -- a sort of disease or pollution in our debate about an issue, something to be prevented or neutralized whenever possible so that we can make rational assessments of the evidence.

On the contrary, I view it is an indispensable element of rational thought, one that contributes in a fundamental way to the capacity of individuals to participate in, and thus extend, collective knowledge. See generally:  

Cultural cognition conduces to persistent states of public controversy over what's known only in a polluted science communication environment: one in which antagonistic cultural meanings become attached to positions on risk and policy-relevant facts, and transform them into badges of membership in opposing cultural groups.  

That's not normal.  It is a pathology that disables rational thought precisely because it disconnects cultural cognition from discernment of the best available evidence.

We can treat this pathology, and better still avoid the occurrence of it, through evidence-based science-communication-environment protection practices

See generally:  

I also agree, by the way, that "messaging" campaigns aimed at influencing "public opinion" generally are an absurd waste of time, not to mention waste of the money of those eager to support climate-science communication efforts.  This approach to "science communication" not only reflects a psychologically unrealistic account of how people come to know what's known by science but betrays an elementary-school level of comprehension of basic principles of political economy

Don't "message" people with "struggle for the soul of America" appeals. 

Show them that engaging climate science is "normal" by enabling them to see that people they recognize as competent and informed are using it to guide their practical decisions.  That is how ordinary people -- very rationally -- recognize how to orient themselves appropriately with the best available evidence on all manner of issues. 

Understanding the contribution that cultural cognition makes to individuals' rational apprehension of what is known is, I believe, is indispensable to that strategy for promoting constructive public engagement with climate science.  I'm glad to see that you agree with me on that -- even if you hadn't discerned that I agree with you! 

Those "risk experts" who want to contribute, moreover, should stop telling just-so stories-- give up the facile "take-'biases'-&-'heuristics'-literature-add-water-&-stir" form of "instant decision science"-- and go to the places where real people are trying to figure out how to use climate science to make their lives better.

Go there and genuinely help them by systematically testing their experience-informed hypotheses about how to reproduce in the world the sorts of things that experimental methods using cultural cognition and other theories suggest will improve public engagement with climate science.

We don't need more stylized lab experiments that try to convince us that things that real-world evidence manifestly show won't work actually will if we just keep doing them (followed when they don't by whinging about "the forces of evil" who--as was perfectly foreseeable--told members of the public whom you were targetting not to believe your "message").

Climate scientists update their models to reflect ten years of data.  Climate advocates should too.  




Want to improve climate-science communication (I mean really, seriously)? Stop telling just-so stories & conducting "messaging" experiments on MTurk workers & female NYU undergraduates & use genuine evidence-based methods in field settings instead

From Kahan, D., "Making Climate Science Communication Evidence-based—All the Way Down," in Culture, Politics and Climate Change, eds. M. Boykoff & D. Crow, pp. 203-21. (Routledge Press, 2014):

a. Methods. In my view, both making use of and enlarging our knowledge of climate science communication requires making a transition from lab models to field experiments. The research that I adverted to on strategies for counteracting motivated reasoning consist of simplified and stylized experiments administered face-to-face or on-line to general population samples. The best studies build explicitly on previous research—much of it also consisting in stylized experiments—that have generated information about the nature of the motivating group dispositions and the specific cognitive mechanisms through which they operate. They then formulate and test conjectures about how devices already familiar to decision science—including message framing, in-group information sources, identity-affirmation, and narrative—might be adapted to avoid triggering these mechanisms when communicating with these groups.[1]

But such studies do not in themselves generate useable communication materials. They are only models of how materials that reflect their essential characteristics might work. Experimental models of this type play a critical role in the advancement of science communication knowledge: by silencing the cacophony of real-world influences that operate independently of anyone’s control, they make it possible for researchers to isolate and manipulate mechanisms of interest, and thus draw confident inferences about their significance, or lack thereof. They are thus ideally suited to reducing the class of the merely plausible strategies to ones that communicators can have an empirically justified conviction are likely to have an impact. But one can’t then take the stimulus materials used in such experiments and send them to people in the mail or show them on television and imagine that they will have an effect.

Communicators are relying on a bad model if they expect lab researchers to supply them with a bounty of ready-to use strategies. The researchers have furnished them something else: a reliable map of where to look for them. Such a map will (it is hoped) spare the communicators from wasting their time searching for nonexistent buried treasure. But the communicators will still have to dig, making and acting on informed judgments about what sorts of real materials they believe might reproduce these effects outside the lab in the real-world contexts in which they are working.

The communicators, moreover, are the only ones who can competently direct this reproduction effort. The science communication researchers who constructed the models can’t just tell them what to do because they don’t know enough about the critical details of the communication environment: who the relevant players are, what their stakes and interests might be, how they talk to each other, and whom they listen to. If researchers nevertheless accept the invitation to give “how to” advice, the best they will be able to manage are banalities—“Know your audience!”; “Grab the audience’s attention!”—along with Goldilocks admonitions such as, “Use vivid images, because people engage information with their emotions. . . but beware of appealing too much to emotion, because people become numb and shut down when they are overwhelmed with alarming images!”

Communicators possess knowledge of all the messy particulars that researchers not only didn’t need to understand but were obliged to abstract away from in constructing their models . Indeed, like all smart and practical people, the communicators are filled with many plausible ideas about how to proceed—more than they have the time and resources to implement, and many of which are not compatible with one another anyway. What experimental models—if constructed appropriately—can tell them is which of their surmises rest on empirically sound presuppositions and which do not. Exposure to the information such modeling yields will activate experienced-informed imagination on the communicators’ part, and enable them to make evidence-based judgments about which strategies they believe are most likely to work for their particular problem.

At that point, it is time for the scientist of science communication to step back in—or to join alongside the communicator. The communicator’s informed conjecture is now a hypothesis to be tested. In advising field communicators, science of science communication researchers should treat what the communicators do as experiments. Science communication researchers should work with the communicator to structure their communication strategies in a manner that yields valid observations that can be measured and analyzed.

Indeed, communicators, with or without the advice of science of science communication researchers, should not just go on blind instinct. They shouldn’t just read a few studies, translate them into a plausible-sounding plans of action, and then wing it. Their plausible surmises about what will work will be more plausible, more likely to work, than any that the laboratory researchers, indulging their own experience-free imaginations, concoct. But they will still be only plausible surmises. Still be only hypotheses. Without evidence, we will not learn whether policies based on such surmises did or didn’t work. If we don’t learn that, we won’t learn how to do even better.

Genuinely evidence-based science communication must be based on evidence all the way down. Communicators should make themselves aware of the existing empirical information that science communication researchers have generated (and steer clear of the myriad stories that department-store consumers of decision science work tell) about why the public is divided on climate science. They should formulate strategies that seek to reproduce in the world effects that have been shown to help counter the dynamics of motivated reasoning responsible for such division. Then, working with empirical researchers, they should observe and measure. They should collect appropriate forms of pretest or preliminary data to try corroborate that the basis for expecting a strategy to work is sound and to calibrate and refine its elements to maximize its expected effect. They should also collect and analyze data on the actual impact of their strategies once they’ve been deployed.

Finally, they should make the information that they have generated at every step of this process available to others so that they can learn from it to. Every exercise in evidence-based science communication itself generates knowledge. Every such exercise itself furnishes an instructive model of how that knowledge can be intelligently used. The failure to extract and share the intelligence latent in doing science communication perpetuates the dissipation of collective knowledge that it is the mission of the science of science communication to staunch. 


[1] Unrepresentative convenience samples are unlikely to generate valid insights on how to counteract motivated reasoning. Samples of college undergraduates are perfectly valid when there is reason to believe the cognitive dynamics involved operate uniformly across the population. But the mechanisms through which motivated reasoning generates polarization on climate change don’t; they interact with diverse characteristics—worldviews and values, but also gender, race, religiosity, and even regions of residence. It is known, for example, that white males who are highly hierarchical and individualistic in worldviews or conservative in their political ideologies, and who are likely to live in the South and far west, tend to react dismissively to information about climate change (McCright & Dunlap 2013, 2012, 2011; Kahan, Braman, Gastil, Slovic & Mertz 2007). Are they likely to respond to a “framing” strategy in the same way that a sample of predominantly female undergraduates attending a school in New York City does (Feygina, Jost & Goldsmith 2010)? If not, that’s a good reason to avoid using such a sample in a framing study, and not to base practical decisions on any study that did.


Vaccine risk perceptions and risk communication: study conclusions & recommendations

From CCP's "Vaccine Risk Perceptions and Ad Hoc Risk Communication: An Empirical Assessment" report: 

II. Summary conclusions

A. Findings

1.   There is deep and widespread public consensus, even among groups strongly divided on other issues such as climate change and evolution, that childhood vaccinations make an essential contribution to public health. A very large supermajority believes that the benefits of childhood vaccinations outweigh their risks and that public health generally would suffer were vaccination rates to fall short of the goals set by public health authorities.

2.   In contrast to other disputed science issues, public opinion on the safety and efficacy of childhood vaccines is not meaningfully affected by differences in either science comprehension or religiosity. Public controversies over science, including those over evolution and climate change, often feature conflict among individuals of varying levels of religiosity, whose difference of opinion intensify in proportion to their level of science comprehension. There is no such division over vaccine risks and benefits.

3.   The public’s perception of the risks and benefits of vaccines bears the signature of a generalized affective evaluation, which is positive in a very high proportion of the population. The high degree of coherence in responses to items relating to the contribution that childhood vaccinations make to public health strongly implies that public assessments of vaccine risks and benefits reflect a unitary latent affective orientation. The distribution of that orientation is strongly skewed in a positive direction—indicating that a substantial majority of the population (in the vicinity of 75%) has a positive attitude toward childhood vaccines.

4.   Among the manifestations of the public’s positive orientation toward childhood vaccines is the perception that vaccine benefits predominate over vaccine risks and a high degree of confidence in the judgment of public health officials and experts. By large supermajorities, the survey participants endorsed the proposition that vaccine benefits outweigh their risks, and rejected claims that deterioration in vaccination coverage would pose no serious public health danger. They also expressed confidence in the judgment of officials who identify which vaccinations should be universally administered, and in the judgment of experts that vaccines are safe.

5.   Perceptions of the relationship between vaccines and specified diseases reflect the same positive affective orientation that informs public perceptions of the contribution that childhood vaccines make to public health generally. Responses to items on the link between vaccines and autism, cancer, diabetes—as well as a fictional disease not asserted by anyone to be connected to childhood vaccinations—displayed the same pattern as the responses to all the other public-health items. Under these circumstances, responses to these items can confidently be viewed only as indicators of the same latent affective attitude reflected in the public’s assessments of the contribution childhood vaccines make to public health generally. Public health officials should resist the mistake of construing responses to survey items such as these as measuring public knowledge about or beliefs on specific issues relating to childhood vaccinations.

6.   The demographic characteristics and political outlooks typically associated with group conflict over risk and related aspects of decision-relevant science are not meaningfully associated with disagreement about childhood-vaccination risks. Members of all such groups believe that vaccine risks are low, vaccine benefits high, and mandatory vaccination policies appropriate. Those who believe otherwise are outliers in every one of these groups.

7.   There is no meaningful association between concern over vaccine risks and the sharp cultural cleavage that characterizes perceptions of either “public safety risks,” a cluster of putative hazards associated with environmental issues and gun control, or “social deviancy risks,” a cluster associated with legalization of marijuana and prostitution and with teaching high school students about birth control. The opposing cultural allegiances that are associated with disputed societal and public health risks do not generate meaningful disagreement over vaccine risks and benefits. At most, such dispositions mildly influence the intensity with which culturally diverse members of the public approve of childhood vaccination.

8.   Existing universal vaccination policies appear to enjoy widespread support, but proposals to restrict existing grounds for exemption divide the public along partisan lines. Despite support for universal vaccination policies and widespread disapproval of parents who refuse to permit vaccination of their children based on concerns about vaccine risks, proposals to restrict or eliminate moral or religious grounds for opting out of vaccination requirements provoke dissensus along largely partisan lines consistent with citizens’ general orientation toward government regulation.

9.   The public generally underestimates vaccination rates and overestimates the rate of exemption. Only 9% of the survey respondents recognized that the vaccination rate among U.S. children aged 19-35 months for recommended childhood vaccinations has been over 90% in recent years. The median estimate was between 70-79%. The median estimate of children receiving no vaccinations was 2-10%; only 9% correctly indicated that less than 1% of children aged 19-35 months receive none of the recommended childhood vaccinations.

10.  Communications that assert the existence of growing concern over vaccination risks and declining vaccination rates magnify misestimations of vaccination rates and of exemptions. Experiment subjects who read communications patterned on real media communications underestimated vaccine coverage by an even larger amount than subjects in the control.

11.  Communications that connect “growing concern” over vaccine risks to disbelief in evolution and climate change generate cultural polarization. Relative to their counterparts in a control condition, experiment subjects exposed to such a communication divided along lines that reflected their predispositions toward currently disputed societal risks.

12.  Factually accurate information on vaccine rates, when issued by the CDC, substantially corrects underestimation of vaccination rates. Exposure to a story patterned on the press statements that the CDC typically issues in connection with annual NIS updates resulted in a significant correction of experiment subjects’ underestimation of national vaccination coverage.

B. Normative and prescriptive conclusions

1.   Risk communicators—including journalists, advocates, and public health professionals—should refrain from conveying the false impression that a substantial proportion of parents or of the public generally doubts vaccine safety. Such information risks creating anxiety rather than dispelling it. Moreover, by aggravating underestimation of vaccination rates, communications of this nature obscure a signal that conveys public confidence in vaccine safety and stimulates reciprocal motivations to contribute to the collective good of herd immunity.

2.   Risk communicators should avoid resort to the factually unsupportable, polemical trope that links vaccine risk concerns to climate-change skepticism and to disbelief in evolution as evidence of growing societal distrust in science. Such rhetoric, in addition to being facile, risks generating an affective or symbolic link between vaccines and issues on which cultural polarization is currently a significant impediment to public science communication.

3.   Risk communicators, including public health officials and professionals, should aggressively disseminate true information on the historically and continuing high rates of childhood vaccination in the U.S. The high levels of vaccination in the U.S. are a science communication resource. That resource should be exploited, not obscured or dissipated.

4.   Because there is a chance that it would make mandatory vaccination policies a matter of partisan contestation, campaigns to promote legislative elimination or contraction of existing grounds for exemptions should be viewed with extreme caution. There is reason to believe—from real-world experience as well as the results of this study—that proposals to restrict nonmedical exemptions from existing mandates would generate partisan division in the public. As evidenced by the controversy over the HPV vaccine, such divisions disrupt the processes by which ordinary citizens recognize and orient themselves with respect to the best-available evidence on public-health and other risks. Accordingly, the potential for creating polarization over childhood vaccination risks is a cost that must be balanced against whatever benefit might be obtained from reforms in law aimed at reducing the already very low percentage of parents that exempt their children from mandatory vaccination.

5.   Vaccine-risk assessments and communication should not be based on creative extrapolations from general theories. Because decision-science mechanisms can be imaginatively manipulated to support a wide variety of explanations and prescriptions, it is a mistake to present theoretical syntheses of work in this field as a guide for action. Instead, conjectures informed by decision-science frameworks should be treated as hypotheses for empirical investigation.

6.   Hypotheses relating to vaccine-risk perceptions and vaccine-risk communication should be tested with valid empirical methods specifically suited to measuring matters of consequence. Opinion polls cannot be expected to generate significant insight into vaccine risk perceptions, either on the part of parents, whose responses are unreliable indicators of behavior, or the general public, in whom demographic and attitudinal measures fail to explain practically meaningful levels of variance. Rather, behavioral measures (including validated attitudinal indicators of behavior) should be used to gauge parental risk concern and fine-grained, local methods used to investigate the characteristics of enclaves of demonstrated vaccine hesitancy.

7.   The public health establishment should take the initiative to develop comprehensive proposals for better integrating the science of science communication into its culture and practices. Procedures should be adopted, within government public health agencies and within the medical profession, for making use of the best available empirical methods for anticipating and averting influences that distort public risk perceptions. The public health establishment should also propagate professional norms geared to curbing ill-informed and ill-considered forms of ad hoc risk perception by the media and by individual members of the public-health establishment. The most effective step to discouraging this form of feral risk communication is to populate the niche it now occupies with an empirically informed and systematically planned alternative.

Page 1 ... 7 8 9 10 11 ... 29 Next 20 Entries »