follow CCP

Recent blog entries
popular papers

What Is the "Science of Science Communication"?

Climate-Science Communication and the Measurement Problem

Ideology, Motivated Cognition, and Cognitive Reflection: An Experimental Study

'Ideology' or 'Situation Sense'? An Experimental Investigation of Motivated Reasoning and Professional Judgment

A Risky Science Communication Environment for Vaccines

Motivated Numeracy and Enlightened Self-Government

Ideology, Motivated Cognition, and Cognitive Reflection: An Experimental Study

Making Climate Science Communication Evidence-based—All the Way Down 

Neutral Principles, Motivated Cognition, and Some Problems for Constitutional Law 

Cultural Cognition of Scientific Consensus

The Tragedy of the Risk-Perception Commons: Science Literacy and Climate Change

"They Saw a Protest": Cognitive Illiberalism and the Speech-Conduct Distinction 

Geoengineering and the Science Communication Environment: a Cross-Cultural Experiment

Fixing the Communications Failure

Why We Are Poles Apart on Climate Change

The Cognitively Illiberal State 

Who Fears the HPV Vaccine, Who Doesn't, and Why? An Experimental Study

Cultural Cognition of the Risks and Benefits of Nanotechnology

Whose Eyes Are You Going to Believe? An Empirical Examination of Scott v. Harris

Cultural Cognition and Public Policy

Culture, Cognition, and Consent: Who Perceives What, and Why, in "Acquaintance Rape" Cases

Culture and Identity-Protective Cognition: Explaining the White Male Effect

Fear of Democracy: A Cultural Evaluation of Sunstein on Risk

Cultural Cognition as a Conception of the Cultural Theory of Risk


Cross-cultural cultural cognition road trip

Here's my schedule for next week and a half -- or at least parts of it.  


Stop by if in the neighorhood -- otherwise I'll send postcard reports now & again!

(Actually, I'm surprised that I'm giving the same talk at Cardiff & Nottingham--but I doubt that I really will!)


Science journalists: Ask not what the science of science communication can do for you . . . 

A reflective correspondent & friend wrote to me to ask what I made of the relative inattention of science journalists to the empirical study of science communication--& what might be done to remedy this.  She had many great ideas for how to make such work more familiar and accessible to them.  I had a somewhat different, but I think complementary reaction:

I think it is unsurprising how infrequently empirical research is featured in social media and similar fora in which science journalists exchange ideas.

The explanation, moreover, isn't merely that how to communicate to curious members of the public is only 1 of the n things that science of science communication studies. It's that those who are engaged in scientifically studying science communication -- including the sorts science journalists do -- aren't trying to answer the questions that journalists most often are, and should be, asking.  

The journalists' questions relate to their own craft norms -- the professional understandings that they absorb and generate and transmit and that guide and animate them.  They argue about various ones of them all the time, in many cases persistently (or at least intermittently; they have jobs—very interesting ones!) over long periods of time.

That means that they have questions that in the judgment of those endowed with the requisite, experience-informed professional judgment admit of more than one plausible (but not, the debate presupposes, more than one correct or best) answer.  

Under those circumstances, arguments will be interminable and make no progress. Evidence is needed -- not as a substitute for the exercise of professional judgment but as raw material for it to operate on.  

Well, very very few (maybe zero) scholars are using empirical methods to answer questions of consequence to the quality and evolution of science journalism's' craft norms.  

Most “science of #scicomm” scholars, of course, aren't studying science journalism at all.  

Others actually are-- but to answer questions that are parts of the scholarly conversations those researchers are part of.  They have converged on (or joined) collective inquiries into how one or another general mechanism—cognitive, political, or both—operate to shape the path of scientific information through the media and to the public.  Their research (much of which is excellent!) is, nearly always, trying to answer questions that admit of more than one plausible (but not more than one correct or best) answer about those processes—not about how science journalists can be excellent science journalists.

Maybe sometimes these scholars mistakenly think that what they are studying when they examine these more general dynamics of communication supplies the "answers" to the questions science journalists pose about their own craft norms. Other times they present their work this way knowing full well that it is a mistake (it's a very disturbing spectacle when they do).

In either case, science journalists react negatively -- "that's ridiculous" or (in a refrain that becomes a chorus after an event like NAS “science of #scicomm” colloquia) "that's completely irrelevant to what we do; I've not learned a thing!"  ...

Well, the problem actually isn't in the researchers here; it's in the science journalists!

The mistake is in part for them to think that "everything is about them": the science of science communication isn't one thing—it’s 7 (± 2).  

But even more fundamentally, it is a mistake for the science journalists to think that anyone besides them can be expected to create the scientific insight that is relevant to their craft!

No one else knows (or likely genuinely cares: nonjounralists don't even know enough to care) what the empirical questions of consequence to science journalism's craft norms are. No one else can reliably recognize the form of evidence that helps professional conversation about those questions to advance; only those with the sense of the professional science journalists can.

This isn't to say that individual journalists must start designing studies and collecting data.  Rather it is to say that they must exercise control over research using empirical methods so that it in fact is designed to address questions of consequence to them and uses designs that can support inferences relevant to the sorts of questions experienced science journalists recognize as admitting of more than one plausible (but not more than one correct or best) answer.  

Science journalists will often observe, correctly, that “science of #scicomm” scholars' work on general mechanisms are generating insights of indisputable relevance to their craft.  But the journalists--not the scholars--will know when that's so.  

In that situation, moreover, science journalists will be filled with hypotheses--ones that are concrete and relevant to those who share their situation sense—about how those mechanisms might interact with their professional craft norms.

Even if they did not themselves create the studies, they will recognize when one designed to test such a hypothesis is genuinely capable of supporting inferences on the basis of which they can will know more than they otherwise would have

They are the ones, then, who must direct the empirical enterprise that is the science of science communication for science journalists.


There are an infinite number of ways -- but none of them consists in passively consuming journal articles.

Here as in the other practical domains in which a science of science communication is needed, the answer of the thoughtful and honest scholar who actually wants to help when asked (over & over) by communicators to "so what should we do" is, "You tell me -- and I will help by measuring what you confirm for me is the right thing!"


Want to know what empirically *informed* vaccine risk communication looks like?

Drawing on material from the CCP Vaccine Risk and Ad Hoc Risk Communication study, the last few posts reported experimental results on the potentially deleterious effects of empirically uniformed risk communication.

By “empirically uninformed risk communication,” I mean to refer to information that accurately conveys the safety and efficacy of vaccines but that embeds that information in mischaracterizations of the extent, nature and consequences of public hostility to universal childhood immunization.  Ad hoc risk communication of this sort—which abounds in the media and on the internet—itself can produce misunderstandings that undermine the motivation to cooperate with universal immunization programs and that drag childhood vaccinations into the reason-effacing maelstrom of cultural conflict (Kahan 2013).

What’s more, this style of risk communication distracts those who want to promote public understanding of vaccine safety from the need to perfect empirically informed strategies for achieving this critical goal.

Such research is well underway.

As discussed in the Report, it consists not in general public opinion surveying: opinion polls lack sufficient discernment to identify the sources and mechanisms of genuine vaccine hesitancy in the public, and are not a reliable or valid measure of vaccine behavior by parents.

The most valuable research now being conducted on vaccine hesitancy uses focused and fine-grained methods tied to actual behavior.

Dr. Douglas Opel and his collaborators (2011a, 2011b, 2013), e.g., have devised—and are refining—an attitudinal screening instrument that can be used to predict parents’ willingness to obtain timely vaccinations for their children.  Such a screening device would be comparable to ones used in diverse fields from credit assessment (e.g., Klinger, Khwaja & Lamonte 2013) to organizational staffing (e.g., Ones et al. 2007), not to mention to ones used to predict or diagnose disease vulnerability (e.g., Wilkins et al. 2013).

If perfected, such an instrument could be used by physicians to identify genuinely vaccine-hesitant parents, by public health administrators to detect local pockets of under-vaccination that pose a genuine public health threat, and by researchers to develop genuinely effective risk communication materials (Sadaf et al. 2013; Opel et al. 2012)—ones that can convey factually accurate information to the right people and avoid all the hazards associated with blunderbuss, empirically uninformed ad hoc risk communication.

Of course, the public health risks posed by local enclaves of under-vaccination, as well as by misinformers who sow unfounded anxiety in these and other communities, are ones that merit response right now.

The public health establishment doesn’t have as much evidence as it needs to address these dangers as effectively as it could.

But as such evidence is being developed by valid empirical research, those who favor universal childhood immunization should make intelligent use of the currently best available evidence to promote constructive and open-minded public engagement with valid information on vaccine risks and benefits.

If you want an example, I urge you to read this excellent essay from Moms Who Vax:

It may surprise you to know that the anti-vaccine movement has long claimed to speak for parents in this country when it comes to vaccines. And it is because they are so vocal and we are so, well, busy living our lives, that legislators, government officials, and even some public health organizations think that anti-vaccine activists who believe the MMR causes autism and that the decline of vaccine-preventable disease is due to "better hygiene" represent parents as a whole, when it comes to immunization in this country.

The vast--vast--majority of us choose to vaccinate our children for two reasons: one, we don't want our children to suffer from a preventable disease, possibly become seriously ill, or even die; and two, we don't want any of those things to happen to our neighbors either. Here's the problem: we don't talk about it. I suspect this is because we consider it commonsense. One mother on this blog wrote a post titled: "There's an Anti-Vaccine Movement?" because it had never occurred to her before she had children that people would willingly forgo something that has nearly eliminated one of the most dreaded diseases in human history (polio) and saved the lives of countless children and adults from other diseases that, if not kept in check by widespread immunization, cause unimaginable amounts of suffering.

We never thought we'd have to advocate for something that saves lives, especially the lives of children.

But here we are, and our complacency and our silence has allowed a fringe minority to sit at the table of public health in our place. And there are now consequences for our silence.

If I sound a little more passionate than usual, it's because I'm angry. We must rise up as a group and take back the conversation. … Right now, there are legislators in Oregon who believe that millions of parents do not believe in vaccination.... Let's prove them wrong. … Let's do this--let's go letter for letter, and beyond. Let's make sure the people who make our immunization law know that we are here, that we care, that we are the 95%.

In addition to being much more eloquent and inspiring than the boilerplate “growing crisis of confidence” and “creeping anti-science” tropes that dominate ad hoc risk communication, this essay brilliantly exploits dynamics that a reflective communicator would surmise are important based on existing, evidence-based understandings of science communication:

  • Because individuals (quite sensibly!) form their assessments of risk by observing how others who are like situated are responding (Kasperson et al. 1988), the (clear, unassailablefact that the “vast majority” of U.S. parents arrange for their children to receive all recommended immunizations is itself an important and effective piece of evidence to communicate to parents—many of whom are likely to become fearful if bombarded by thoughtless repetition of the false message that an “epidemic of fear” has led to an erosion in immunization rates. 
  • Similarly, people tend to contribute voluntarily to collective goods when they perceive that others are doing so but to refrain when they think that others are shirking or free-riding (Bowles & Gintis 2013).  So again, the message here—“we are the 95%” who contribute—is spot on.  It reinforces reciprocal motivations to contribute to the collective good of herd immunity (Hershey et al. 1994) rather than undermines them, as empirically uninformed risk communicators do by proclaiming—falsely—that a “large and growing number” of “otherwise mainstream parents” are refusing to vaccinate their children.   
  • The communication manifests the willingness of the vast majority who are contributing to the public good of herd immunity to contribute to another: condemnation of the few who are free-riding. Experimental behavioral economics shows that individuals are most likely to converge on and stick to a high-cooperation equilibrium in a collective action setting when they can observe that other individuals are moved voluntarily to accept the burden of informally punishing (e.g., by shaming) the relatively few selfish actors who free-ride.  In contrast, demands for increased, centrally administered formal punishments can vitiate reciprocal motivations by convey an expectation that the disposition to voluntarily comply is lower than it actually is (Kahan 2004)—another of the many sources of scientific insight that empirically uninformed risk communicators ignore
  • Finally, this essay is inspiringly inclusive.  It doesn’t use the cheap trick of ramping up one cultural group’s indignation by attributing socially undesirable behavior to a competing one. Characteristic of communications that—again, falsely—attribute vaccine hesitancy to one or another recognizable cultural or political group, this style of advocacy is what threatens to envelop childhood vaccines in exactly the same forms of persistent cultural conflict that inhibit public recognition of the best available evidence on myriad issues—from climate change to nuclear power to the HPV vaccine.

We need to acquire more valid empirical evidence on how to communicate vaccine risks and benefits.

But we also need to act in an informed way in the meantime.

Another of the many defects of empirically uninformed vaccine risk communication is that it diverts attention away from the most instructive and inspiring examples of how public-spirited citizens and scientists are pursuing these objectives.


Bowles, S. & Gintis, H. A cooperative species : Human reciprocity and its evolution (Princeton University Press, Princeton, 2013).

Kahan, D.M. The Logic of Reciprocity. in Moral Sentiments and Material Interests: The Foundation of Cooperation in Economic Life (ed. H. Gintis, S. Bowler & E. Fehr) 339-378 (MIT Univ. Press, Cambridge, MA, 2004).

Kahan, D.M. A risky science communication environment for vaccines. Science 342, 53-54 (2013).

Kasperson, R.E., et al. The Social Amplification of Risk: A Conceptual Framework. Risk Analysis 8, 177-187 (1988).

Klinger, B., Khwaja, A. & LaMonte, J. Improving credit risk analysis with psychometrics in Peru. (Inter-American Development Bank, 2013).

Ones, D.S., Dilchert, S., Viswesvaran, C. & Judge, T.A. In support of personality assessment in organizational settings. Personnel Psychology 60, 995-1027 (2007).

Opel, D.J., Mangione-Smith, R., Taylor, J.A., Korfiatis, C., Wiese, C., Catz, S. & Martin, D.P. Development of a survey to identify vaccine-hesitant parents: The parent attitudes about childhood vaccines survey. Human Vaccines 7, 419-425 (2011a).

Opel, D.J., Robinson, J.D., Heritage, J., Korfiatis, C., Taylor, J.A. & Mangione-Smith, R. Characterizing providers’ immunization communication practices during health supervision visits with vaccine-hesitant parents: A pilot study. Vaccine 30, 1269-1275 (2012).

Opel, D.J., Taylor, J.A., Mangione-Smith, R., Solomon, C., Zhao, C., Catz, S. & Martin, D. Validity and reliability of a survey to identify vaccine-hesitant parents. Vaccine 29, 6598-6605 (2011b).

Opel, D.J., Taylor, J.A., Zhou, C., Catz, S., Myaing, M. & Mangione-Smith, R. The relationship between parent attitudes about childhood vaccines survey scores and future child immunization status: A validation study. JAMA pediatrics 167, 1065-1071 (2013)

Sadaf, A., Richards, J.L., Glanz, J., Salmon, D.A. & Omer, S.B. A Systematic Review of Interventions for Reducing Parental Vaccine Refusal and Vaccine Hesitancy. Vaccine 31, 4293-4304 (2013)

Wilkins, C.H., Roe, C.M., Morris, J.C. & Galvin, J.E. Mild physical impairment predicts future diagnosis of dementia of the Alzheimer’s type. Journal of the American Geriatrics Society 61, 1055-1059 (2013).


The culturally polarizing effect of the "anti-science trope" on vaccine risk perceptions 

The “ ‘anti-science’ trope” refers to a common theme in ad hoc risk communication that links concern about vaccine risks to disbelief in evolution and climate skepticism, all of which are cited as instances of a creeping hostility to science in the U.S. general public or at least some component of it.

In the last post, I presented evidence, collected as part of the CCP Vaccine Risk Perception study, that showed that the trope has no meaningful connection to fact. 

Those who accept and reject human evolution, those who believe in and those who are skeptical about climate change, all overwhelmingly agree that vaccine risks are low and vaccine benefits high.

The idea that either climate change skepticism or disbelief in evolution denotes hostility to science or lack of comprehension of science is false, too. That’s something that a large number of social science studies show.  The CCP Vaccine Risk study doesn’t add anything to that body of evidence.

But the CCP Vaccine Risk study did examine how differences in science comprehension and religiosity, which interact in an important way in disputes over climate change and evolution, don’t have any meaningful impact on perceptions of vaccine risk perceptions.

In addition to examining whether there was any factual substance to the anti-science trope, the CCP Vaccine Risk Perception study also investigated what the impact of the trope is—or at least could be if it were propagated widely enough—on public opinion.

For that purpose, the study used experimental methods. The experiment had three key elements.

First was a measurement of subjects’ cultural predispositions toward societal risks.

 I’ve actually described the strategy used to do so in several earlier posts.  But basically, the experiment used an “interpretive community” strategy, in which unobserved or latent group predispositions are extracted from subjects perceptions of a host of societal risks that are known to divide people with diverse cultural and political outlooks. This approach, as I’ve explained, furnishes the “highest resolution” for measuring the influence group predispositions might be having on perceptions of a risk on which there is reason to believe the impact might be small.


That analysis identified two cross-cutting or orthogonal dimensions along which risk predispositions could be measured.  I labeled them the “public safety” and “social deviancy” dimensions, based on their respective indicators (various environmental risks, guns, second-hand smoke in the former case; legalization of marijuana and prostitution and teaching of high school sex ed in the latter). 

 Subjects in the diverse 2,300-person sample of U.S. adults could thus be assigned to one of four “interpretive communities” (ICs) based on their score relative to the mean of both of these two “risk perception dimensions”: IC-α (“high public-safety” concern, “low social-deviancy”);  IC-β (“high public-safety,” “high social-deviancy); IC-γ (“low public-safety,” “low public-safety”); and IC-δ (“low public-safety,” “high social-deviancy”).  The intensity of the study subjects' commitment to one or the other of these groups can be measured by their scores on the public-safety and societal-deviancy risk-perception scales.

The second element was exposure of the subjects to examples of “ad hoc risk communication.”

The subjects were assigned to experimental conditions or groups, each of which read a different communication patterned on information in the media or internet.

One of these communications used the “anti-science trope.” Patterned on real-world communications (including ones reproduced in the Appendix to the Report), it was in the form of an op-ed that described disbelief in evolution, climate skepticism, and the belief that vaccines cause autism as progressive manifestations of a mutating “anti-science virus.” As is so for most real-world communications embodying the anti-science trope, the experiment communication displayed an unmistakably partisan orientation and conveyed contempt for members of the public who are skeptical of climate change and disbelieve evolution.

The third element was measurement of the subjects’ perceptions of vaccine risks and benefits.

The study used a large battery of risk and benefit items, which were combined into a highly reliable scale, “PUBLIC_HEALTH” (Cronbach’s α = 0.94), scores of which were transformed into z-scores (i.e., normalized so that increments reflected standard deviations from the mean) and coded so that lower ones denoted relatively negative assessments of vaccines and higher scores relatively positive ones.

In the experiment, then, the risk perceptions of subjects exposed to different forms of “ad hoc risk communication” were compared to the perceptions of survey participants, who were assigned to read a news story unrelated to vaccines and whose members served as the “control” group.

The results . . . .

As previewed in an earlier blog post, the study found that among members of the control group there was no practically meaningful relationship between  vaccine risk perceptions and the cultural risk predispositions measured by the “public safety” and “social deviance” IC dimensions.  IC-αs (“high public-safety,” “low social-deviancy”) scored highest on PUBLIC_HEALTH and IC- δs the lowest.  But the difference between them was trivially small—less than one-third of one standard deviation of the mean score.

As a measure of the practical difference in these scores, the predicted probability of agreeing that the “benefits of obtaining generally recommended childhood vaccinations outweigh the health risks” was estimated to be 84% (± 3%, LC = 0.95) for a typical IC‑α and 74% ( ± 4%) for a typical IC‑δ.

This was consistent with the findings of the Vaccine Risk Perception study's survey component generally, which found that there is broad-based consensus, even among groups that are bitterly divided on issues like climate change and evolution, that vaccine benefits are high and their risks low.  As of today, at least, vaccine risks are not culturally polarizing.

But that could change, the experiment results suggested.

This very modest difference in the perceptions of subjects displaying the IC-α  and IC-δ risk disposition widened significantly among their counterparts in the “anti-science trope” condition. Exposure to the “anti-science” op-ed also drove a wedge between subjects displaying the IC-β (“high public-safety,” “high social-deviancy) and IC-γ (“low public-safety,” “low social-deviancy”) dispositions, groups whose scores on the PUBLIC_HEALTH scale were indistinguishable in the control.

The practical significance of the difference can be illustrated by examining the impact of the experiment on the predicted probability of agreement with the item measuring “confidence in the judgment of the American Academy of Pediatrics that vaccines are a ‘ safe and effective way to prevent serious disease.’ ” Subjects responded to this item immediately after reading a statement issued by the AAP on vaccine testing and safety. The predicted probability that a subject with a typical IC-δ disposition would indicate a positive level of confidence dropped from 73% (± 4%, LC = 0.95) in the control to 64% (± 7%, LC = 0.95) in “anti-science”; the gap between the predicted probability of a positive assessment by a typical IC-δ and a typical IC-α grew 14% (± 9%) in the two conditions. The gap between the typical IC-β and both the typical IC-α (7%; ± 7%, LC = 0.95) and typical IC-δ (6%; ± 6%, LC = 0.95) grew, too, but by a more modest level. As one would expect, similar divisions characterized responses to other items in the PUBLIC_HEALTH scale.

There was no similar decrease in the predicted probability that a typical IC-δ would express a positive level of confidence in the other experiment conditions, one of which which featured a composite news story proclaiming an impending public health crisis from “declining vaccine rates,” and another of which a communication patterned on a typical CDC press release that conveyed accurate information on the high and steady level of vaccine rates in the U.S. in the last decade.  But as discussed in a previous post, subjects in the “crisis” condition, not surprisingly, grossly overestimated the degree of parental resistance to universal immunization—an effect that could negatively affect reciprocal motivations to contribute to the public good of herd immunity.

It is important to realize that the polarizing impact of the “Anti-science” op-ed resulted both from the positive effect it had on the vaccination attitudes of IC-α subjects and the negative effect it had on IC-δ ones. The overall effect of the “Anti-science” treatment was negligible.

The practical importance of the result, then, turns on the significance attached to the intensified levels of disagreement among subjects of diverse outlooks.

Previous CCP studies, including one involving controversy over the HPV vaccine, suggest that the status of a putative risk source as a symbol or focus of cultural contestation is what disrupts the social processes that ordinarily result in public convergence on the best available evidence relating to societal and health risks.

If this is correct, then any influence that intensifies differences among such groups should be viewed with great concern.

The "anti-science trope," in sum, is not just contrary to fact.  It is contrary to the tremendous stake that the public has in keeping its vaccine science communication environment free of reason-effacing forms of pollution.


How are climate skepticism, disbelief in evolution & vaccine hesitancy related?

The dominant theme of ad hoc risk vaccine risk communication warns of a “growing wave of public resentment and fear” that has induced a “large and growing number” of “otherwise mainstream parents” to refuse to vaccinate their children.

As discussed in the last post, this trope is not based on fact: there hasn't been an erosion in immunization rates”; on the contrary, coverage for all recommended childhood vaccines has held steady at 90% (the HHS "healthy person" target) or above for over a decade.

And while there's zero evidence of a " “growing crisis of public confidence in vaccines at present, emphatic assertions that there is one can be shown to induce misunderstandings and confusion inimical to the willingness of people to make voluntary contributions to public goods--like the herd immunity associated with universal immunization.

A secondary theme of ad hoc risk communication is the "anti-science" trope.  This claim links "growing" concern over vaccine safety to disbelief in evolution and skepticism toward climate change, all of which are depicted as evidence of a creeping hostility to science in the general public.

The CCP Vaccine Risk Perception study found this assertion, too, to be both contrary to fact and antithetical to maintaining the existing, broad-based public consensus in favor of universal immunization.

Below is a section of the Report that presents survey evidence on the relationship between vaccine risk perceptions, on the one hand, and climate change skepticism, disbelief in evolution, and science comprehension generally, on the other.  Tomorrow I'll post material relating to the Study's experimental component, which illustrates the potential of the "anti-science trope" to generate cultural conflict over vaccines.

A. Some benchmarks: evolution and climate change, science comprehension and religiosity.  

As emphasized, the aim of the survey component of the study was to evaluate the nature of the general public’s perception of childhood vaccine risks. Is there a shared or dominant affective orientation toward vaccine safety in the U.S. public? Or do childhood vaccines provoke mixed and opposing reactions—and if so, among whom?

Meaningful answers to these questions require an intelligible reference point with which to compare the survey responses. Dispute over universal vaccination laws—provisions that make immunization a condition of school enrollment, subject to medical or religious and in some states moral-objection “exemptions”—are frequently likened to conflicts over acceptance of mainstream science, including the teaching of evolution in public schools and the adoption of policies to mitigate the environmental impact of climate change. Associated with religious, cultural, and political divisions, the intensity and character of these conflicts can be used to help assess the intensity and character of any divisions of opinion observed on childhood vaccine risks.

The study measured study participants’ beliefs about both evolution and global warming. On evolution, subjects responded to an item from the National Science Foundation (2012) “Science Indicators” battery, which is conventionally used to measure science literacy. That item instructs respondents to respond to the statement “Human beings, as we know them today, developed from earlier species of animals.” In line with many public opinion polls (e.g., Newport 2012), 56% of the survey respondents classified this statement as “true,” and 44% as “false.”

On climate change, 52% of the survey respondents indicated that they believe scientific evidence supports the proposition that the earth’s temperature has been increasing in “the last few decades” as a result “of human activity such as burning fossil fuels.” Thirty percent indicated that they did not believe there was “solid evidence” of increasing global temperatures over the “past few decades,” while another 18% indicated that they believed there was “solid evidence” of warming but that the cause was “mostly. . . natural patterns in the earth’s environment,” as opposed to “human activity.” These figures, too, are in line with recent national opinion surveys (Silver 2013).

Irrespective of their responses to these items, however, the overwhelming majority of survey respondents agreed with the proposition that the “health benefits of obtaining generally recommended childhood vaccinations outweigh the health risks” (BALANCE). Eighty percent of the respondents who believe in human-caused climate change agreed with this proposition. So did 81% of those who believe the earth’s temperature has increased as a result of “natural patterns,” and 73% of those who believe the earth’s temperature has not increased in recent decades. Eighty percent of the respondents who believe in evolution and 77% who do not  (a difference smaller than the survey margin of error) likewise indicated that they agree the benefits of childhood vaccinations outweigh their risks (Figure 5).

Study participants also responded to items measuring both their religiosity and their knowledge of and facility with scientific evidence. The former was assessed with a scale that aggregated self-reported church attendance, frequency of prayer, and “importance of God” in the respondents’ lives (α = 0.86). Subjects’ “science literacy” was assessed with 11 items from the NSF’s Science Indicator battery, which is conventionally used to study public understanding of science in the U.S. and abroad (NSF 2012). In addition, subjects completed a ten-item version of the Cognitive Reflection Test (Frederick 2005; Toplak, West & Stanovich 2013), which assesses the motivation and capacity to consciously interrogate one’s views on the basis of available information, a critical-reasoning disposition integral to forming evidence-based beliefs (Toplak, West & Stanovich 2011).

The NSF and CRT items formed a reliable scale (Cronbach’s α = 0.82), which can be interpreted as measuring a “science comprehension” aptitude (Kahan, Peters et al. 2012). Consistent with previous studies (Pennycook 2012, 2013; Shenhav, Green & Rand 2011; Gervais & Norenzayan 2012), there was a modest negative correlation between the religiosity and science comprehension (r = -0.26, p < 0.01).

Religiosity and science comprehension also were meaningfully—but not straightforwardly—associated with the study subjects’ positions on evolution and climate change. Science comprehension was modestly associated (r = 0.28, p < 0.01) with belief in evolution and weakly associated with belief in human-caused climate change (r = 0.10, p < 0.01) for the sample as a whole (including both survey and experiment subjects). But the impact was moderated by subjects’ religiosity: among those low in religiosity, higher science comprehension substantially increased belief in evolution and in human-caused global warming; among those high in religiosity, however, higher science comprehension had next to no impact on belief in evolution and substantially reduced belief in human-caused global warming (Figure 7).

The interaction between religiosity and science comprehension is not surprising. Science literacy and critical reasoning dispositions have been found to magnify cultural and ideological predispositions toward global warming (Kahan, Peters et al. 2012; Kahan 2013b). So it stands to reason that they would have the same impact on predispositions associated with religiosity, which plays a comparable role to shared cultural and political outlooks in the web of social relationships that orient individuals toward what is known by science. “Belief in evolution” is not a reliable indicator of either a scientifically literate understanding of evolutionary mechanisms (Schtulman 2006; Bishop & Anderson 1990) or the species of science literacy measured generally by the NSF Science Indicators. Rather, how one responds to the question “do you believe in human evolution” indicates a form of identity that features religiosity (Roos 2012). It is perfectly plausible that the significance of “disbelief” in evolution as an expression of personal identity would be unaffected by science knowledge—or possibly even reinforced by habits of mind associated with critical reasoning. Indeed, experimental evidence supports this inference (Lawson & Worsnop 2006).

These relationships—which are integral to making sense of the salience and ferocity of societal conflict over climate change and over evolution—were absent from the views of the survey respondents toward childhood vaccines (Figure 7). Both science comprehension (r = 0.12, p < 0.01) and religiosity (r = -0.14, p < 0.01) displayed only weak relationships with the battery of items that formed the PUBLIC_HEALTH scale. There was an interaction between religiosity and science comprehension in the survey respondents’ scores on the scale, but it was small in size and, more importantly, moderated only the intensity of the positive orientation that subjects of varying levels of religiosity expressed toward childhood vaccines (App. 1, Table 1).

A more detailed examination of the participants’ responses to the various survey items follows. Unsurprisingly, there is unanimity on none. Nevertheless, understood in relation to contested societal issues that feature conflict among large and readily identifiable societal groups, the uniform and uniformly supportive margins of agreement reflected in survey responses is of fundamental interpretive significance. As will become even more apparent, in probing the nature of opposition to universal childhood immunization, one is necessarily assessing the attitudes of a segment of the population that is small in size and that defies identification by the sorts of characteristics associated with recognizable cultural styles in American society.


To download Vaccine Risk Perceptions and Ad Hoc Risk Communication: An Empirical Assessment, click here.


The logic of reciprocity--and the illogic of empirically uninformed vaccine risk communication

One of the aims of the CCP Vaccine Risk Perception and Ad Hoc Risk Communication study was to examine the impact of empirically uninformed vaccine risk communication.

By “empirically uniformed risk communication,” I don’t mean vaccine-risk misinformation, such as the claim that childhood vaccines cause autism. I take as a given that false assertions like that generate unwarranted public concern.

Rather, I’m using “empirically uniformed risk communication” to denote information that accurately conveys the risks of childhood vaccines--likely for the very appropriate purpose of counteracting misinformation-- but that nevertheless embeds that information in empirically insupportable representations about the extent, sources, and consequences of public unease toward universal immunization.

Disseminated by journalists, advocates, and even some public health professionals, this kind of vaccine-risk communication is the type that ordinary members of the public are in fact most likely to be exposed to.

It’s core message is that public health in the U.S. is being threatened by a “growing crisis of public confidence” in vaccines.  No longer confined to “[t]he fringe who don’t believe in medicine for religious reasons,” a “growing distrust of vaccinations” is now sweeping across “our nation’s parents,” we are told (& told & told & told).

The resulting “erosion in immunization rates” is predictably ... leading to the resurgence of diseases considered vanquished long ago” including whooping cough and measles. “From Taliban fighters to California soccer moms,” one source concludes, “those who choose not to vaccinate their children against preventable diseases are causing a public health crisis.”

The CCP study didn’t purport to test these claims. Instead, it deferred to sources that employ valid empirical methods specifically suited to measuring immunization rates and the incidence of childhood diseases.

These sources belie the claim that vaccination rates are declining at all, much less “eroding.”

According to data compiled by the U.S. Centers for Disease Control, coverage for recommended childhood immunizations—including MMR, pertussis, Hepatitis-b, and polio—have all been holding steady at or above 90% (the target threshold) for well over a decade (CDC 2013a, 2008, 2006). The proportion of children receiving no vaccinations has persisted at or below 1%, despite the ready availability of nonmedical “exemptions” from state-administered universal immunization policies for objecting parents.

Rather than a “large and growing number” of “otherwise mainstream parents” refusing to vaccinate their children, the CDC reports (in annual press releases, the language of which varies little from year to year) that“ ‘nearly all parents are choosing to have their children protected against dangerous childhood diseases’ ” (CDC 2010)

There are local enclaves, the CDC cautions, in which vaccination rates are significantly lower than the national average. These enclaves are often the site of recurring localized outbreaks of diseases, like measles, which public health officials have deemed eliminated in the United States but which can be introduced into such communities by individuals infected during travel abroad (CDC 2013b).

Fortunately, “[h]igh MMR vaccine coverage in the United States (91% among children aged 19–35 months),” the CDC states, “limits the size of [such] outbreaks,” which averaged 60 cases per year over the last decade (ibid.).

The incidence of whooping cough, which has not been eliminated in the United States, is also likely higher in low-vaccination enclaves (Atwell et al. 2013; Glanz et al. 2013; Omer et al. 2008). But “[p]arents refusing to get their children vaccinated,” according to the CDC, are “not the driving force behind the large scale outbreaks” of this disease in recent years (CDC 2013c). In addition to “increased awareness, improved diagnostic tests, better reporting, [and] more circulation of the bacteria,” the CDC (2013c) has identified “waning immunity “from an ineffective booster shot as one of the principal causes, a view shared by public health experts (Cherry 2012).

Pockets of resistance to vaccination pose a serious and unmistakable public health concern. They merit considered attention informed by empirical methods suited to assessing the influences that generate them, the contributions they make to the incidence of childhood diseases, and the measures that might be employed to counteract and contain them (Opel 2011, 2012; Mnookin 2011; Omer et al. 2008).

But the existence of anti-vaccine enclaves and the dangers they pose do not furnish empirical support for asserting that there is a “growing crisis of public confidence” in childhood vaccines, that “immunization rates with MMR have dropped in . . . the US,” or that a “rising tide of … vaccine reluctance” has generated “a resurgence of diseases gone so long that some doctors don’t even recognize them.

Such claims reflect not an “epidemic of fear” among ordinary parents, but an epidemic of hyperbole among a diverse collection of actors resorting to ad hoc, empirically uninformed alternatives to genuinely evidence-based forms of risk communication.

I'm sure those engaging in empirically uninformed vaccine risk communication are not doing so in bad faith.

Most probably just don’t know what they are talking about—in part because of the prevalence of empirically uninformed vaccine risk communication.

But some probably do realize that they are in fact grossly mischaracterizing the extent of vaccine avoidance in the U.S., and misattributing to it disease outbreaks that in fact stem from other causes such as the ineffective pertussis booster shot.

They probably figure that this fact-disconnected style of risk communication is okay because it will grab people’s attention and stir them to anger at parents who are not vaccinating their children and definitely should be.

But that way of thinking is empirically uninformed, too.  Indeed, the scientific study of science communication suggests that understating the high level of vaccination in the U.S. could actually weaken public support for and cooperation with universal immunization programs.

The “herd immunity” associated with universal childhood vaccination is a collective good (Olson 1965). That is, by complying with universal vaccination policies, parents confer a benefit—reduced risk of contracting a disease—not only on their own children but also on those who as a result of age, medical restrictions, or material disadvantage have not been able to secure the protection that such vaccinations confer.

In collective actions settings—from tax compliance to recycling, from voting to observation of informal norms on picking up one’s children on time from daycare centers—individuals tend to behave like moral and emotional reciprocators (Gintis et al. 2004). That is, rather than engage in purely self-interested calculation, they are motivated to contribute voluntarily to collective goods if they perceive that others are doing so, but to refrain from contributing if they think free-riding is widespread.

This dynamic makes it imperative that people not be induced to underestimate the extent to which others are voluntarily contributing to a collective good. If they do, a higher number of individuals will themselves refrain from contributing—behavior that can be expected to induce others to do the same, generating a self-reinforcing spiral of non-cooperation (Kahan 2004).

The logic of reciprocal cooperation implies that people who believe that others are refraining from getting vaccinated and instead free-riding on the contributions of others to herd immunity will themselves be less willing to get vaccinated. One experimental study using self-report data suggested that students exposed to information that suggested other students were forgoing vaccination and effectively free-riding on the decisions of others to get their flu shots responded in exactly this way (Hershey et al. 1994).

The results of CCP Vaccine Risk Perception study suggest that members of the general public substantially underestimate childhood-vaccine coverage. Asked to estimate “about what percentage of U.S. children (age 19-35 months) received the recommended vaccinations for childhood diseases” in recent years, only 9% of the survey subjects indicated “90% or above”; the median estimate was “70-79%.” In addition, approximately 40% of the sample indicated that the “trend in the rate of vaccination for U.S. children (age 19-35 months)” had gone down either “a little” or “a lot.”

The survey participants likewise grossly overestimated the proportion of children who receive no vaccinations. Only 9% correctly put the figure at “1% or less.” The median response was “2%-10%.” Over one-third of the sample selected either 11% to 20% or 21% to 30%.

Because of the contribution that reciprocity makes to individuals’ motivations to contribute to public goods like herd immunity, this kind of misunderstanding is not good.

Even worse, the experimental component of the CCP study found that individuals’ levels of misunderstanding grew when they were exposed to empirically uninformed vaccine risk communication.

In the experiment, subjects were assigned to one or another condition, members of which read different materials patterned on real-world media and internet sources.  Those assigned to the “Crisis” condition read a news that assert “growing” parental resistance to vaccination and a resulting decline in vaccination rates. Those assigned to the  “Anti-science” condition read an op-ed commentary that similarly implied that vaccination rates were declining based on fear of vaccine side-effects among individuals disposed to distrust scientific information on issues like evolution and climate change.

Subjects in both the “Crisis” and “Anti-science” conditions underestimated national vaccination coverage.  They also were more likely to report that vaccine rates had gone down in recent years.

Subjects in the “CDC” condition, in contrast, received a news story that quoted CDC officials accurately indicating that “vaccination coverage rates . . . have remained stable at or above 90 percent for over a decade,” and that “less than 1% of toddlers had received no vaccines at all,” but also warning of the “the existence of local communities in which vaccination coverage is lower than target levels for certain diseases.”

This communication was patterned on the annual CDC press releases that announce the latest NIS results.  Relative to the Crisis, Anti-science, and control group subjects, those in the CDC condition formed much more accurate estimates of the high existing and historical rates of vaccination in the U.S.

Obviously, the high existing rates of vaccination in the U.S. suggest that the degree to which the public currently underestimates vaccination rates is not now inducing widespread noncompliance. But because reciprocity dynamics have been shown to be robust across collective action settings, there is ample reason to discourage journalists, advocates, and health professionals from propagating the misimpression that there is a “growing wave of public resentment and fear” toward vaccines that has resulted in “ ‘erosion in immunization rates’ ”—a refrain, ironically, that strident vaccine opponents gleefully embrace.

Indeed, public awareness that the U.S. has historically enjoyed and continues to enjoy exceptionally high rates of compliance with universal vaccination programs should be regarded as an important public-health resource.

The best available evidence on science and risk communication implies that “public health campaigns that describe the already wide acceptance of pertussis vaccination” and other immunizations against childhood disease is the most reliable way to sustain that widespread acceptance (Hershey et al. 1994, p. 186).


Cultural Cognition Project, Vaccine risk perceptions and ad hoc risk communication: An experimental investigation. Cultural Cognition Risk Perception Studies Rep. No. 17 (Jan. 27, 2014).

CDC. CDC National Survey Finds Early Childhood Immunization Rates Increasing. (Sept. 1, 2011), available at

CDC. CDC Survey Finds Childhood Immunization Rates Remain High. (Sept. 16, 2010), available at

CDC. National, State, and Local Area Vaccination Coverage Among Children Aged 19–     `35 Months — United States, 2012. Morbidity and Mortality Weekly Reports 62, 733-740 (2013a).

CDC. National, State, and Local Area Vaccination Coverage Among Children Aged 19–35 Months — United States, 2007. Morbidity and Mortality Weekly Reports 57, 961-966 (2008).

CDC. National, State, and Local Area Vaccination Coverage Among Children Aged 19–35 Months — United States, 2006. Morbidity and Mortality Weekly Reports 56, 880-885 (2006).  

CDC. Pertussis Frequently Asked Questions (Dec. 9, 2013b). Available at

Cherry, J.D. Epidemic Pertussis in 2012 — the Resurgence of a Vaccine-Preventable Disease. New England Journal of Medicine 367, 785-787 (2012).

Gintis, H., Bowles, S., Boyd, R.T. & Fehr, E. eds. Moral Sentiments and Material Interests: The Foundations of Cooperation in Economic Life (MIT Press, Cambridge, Mass., 2004).

Glanz, J.M., et al. A Population-Based Cohort Study of Undervaccination in 8 Managed Care Organiza-tions across the United States Managed Care Organizations. JAMA pediatrics 167, 274-281 (2013).

Hershey, J.C., Asch, D.A., Thumasathit, T., Meszaros, J. & Waters, V.V. The Roles of Altruism, Free Riding, and Bandwagoning in Vaccination Decisions. Organ Behav Hum Dec 59, 177-187 (1994)

Kahan, D.M. The Logic of Reciprocity. in Moral Sentiments and Material Interests: The Foundation of Cooperation in Economic Life (ed. H. Gintis, S. Bowler & E. Fehr) 339-378 (MIT Univ. Press, Cambridge, MA, 2004).

Mnookin, S. The Panic Virus : A True Story of Medicine, Science, and Fear (Simon & Schuster, New York, 2011).

Olson, M., Heckelman, J.C. & Coates, D. Collective choice : Essays in honor of mancur olson (Springer, Berlin ; New York, 2003).

Omer, S.B., et al. Geographic Clustering of Nonmedical Exemptions to School Immunization Requirements and Associations with Geographic Clustering of Pertussis. American Journal of Epidemiology 168, 1389-1396 (2008).

Opel, D.J., et al. The Relationship between Parent Attitudes About Childhood Vaccines Survey Scores and Future Child Immunization Status: A Validation Study. JAMA pediatrics 167, 1065-1071 (2013).

Opel, D.J., et al. Validity and reliability of a survey to identify vaccine-hesitant parents. Vaccine 29, 6598-6605 (2011b). 


Who fears childhood vaccines and why? Research report & project

Just posted new report, Vaccine Risk Perceptions and Ad Hoc Risk Communication: An Empirical Assessment. It presents the results of a  large (N = 2300) national study of the public's perception of the risks and benefits of childhood vaccines. The study also includes an experimental component that examines how those perceptions are influenced by "ad hoc" risk communication --  information from popular sources that feature empirically uninformed claims about the extent, nature, and consequences of public concern about vaccine risks (there's very little concern to speak of, and views do not vary meaningfully across political or cultural groups).

The Report is part of a new CCP project on "Protecting the Vaccine Science Communication Environment." The project has its own page, which explains the project mission and links to various content.

I'll likely be featuring bits & pieces of the Report in the blog over the next  couple weeks.  I'm eager not merely to alert potentially interested readers that it is available but also to solicit comments, questions, and proposals for additional analyses.  Indeed, I anticipate issuing "updates" to the Report based on such feedback.

Here is the Report "Executive Summary":

Executive Summary

This Report presents empirical evidence relevant to assessing the claim—reported widely in the media and other sources—that the public is growing increasingly anxious about the safety of childhood vaccinations. The Report presents two principal findings: first, that vaccine risks are neither a matter of concern for the vast majority of the public nor an issue of contention among recognizable subgroups; and second, that ad hoc forms of risk communication that assert there is mounting resistance to childhood immunizations themselves pose a risk of creating misimpressions and arousing sensibilities that could culturally polarize the public and diminish motivation to cooperate with universal vaccination programs.

The basis for these findings was a study of a demographically diverse sample of 2,300 U.S. adults. In a survey component administered to a nationally representative 800-person subsample, the study found a high degree of consensus that vaccine risks are low and their benefits high. These perceptions, the data suggest, reflect the influence of a pervasively positive and widely shared affective orientation toward vaccines. This same affective orientation is reflected in widespread support for universal immunization and expressions of trust in the judgment of public health officials and professionals.

There was a modest minority of respondents who held a negative orientation toward vaccines. These respondents, however, could not be characterized as belonging to any recognizable subgroup identified by demographic characteristics, religiosity, science comprehension, or political or cultural outlooks. Indeed, groups bitterly divided over other science issues, including climate change and human evolution, all saw vaccine risks as low and vaccine benefits as high. Even within those groups, in other words, individuals hostile to childhood vaccinations are outliers.

In an experimental component administered to the entire sample, the study examined the impact of media and other reports that warn of escalating public concern over vaccine safety. Such information induced study participants to substantially underestimate vaccination rates and to substantially overestimate the proportion of parents invoking “exemptions” to universal immunization policies. This result is troubling because existing research shows that the motivation to contribute to collective goods, such as the herd immunity conferred by mass vaccination, declines when members of the public perceive that others are refusing to contribute. In contrast, exposure to a communication patterned on a typical CDC press statement induced subjects to form estimates much closer to actual U.S. vaccine rates (90% or above for over a decade) and of the proportion of children receiving no vaccinations (1%).

The experiment also examined the effect of information patterned on popular sources that link the belief that vaccines cause autism to disbelief in evolution and climate change. Among study subjects exposed to this information, perceptions of vaccine risks showed signs of dividing along the same cultural lines that inform disputes over highly contested societal issues, including the dangers of climate change, the consequences of drug legalization, and the impact of educating high school students about birth control. This result is also troubling: group-based conflicts are known to create strong psychological pressures that interfere with the normally reliable capacity that members of the public use to recognize valid decision-relevant science. This very dynamic is thought to have affected acceptance of the HPV vaccine.

Based on these findings the Report offers a series of recommendations. The most important is that the public health establishment play a more active leadership role in risk communication. Governmental agencies and professional groups should (1) promote the use of valid and appropriately focused empirical methods for investigating vaccine-risk perceptions and formulating responsive risk communication strategies; (2) discourage ad hoc risk communication based on impressionistic or psychometrically invalid alternatives to these methods; (3) publicize the persistently high rates of childhood vaccination and high levels of public support for universal immunization in the U.S.; and (4) correct ad hoc communicators who misrepresent vaccination coverage and its relationship to the incidence of childhood diseases.




MAPKIA! Episode 31 "Answer": culturally programmed risk predispositions alert to "fracking" but say "enh" (pretty much) to GM foods

Okay!  "Tomorrow" has arrived, which means it's time to real the "answer" "yesterday's" "MAPKIA!" episode.

As you no doubt recall, the question was ...

(i) What is the relationship between environmental-risk predispositions, as measured by ENVRISK_SCALE, and perceptions of GM food risks and fracking, respectively? And (ii), how, if at all, does science comprehension, as measured by SCICOMP, affect the relationship between people's environmental-risk predispositions and their perceptions of the dangers posed by GM food and fracking, respectively?

What made this an interesting question, I thought, was that both "fracking" and GM foods are novel risk sources.

If you read this blog ... Hmmm...

I was going to say if you read this blog this might surprise you, because in that case you have a weridly off-the-scale degree of interest in political debates over environmental risks and thus are grossly over-exposed to people discussing and arguing about fracking and GM food risks and what "the public" thinks about the same.  

But if you do regularly read this blog, then you, unlike most of the other weird people who fit that description, actually know that most Americans haven't heard of fracking and aren't too sure what GM foods are either.

Indeed, if you regularly read this blog (why do you? weird!), then you know that the claim "GM foods are to liberals what climate change is to conservatives!!" is an internet meme with no genuine empirical substance.  I've reported data multiple times showing that GM foods do not meaningfully divide ordinary members of the public along partisan or cultural lines.  The idea that they do is not a fact but a "rule" that one must accept to play a parlor game (one much less interesting than "MAPKIA!") that consists in coming up with just-so explanations for non-existent trends in public opinion.

But I thought, hey, let's give the claim that GM foods are politically polarized etc. as sympathetic a trial as possible. Let's take a look after turning up the resolution of our "cultural risk predisposition" microscope and see if there's anything going on. 

To make what I mean by that a bit clearer, let's step back and talk about different ways to measure latent risk predispositions.

"Cultural cognition" is one framework a person genuinely interested in facts about risk perceptions can use to operationalize the hypothesis that motivated reasoning shapes individuals' perceptions of culturally or politically contested risks.

What's distinctive about cultural cognition -- or at least most distinctive about it -- is how it specifies the latent motivating disposition.  Building on Douglas and Wildvasky's "cultural theory of risk," the cultural cognition framework posits that individuals will assess evidence (all kinds, from the inferences they draw from empirical data to the impressions they form with their own senses) in a manner that reinforces their connection to affinity groups, whose members share values or cultural worldviews that can be characterized along two dimensions--"hierarchy-egalitarianism" and "individualism-communitarianism."  Attitudinal scales, consisting of individual survey items, are used to measure the unobservable or latent risk predispositions that "motivate" this style of assessing information.

But there are other ways to operationalize the "motivated reasoning" explanation for conflict over risk.  E.g., one could treat conventional left-right political outlooks as the "motivator," and measure the predispositions that they generate with valid indicators, such as party identification and self-reported liberal-conservative ideology.

Do that, and in my view you aren't offering a different explanation for public controversy over risk and like facts. Rather you are just applying a different measurement scheme.

And for the most part, that scheme is inferior to the one associated with cultural cognition. By that, I mean (others might have other criteria for assessment, but to me these are the only ones that are worth any thoughtful person's time) that the cultural worldview measures of latent risk predispositions have more utility in explainining, predicting, and fashioning prescriptions than does any founded on left-right ideology.

I've illustrated this before by showing how much left-right measures understate the degree of cultural polarization that exists among ordinary, relatively nonpartisan members of the public (the vast number who are watching America's Funniest Pet Videos when tiny audiencies tune in to either Madow or O'Reily) on certain issues, including climate change.

Cultural worldviews are more discerning if one is trying to measure the unobserved or latent group affinities at work in this setting. 

But certainly it should be possible to come up with even more discerning measures still. In fact, in between blog posts, that's all I spend my time on (that and listening to Freddie Mecury albums).

In a previous blog post, I referred to an alternative measurement strategy that I identified with Leiserowtiz's notion of "interpretive communities."  In this approach, one measures the latent, shared risk predisposition of the different affinity groups' members by assessing their risk perceptions directly.  The risk perceptions are the indicators for the scale one forms to explore variance and test hypotheses about its sources and impact.

I formed a set of "interpretive community" measures by running factor analysis on a battery of risk perceptions assessed with the "industrial strength" measure.  The analysis identified two orthogonal latent "factors," which, based on their respective indicators, I labeled the "public safety" and "social-deviancy" risk predispositions.

How useful is this strategy for explaining, predicting, and forming prescriptions relating to contested risk?

The answer is "not at all" if one is interested in explaining etc. any of the risk perceptions that are the indicators of the "interpretive community" scale.  If one goes about things that way, then the explanans -- the interpregtive community (IC) scale-- has been analytically derived from the explanandum-- i.e., the risk one is trying to explain. This approach is obviously circular, and can yield no meaningful insight.

But if one is trying to make sense of perceptions of a novel or in any case not yet well understood risk perception, then a latent-measurement strategy like the IC one could well be quite helpful.

In that case, because the risk perception that one is interested in examining is not an indicator of the IC scale, there won't be the circularity that I just described.

In addition, the IC risk measure is likely to be more discerning with respect to that risk than the cultural cognition worldview scales.  

That's because individual risk perceptions are necessarily even more proximate, measurementwise, to the latent risk-perception predisposition that generates them than are latent-variable indicators relating to values and other individual characteristics.

Accordingly, if we think the relationship between a motivating predisposition and a risk perception might be weak -- or if we just aren't sure what the relationship might be -- then it might be quite sensible to use an IC method to measure the predisposition.

The inferences we'll be able to draw about why any relationship exists will be less suggestive of the operative social and psychological influences than ones we could have drawn if we measured the predisposition with indicators more remote ("distal") from individual risk perceptions. But if a valid IC scale picks up a relationshp that is too weak to have registered otherwise, then we'll know at least a bit more than we would have.  And if nothing shows up, we can be even more confident that the risk perception in question just isn't one that originates in the sort of dynamics that generate cultural cognition & like forms of motivated reasoning. . . . 

So I thought I'd try an IC apparoach for genetically modified foods rather than just repeat for the billionth time that there isn't any reason for characterizing them as a source of meaningful public conflict, much less one that pits "anti-science scared liberals" against conservatives. 

I formed a simple aggregate Likert scale by normalizing the sum of the (normalized) scores on responses to the industrial-strength risk perception measure as applied to global warming, nuclear power, toxic waste disposal, and air pollution.  I confirmed not only that the resulting scale was highly reliable (Cronbach's α = 0.82) but also that it generated a sharp division among individuals whose cultural outlooks-- "egalitarian communitarian" and "hierarch individualist," respectively--tend to divide over environmental and technological risks.

I confirmed too that the degree of cultural division associated with these risks increases as people with these outlooks score higher on a science-comprehension measure -- as one would expect if cultural cognition is motivating individuals to use their critical reasoning abilities to form identity-congruent rather than truth-congruent beliefs.

That gives me confidence that ENVRISK_SCALE, the aggregate Likert measure, supplies the high-resolution instrument I was after to examine GM food risk perceptions, and fracking ones, too, just for fun.

To appreciate how cool what one can see with ENVRISK_SCALE is, consider first the blurry, boring view one gets with a right-left political-outlook scale, which as I indicated supplies only a low-resolution measurement of the relevant motivating dispositions.

These scatterplots array members of the 1800-or-so-member, nationally representative sample with respect to their right-left political outlooks, measured with a composite scale formed by aggregating their responses to a party-identification measure and to a liberal-conservative ideology measure, and their perceptions of global warming, fracking, and GM food risks, all of which are assessed with the industrial-strength measure.

The visible diagonal pattern formed by the observations, which are colored "warm," red & orange for "high" risk concern" and "cold" green/blue for "low," shows that there is a strong right-left political influence on climate-change risk perceptions.

By the same token, the absence of much of a diagonal pattern for GM food risk perceptions illustrates how trivially political outlooks influence them.

To quantify this, I plotted regression lines, and also reported the R^2's, which reflect the "percentage of variance" in the respective risk perceptions (models here) "explained" by the right-left political outlook measure.  In the case of global warming, left-right outlooks explain an "impressively large!" 42% of the variance.  For GM food risks, political outlooks explain a humiliatingly small 2%.... But hey, don't let facts get in the way if you want to keep "explaining" why liberals are so worried about GM food risks!

Now, interestingly, right-left political outlooks explain 30% of the variance in fracking risk perceptions.  That's also "impressively large!"  Seriously, it is, because as I said, most members of the public don't know much if anything about fracking; I suspect at least 50% had never heard of it before the study!

I could turn up the resolution with cultural outlook measures but I've done that a bunch of times in the past and not seen anything interesting on GM foods.

So now let's zoom in with the even higher-resolution ENVRISK_SCALE.

Here I've just plotted fitted regression lines for the sample as a whole, and lowess ones for those subjects in the bottom 50% & top 10% on the "science comprehension" scale. I've left out global warming, for as I indicated, it makes zero sense to use an attitudinal scale to explain variance in one of its indicators.

Clearly, ENVRISK_SCALE is more discerning than are right-left political outlooks.  The R^2s have gone up a lot!

Indeed, at this point, I'm willing to accept that something at least slightly interesting seems to be going on with GM foods.  There are no "hard and fast" rules in assessing when an R^2 is "impressively large!" (I think the main value of R^2 is in comparing the relative fit or explanatory power of models, in fact).  But my practical sense is that the "action" that ENVRISK_SCALE is indeed meaningful, and suggestive of at least a weak predisposition among individuals, mainly egalitarian communitarians, who are on the "risk concerned" side of issues like climate change and nuclear power to worry.

The impact of science comprehension is also quite revealing, however, and cuts the other way!

As one would (or ought to) expect for risk perceptions that genuinely trigger motivated reasoning, science comprehension magnifies the polarizing effect of the disposition measured by ENVRISK_SCALE for fracking.

But it doesn't for GM foods.  Science comprehension predicts less risk concern, but it does so pretty uniformly across the range of the disposition measured by ENVRISK_SCALE.  

That suggests positions on GM foods aren't particularly important to anyone's identity.  If they were, then we'd expect the most science-comprehending members of competing groups to be picking up the scent of incipient conflict & assuming their usual vanguard role.

So on balance, I'm a little more open to the idea that GM foods could be a source of meaningful societal conflict--but only a tiny bit more.  More importantly, I'm less sure of what I believed than before & anticipate that someone or something might well surprise me here -- that would be great.

I'm really excited, though, about fracking!

Fracking already seems to warrant being viewed as a matter of cultural dispute despite its relatively novelty.  There's something about it that jolts individuals into assimilating their impressions of it to the ones they have on cluster of very familiar contested risks (climate, nuclear, air pollution, chemical wastes) that are the focus of the ENVRISK_SCALE.  That the most science-comprehending individuals are even more polarized on fracking suggests that the future for fracking might well look a lot like that for climate change.

As I adverted to last time, it's possible -- likely even-- that the wording of the fracking item, by referring to to "natural gas" being "extracted" from the earth, helped to cue relatively unfamiliar or even completely unfamiliar respondents as to what position to form.  But I think the settings in which people are likely to encounter information about fracking are likely to be comparably rich in such cues.

So watch out fracking industry!  And everyone else, for that matter.

Well, who won the game this particular "MAPKIA!" contest?

I'm going to have to say no one.

There were literally thousands of entries, most sent in via postcards from around the globe.

But for the most part, people just assumed that GM food risks perceptions would behave like the other risk perceptions measured by the ENVRISK_SCALE, both in the nature and extent of variance and their interaction with science comprehension.

Given the hundreds of thousands of Macanese children who never miss a "MAPKIA!" episode and who undertandably view its players as role models, I can't in good conscience declare anyone the winner under these circumstances! 

As I've emphasized -- zillioins of times -- cultural polarization on risks is the exception and not the rule. Ignoring the denominator-- as commentators sadly do all too often -- makes cogent explanations of this dynamic impossible

No problem whatsoever, of course, to predict a polarized future for GM food risks. But we're not there yet, and any interesting prediction of why that's where we'll end up would have to reflect a decent theoretical account of why GM foods will emerge as one of the lucky few risk sources that get to travel down the polarization path when so many don't.

Feel free to file your appeals, however, in the comments section!


MAPKIA! Episode 31: what is the relationship between "environmental risk perception" predispositions, science comprehension & perceptions of the risks of (a) fracking & (b) GM foods?!

Example MAPKIA winner's prize (actual prize may differ)Okay everybody!

Time for another episode of Macau's favorite game show...: "Make a prediction, know it all!," or "MAPKIA!"!

By now all 14 billion regular readers of this blog can recite the rules of "MAPKIA!" by heart, but here they are for the 16,022 new 2014 subscribers:

I, the host, will identify an empirical question -- or perhaps a set of related questions -- that can be answered with CCP data.  Then, you, the players, will make predictions and explain the basis for them.  The answer will be posted "tomorrow."  The first contestant who makes the right prediction will win a really cool CCP prize (like maybe this or possibly some other equally cool thing), so long as the prediction rests on a cogent theoretical foundation.  (Cogency will be judged, of course, by a panel of experts.)  

The motivation for this week's show came from a twitter exchange between super-insightful psychologist Daniel Gilbert & others on whether "liberals" are "anti-science" on GM Foods.

Kind of ruins the "motivated-reasoning mirror on the wall, who is the most anti-science of all?!" game, but I can't help resorting to data whenever I catch an episode of that particular show.

In this case, however, the data surprised me! (Shit--weird things tend to happen when I say I am surprised by my data.... Oh well, too late.)

So I figured I'd give others a chance to play "MAPKIA!"" & see if they, unlike me, could accurately foresee what the data would say.

There's some background/windup here, so bear with me!

c'mon ... click me!(1) Let's start by constructing a simple scale for measuring "environmental risk perception" predispositions generally.  Members of an N = 2000 nationally representative sample of individuals recruited last summer to take part in CCP studies responded to a battery of "industrial grade" risk perception items, including ones on global warming, air pollution, nuclear power, and disposal of toxic chemical wastes.  The responses to those particular items formed a highly reliable (Cronbach's α = 0.82) aggregate Likert scale, which I labeled ... "ENVRISK_SCALE."

(2) ENVRISK_SCALE can be viewed as measuring a latent or unobserved predispostion toward culturally polarizing environmental risks.  That was my goal in forming it.

Just to confirm that I was measuring what I thought I was measuring, I regressed ENVRISK_SCALE on the "hierarchy-egalitarian" and "individualist-communitarian" worldview scales.  As expected, both scales were negatively associated with ENVRISK_SCALE -- i.e., Egalitarian Communitarians were risk sensitive, and Hierarch Individualists risk dismissive. The model R^2 was an "impressively large!" 0.43.

Moreover, as every school -boy or -girl in Macau would have predicted, these effects interact with science comprehension, an aptitude measured with SCICOMP, a composite formed from the NSF's "science literacy" indicators & a long version of Frederick's "cognitive reflection test. That is, consistent with the signature of "expressive rationality," the polarizing effect of the cultural worldviews grow even more intense as subjects' science comprehension scores increase.

Take a look!

Okay! We are almost ready for the "MAPKIA!" question.  

In addition to the global warming, nuclear power, air pollution, and toxic waste disposal items, the survey instrument also had "industrial grade" measures for both fracking & GM foods. That is, the respondents were asked to indicate "how much risk do you believe" each of those two "pose[] to human health, safety, or prosperity" on a 7-point scale (0 “no risk at all”; 1 “Very low risk”;  2 “Low risk”; 3 “Between low and moderate risk”; 4 “Moderate risk”; 5 “Between moderate and high risk”; 6 “High risk”; 7 “Very high risk”).

I suspected that at least half of the subjects would have no idea what "fracking" was -- after all, like 50% of the rest of the country, 50% of the respondents didn't know the length of the term of a U.S. Senator.

So when respondents got to this particular entry on the randomly ordered (separate page each) list of two dozen or so putative risk sources, they were asked to indicate the seriousness of the risk posed by " 'fracking'  (extraction of natural gas by hydraulic fracturing)."

I didn't use any analogous hints for GM foods.  Respondents were simply instructed to indicate how serious they thought the risks posed by "genetically modified food" were.

But in fact, GM foods are also a fairly novel risk source. Whether they threaten human health is another issue that most ordinary members of the public have given little if any thought to.

Because both "fracking" & GM food risks aren't nearly so salient -- aren't nearly so entangled in relentless, high-profile forms of cultural conflict-- as global warming, nuclear power, air pollution, or even toxic waste disposal, it would be surprising if cultural worldviews explained a lot of variance in individuals' perceptions of how dangerous they are.

If we really want to give these risk perceptions a "fair chance" to show that they are responsive to the gravitational force of cultural contestation, then we need to turn up the resolution of of our measuring instrument to compensate for the remoteness of fracking and GM foods from the center of everyday tribal rivalry.

ENVRISK_SCALE fits the bill. The risk perception items that are its indicators are necessarily even more proximate to whatever the unobserved or latent group affinity is generating the cultural cognition of risk than are the cultural worldview measures.  Why not be really generous, I thought in my own know-it-all way as I reflected on the DG twitter colloquy, & use a culturally infused environmental risk perception measure to show what the evidence really has to say about who fears GM foods & why? 

So now the question, which has two subparts:

(i) What is the relationship between environmental-risk predispositions, as measured by ENVRISK_SCALE, and perceptions of GM food risks and fracking, respectively? And (ii), how, if at all, does respondents' level of science comprehension, as measured by SCICOMP, affect the relationship between their environmental-risk predispositions and their perceptions of the dangers posed by GM food and fracking, respectively?

Ready ... get set ..."MAPKIA!" 


Secular cultural trends punctuated by noisy, emotional peaks & valleys: surveying the psychology landscape of mass opinion, mass shootings, & gun control

Really cool new working paper by Josh Blackman & Shelby Baird on the psychology of mass public opinion on guns.  

Based on a disciplined synthesis of decades of survey data in relation to mass shooting events, plus a textured case study of popular reactions to the Newtown shooting, B&B construct an interesting & plausible model of the psychological dynamics that shape popular support for gun control.

The key pieces consist of [1] an aggregate societal demand for gun restrictions, which comprises a vectoring (essentially) of culturally grounded predispositions; [2] a collection of risk-perception heuristics that, interacting with cultural predispositions, regulate popular attention and reaction to information on gun risks and the efficacy of gun regulation; and [3] sporadic mass shooting events that, feeding on [2], ignite a conflagration of political activity that cools and abates in a recurring, predictable pattern ("the shooting cycle"), leaving no net effect on [1].

The political-economy take home is that gun control supporters can't expect to buy much with the currency of popular opinion. As a result of [2], we can expect the drama of gun control to remain stubbornly anchored to the center of the popular-political stage.  But once [1] and [3] are disentangled, B&B conclude, it becomes clear that the popular demand for gun control is relatively weak and growing progressively weaker over time, notwithstanding the predictably intense but temporary spikes generated by mass shootings.

Because of the psychology of gun risks, the prospect of scoring a decisive victory will thus continue to tantalize gun control supporters, who will respond with convulsive enthusiasm to the "opportunities" episodically furnished by mass shooting tragedies.  But according to B&B, they won't get anywhere unless there is "a significant cultural shift" on guns--one the dimensions of which are significant enough to alter [1].  

Indeed, B&B view the prospects of that sort of development as constrained by [2] as well. Advocacy groups will predictably employ culturally partisan and divisive idioms to milk support from the members of groups that are culturally predisposed to see gun risks as high, thereby reinforcing the political motivation of opposing groups to resist gun regulation as an assault on their identities.

There are lots of things to like about this paper.

One is the interesting and compelling explanatory framework B&B construct.  Even if one isn't sure it is right-- or even strongly suspects it is wrong!--engaging with it is a great way to structure one's collection and assessment of evidence that can be used to advance understanding of gun control politics.  In addition, even if one isn't interested in gun control, one can profitably adapt the framework to other "risk" issues, like, say, climate change, where advocacy seems similarly disoriented by the allure of popular-opinion fool's gold.

Another is the solid style of analysis.  B&B didn't conduct an original observational study or conduct an experiment. But they did use valid empirical methods.  That is, they formulated a set of conjectures, identified sources of evidence that could be expected to support an inference as to whether the conjectures were likely true or not, and then collected the evidence and assessed it in a disciplined and transparent manner that admits of engagement by critically reasoning readers.

Contrast this with the "just-add-water-&-stir, instant decision science" that abounds in both popular and academic commentary.  That style of analysis, which aims to mesmerize credulous readers into thinking that their preconceptions are "scientifically supported," is a counterfeit species of empiricism.

To be sure, the sort of "synthetic empirical" analysis that B&B have performed is open to criticism, particularly given the flexibility those who engage in it have to identify confirming and disconfirming forms of secondary evidence.

But no form of valid empirical analysis is free of doubt.  

A smart person will be willing to accept guidance from any valid form of empirical inquiry--that is, from any that is susceptible of generating more or less reason to believe a proposition than one would otherwise have. Rather than wasting time arguing about "which valid empirical method is best," that person will welcome all forms, the results of which that individual will combine in forming his or her views.

The "gold standard" is the "no gold standard" philosophy of convergent validity.

The final thing to like about this paper: cool graphs!




More on Pew's evolution survey & valid inferences about polarization

Not here-- but over on  Stats Legend Andrew Gelman's Statistical Modeling & Causal Inference blog.  AG also featured the issue on the Monkeycage couple days ago.


What sorts of inferences can/can't be drawn from the "Republican shift" (now that we have enough information to answer the question)?

Okay, so Pew, not surprisingly, happily released the partisan breakdown for all parts of its evolution question.

Pew also offered a useful explanation of what it admitted was a “puzzle” in its report--viz., how the proportion of Republicans "disbelieving" evolution could go up while the proportions of Democrats and Independents as well as the proportion of the general population "believing" in it all stayed "about the same"? Should be obvious, of course, that this was something only Pew, & not others without access to the necessary information, could do.

So now I’ll offer up some reflections on the significance of the “Republican shift”—the 9 percentage-point increase in the proportion of Republicans who indicated that they believe in the “creationist” response and the 11 percentage-point decrease in the proportion who endorsed either the “Naturalistic” or “Theistic” evolution responses to Pew’s “beliefs on evolution” item.

I’ll start with two background points on public opinion, including partisan divisions, on evolution. They are pretty critical to putting the “shift” in context.  Then I’ll offer some points that counsel against treating the “shift” as a particularly important new datum.

But to give you a sense of the theme that motivates the presentation of this information, I think the modal response to the Pew survey in the media & blogosphere was absurd.  Paul Krugman’s reaction is typical & typically devoid of reflection: “Republicans are being driven to identify in all ways with their tribe — and the tribal belief system is dominated by anti-science fundamentalists.”

He and many others leapt to a conclusion without the evidence that logic would have told them was not supplied in the original Pew summary. That’s pretty embarrassing. 

And not surprisingly, the theme of their interpretation – “more evidence of Republicans being driven to anti-science extremism!” – is a testament to confirmation bias: the use of one’s existing beliefs to construe ambiguous data, which is then treated as corroborating one’s existing beliefs.

Background point 1: “Beliefs” on evolution lack a meaningful relationship to understanding evolution, to science literacy generally, or to being “pro/anti-” science.

Only aggressive disregard of empirical data—lots and lots and lots of them!—can explain why popular commentators start screaming about science illiteracy and creeping “anti-science” sensibilities in the U.S, every time a major polling outfit releases an “evolution belief” survey (about once a year).

As I’ve mentioned before, there is zero correlation between saying one “believes” in evolution and being able to give a passable (as in pass a highschool biology test) account of the modern synthesis (natural selection, random mutation, genetic variance) account of it.  Those who say they “believe” are no more likely to have even a rudimentary understanding of how Darwinian evolution works than those who say they “don’t believe” it.

In fact, neither is very likely to understand it at all.  The vast majority of those who say they “believe in evolution” believe something they don’t understand

But that’s okay.  They’d not only be stupid—they’d be dead—if people insisted on accepting as known by science only those insights that they actually can intelligiently comprehend!  There’s way too much scientific knowledge out there, and it matters too much!

What’s not okay is to march around smugly proclaiming “my side is science literate; your’s isn’t!” because of poll results like this one.  That’s illiberal and ignorant.

It is also well established that “belief” in evolution is not a valid indicator of science literacy in general

Answering “yes” to the simplistic “do you believe in evolution” item in the NSF’s “science indicators” battery doesn't cohere with how one does on the rest of this science literacy test—in part because plenty of science know-nothings answer “yes” and in part because plenty of “science know a lots” answer “no.”

The item isn’t measuring the same thing as the other questions in the battery, something NSF itself has recognized.  What it is measuring is a matter I’ll address in a second.

Finally, as Pew, in one of the greatest surveys on U.S. public attitudes toward science ever has shown, “disbelieving” in evolution is not meaningfully associated with being “anti-science.”

The vast majority of people who say “I believe!” and those who say “I don’t”—“tastes great!” vs. “less filling!”—all have a super positive attitude toward science.

The U.S. is an astonishingly pro-science society. If you think otherwise, you just don’t know very much about this area.

Background point 2: “Belief”/“disblief” in evolution is a measure of identity, not a measure of science knowledge or attitudes.

As I’ve indicated, answering “I believe!” to a simple-minded “do you believe in evolution? Huh? Do you? Do you?” survey question is neither a valid measure of understanding evolution nor a valid indicator of science comprehension.

What it is is a measure of cultural identity.  People who say “yes” are expressing one sort of cultural affiliation & associated outlooks; those who say “no” are expressing another.

Religiosity is one of the main indicators of the relevant cultural styles.  The more religious a person is, the more likely he or she is to say “I don’t believe" in evolution.

Again, “belief” has nothing—zero, zilch—to do with science literacy.

Partisan self-identification—“I’m a Democrat!”; “I’m a Republican” (“tastes great! …”)—is simply another indicator of the relevant cultural styles that correspond to saying “believe” & “not believe” in evolution.

The partisan divide on evolution is old old old old news.

"MAFY" (i.e., “Making a fool of yourself based on uniformed reading of Pew poll") point 1: Well, what do you know! Democrats don’t believe in “evolution” either!

Now that Pew has released the partisan breakdowns on its entire evolution item and not just the first half of it, it is clear, as anyone who knows anything about this area of public opinion could have told you, that the vast majority of the U.S. publicDemocrat, Republican, and Indpendentsay they “don’t believe” in evolution.

Pew initially released the breakdown only on that 1/2 of the question that asked whether respondents believed “Humans and other living things have evolved over time” or instead “Humans and other living things have existed in their present form since the beginning of time.”

The next 1/2 asks those who select “evolved” whether they believe that “Humans and other living things have evolved due to natural processes such as natural selection” or whether they believe instead that “A supreme being guided the evolution of living things for the purpose of creating humans and other life in the form it exists today.”

Get that, Paul Krugman et al?  The first position is Darwinian evolution; the second isn’t—it’s something goofy and non-scientific like “intelligent design”!

Only 37% of Democrats say they believe that humans have evolved as a result of “natural selection.”  Over 40% of the Democrats who “believe in evolution” buy either the “supreme guidance” variant or “don’t know” if evolution operates without or without God involved.

Does this mean they are “anti-science”?


What it means to say one “believes” or “disbelieves” in evolution is a complicated, subtle thing.

What groups “believe” about evolution certainly tells us something about their attitudes toward science!

But for sure what it says can’t be reduced to the simplistic (genuinely ignorant) equation “disbelieve = anti-science.”

If you would like to understand these things, rather than be a pin-up cheerleader for an embarrassingly, painfully unreflective bunch of partisan zealots-- your tribe!--then you’ll have to simply accept that the world is complicated.

“MAFY” point 2: There was no meaningful “shift” in the proportion of Republicans who reject “naturalistic” or “Darwinian” evolution.

Now that Pew has released all the numbers, we know that 23% of self-identified Republicans in 2009 said they “believe” in “naturalistic” evolution—evolution via “natural selection” rather than divine “guidance”—and that 21% said that in 2013. 

Not within the statistical margin of error, as far as I can tell.

And definitely not practically significant.


“MAFY” point 3: The Pew survey is really interesting but does not in itself support any inference about a significant “change” in anything since 2009.

As I indicated, the partisan division on evolution is old old old old news.  That’s because the tendency of people with culturally opposing styles to take opposing positions on it—ones that express their identity and not their knowledge of or attitudes toward science—is old old old news.

The question is whether the Pew poll—which is really an excellent piece of work, like everything else they do—justifies concluding that something material has changed in just the last four years.

I've thought & thought about it & concluded it really doesn't.  Here's why.

1st, as emphasized, the shift in the percentage of Republicans who say they believe in Darwinian or naturalistic evolution was a measly 1%.

2d, Pew has given us 2 data points.  Without knowing what the breakdown was on their question prior to 2009, it is logically fallacious to characterize the 2013 result as evidence of Republican “belief in evolution” as having “plummeted.”  For all we know, non-belief is “rebounding” to pre-2009 levels.

I don’t know if it is.  But the point is, all those asserting a shift don’t either.  They are fitting their interpretation of incomplete, ambiguous data to their preconceptions.

3rd, if something real had changed, it wouldn’t show up only in Pew’s data. Gallup has been doing polls on evolution regularly for decades.  It’s numbers show no meaningful change in the numbers, at least through 2012 (go ahead, if you are a story teller rather than a critical thinker, and invent some ad hoc account of the amazing event in 2013 that changed everything etc).

More likely, then, Pew’s result reflects just a blip. 

Also supporting that view is the pretty big discrepancy between the percentage who identify as “naturalistic” as opposed to “theistic evolutionists” in Pew’s poll and those who do so in Gallup’s.  The questions are worded differently, which likely explains the discrepancy.

But that the slight word changes can generate such big effects underscores how much of a mistake it is to invest tremendous significance in a single survey item. 

Good social scientists--& I’d definitely include the researchers who work for Pew in that group—know that discrepancies in the responses to individual survey items mean that individual items not a reliable basis for drawing inferences about public opinion. Because what individual items “measure” can never be determined with certainty, it is always a mistake to take any one item at face value.

Look at lots of related items, and see how they covary.  Then consider what sorts of inferences fit the overall pattern.

Here, the “overall pattern” is too indistinct, too uneven to support the inference that the 9% “shift” in the proportion of Republicans who indicated they “believe” in “creationism” in the 2009 Pew survey and the 2013 one means the world has changed in some way bearing on the relationship between beliefs in evolution and the sorts of identities indicated by partisan self-identification.

Maybe something has!

But the question is whether the survey supports that inference.  If you want to say, “Oh, I’ll construe the survey to support the conclusion that something interesting happened because I already know that’s true,” be my guest.

It’s a free country, as they say, and if you want to jump up & down excitedly & reveal to everyone in sight that you don’t know the difference between “confirmation bias” and valid causal inference, you have every right to do so!


Weekend update: Non-replication of "asymmetry thesis" experiment

A while back I did a couple of posts (here & here) on Nam, H.H., Jost, J.T. & Van Bavel, J.J. “Not for All the Tea in China!” Political Ideology and the Avoidance of Dissonance,  PLoS ONE 8(4) 8, doi:59810.51371/journal.pone.0059837 (2013)

NJV-B requested subjects (Mechanical Turk workers; more on that presently) to write  “counter-attitudinal essays”—ones that conflicted with the positions associated with subjects’ self-reported ideologies—on the relative effectiveness of Democratic and Republican Presidents. They found that Democrats were "significantly" more likely to agree to write an essay comparing Bush II favorably to Obama or Reagan favorably to Clinton than Republicans were to write onecomparing Obama favorably to Bush II or Clinton favorably to Reagan.

NJV-B interpreted this result as furnishing support for the "asymmetry thesis," the proposition that ideologically motivated reasoning is disproportionately associated with a right-leaning or conservative ideology. The stronger aversion of Republicans to writing counter-attitudinal essays, they reasoned, implied greater resistance on their part to reflecting on and engaging evidence uncongenial to their ideological predispositions.

I wrote a post explaining why I thought the design was a weak one.

Well, now Mark Brandt & Jarret Crawford have released a neat working paper that reports a replication study.

They failed to replicate NJV-B result. That is, they found that the subjects' willingness to write a counter-attitudinal essay was not correlated with their ideological dispositions.

That's interesting enough, but the paper also has some great stuff in it on other potential dispositional influences on the subjects' assent to write counter-attitudinal essays.

They found, e.g., that the subjects' score on a "confidence in science" measure did predict their willingness to write counter-attitudinal essays.  

The also found that "need for closure"-- a self-report measure of cognitive style that consists of agree-disagree items such as "When thinking about a problem, I consider as many different opinions on the issue as possible" -- did not predict any lesser or greater willingness to advocate for the superiority of the "other side's" Presidents.

These additional findings are relevant to the discussion we've been having about dispositions that might counteract the "conformity" effects associated with cultural cognition & like forms of motivated reasoning.

One shortcoming -- easily remedied -- relates to BC's reporting of their results.  There are some cacophonous bar charts that one can inspect to see the impact (or lack thereof) of ideology on the subjects' willingeness to write counter-attitudinal essays.  

But the magnitudes of the other reproted effects are not readily discernable.  In the case of the "confidence in science" result, the authors report only a logit coefficient for an interaction term (in a regression model the full output for which is not reported).  Even people who know what a logit coefficient is won't be able to gauage the practical significance of a result reported in this fashion (& what a shame to relate one's findings exclusively in a metric only those who "read regression" can understand, for they comprise only a tiny fraction of the world's curious and intelligent people).

For the need-for-cogniton closure result, the authors don't report anything except that the relevant interaction term in an unreported regression model was non-significant.  It is thus not possible to determine whether the effect of "need for closure" might have been meaningfully associated with aversion to engaging dissonant evidence & failed to achieve "statistical significance" due to lack of an adequately large sample. 

These sorts of reporting problems are endemic to social psychology, where papers typically obsess over p-values & related test statistics & forgo graphic or other reporting strategies that make transparent the nature and strength of the inferences that the data support.  But I've seen worse, and I don't think the reporting here is hiding some flaw in the BC study-- on the contrary, it is concealing the insight that one might derive from it!

The last thing I can think of to say--others should chime in-- is that is super unfortunate that BC, like NJV-B, relied on a Mechanical Turk "workforce" sample.  

As I've written previously, selection bias, repeat exposure to cognitive style measures, and misrepresentations of nationality make MT samples an unreliable (invalid, I'd say) basis for testing hypotheses about the interaction of cognition and political predispositions.

Brandt and Crawford have done several super cool studies on the "asymmetry thesis" (herehere & here,  e.g.).  They are sharp cookies.  

So they should definitely not waste their time -- and their ingenuity -- on junky MT samples.


Have Republicans changed views on evolution? Or have creationists changed party? Pew's (half-released) numbers don't add up ... 

Okay. Something does not compute.

Last few days everybody is chortling about a shift in % of Republicans who say they don't believe in evolution.  

According to Pew Research Center, a higher percentage of Republicans agreed with the statement that "humans ... have existed in their present form since the beginning of time"  in 2013 than in 2009.

One fairly annoying thing is that the information that Pew disclosed about the survey makes it impossible to determine what percentage of Democrats actually believe in "naturalistic" as opposed "theistic" evolution.

Pew's survey item is bifurcated.  First, survey participants respond to the question, "Which comes closer to your view? Humans and other living things have [1a] evolved over time [OR] [1b] Humans and other living things have existed in their present form since the beginning of time?"  Those who select [1a], are then asked: 

And do you think that [2a] Humans and other living things have evolved due to natural processes such as natural selection, or [2b] A supreme being guided the evolution of living things for the purpose of creating humans and other life in the form it exists today?

In both 2009 & 2013, those who selected answer 1a-- "evolved over time" -- split about 60:40 as between 2a & 2b-- the "naturalistic" and "theistic" versions of evolution, respectively.
As a result, only 32%, in both surveys, indicated that the believed in the "naturalistic" position that "Humans and other living things have evolved due to natural processes such as natural selection."

Pew tells us in the most recent survey (in its web page summary and in its Report ) that only 27% of Democrats selected 1a, the "creationist" position that "Humans and other living things have existed in their present form since the beginning of time." It also tells us that 67% of Democrats, "up" from 65% in 2009, "believe in evolution," or in other words that 2/3 of them selected 1b.
But it doesn't tell us -- not on its web page summary, not in the body of its Report, not in the reported "toplines"; not anywhere -- what % of Democrats chose the "naturalistic" (2a) and what % the "theistic" (2b) evolution positions.

Frankly, that's lame.

It's lame, first, because the answer to that question is really interesting and important if one is trying to make sense of how ordinary Americans reconcile their cultural identities, which are indicated by both their political affiliations and their religious practices (among other things), with belief in science. 

Second, it's lame because this sort of deliberate selectivity (make no mistake, it was deliberate: Pew 
made the decision to include the partisan breakdown for only half of the bifurcated evolution-belief item) subsidizes the predictable "ha ha ha!" response on the part of the culturally partisan commentators who will see the survey as a chance to stigmatize Republicans as being distinctively "anti-science."

If in fact, only a minority of Democrats are willing to endorse "naturalistic" evolution -- if a majority of them refuse to assent to a theory of human beings' natural history without God playing a role in guiding it -- then that makes "ha ha ha ha ha!" seem like an unreflective response to a complicated and interesting phenomenon.

But actually, Pew lulled those who are making the response into being this unreflective by deliberately (again, they had to decide to report only a portion of the evolution-survey item by political affiliation) failing to report what % of Democrats who indicated that they "believe in evolution" accept the "naturalistic" variant.
I'd be surprised if more than a minority did.  That would be a significant break with past survey results. For a majority of Democrats to be "naturalistic" evolutionists, they would have to outnmber "theistic" Democrats by a margin of 3:1.

But hey-- I'd love to be surprised, too!  An unchanging world is dull. 

But a world that doesn't change in its catering to petty cultural partisanship is both dull & disappointing. 

All that aside
, the finding that a greater proportion of Republicans now believe in "creationism" -- & not either theistic or naturalistic evolution -- than in 2009 is pretty darn interesting! 

But what exactly has changed? 

There are two obvious possibilities: [A] Republicans are "switching" from belief in evolution (naturalistic or theistic) to creationism; or [B] creationists are switching their party allegiances from Democrat or Independent to Republican &/or evolutionalists (theistic and naturalistic) are switching from Republican to Democrat or Indepedent.

Either [A] or [B] would be really interesting, but they would reflect very different processes. 

So which is it?

Pew doesn't tell us directly (why?! I don't get the attitude of this Report; very un-Pewlike) but we should be able to deduce the answer from what they do report -- the population %s and the partisan breakdowns on "creationism" in 2009 and 2013.

Logically, if the fraction of the overall U.S. population who identifies as creationist stayed same, & more Rs are now identifying creationists, then [B]-- party-shifts by either evolutionists, creationists, or both -- must be correct.  

And in that case,the proportion of Ds & Is who are creationists would have to be correspondingly lower.

Alternatively, If the proportion of Rs who are creationists went up but the proportion of Ds & Is who are creationists stayed same, then [A]-- Republicans are changing position -- would be the right answer. 

And logically, in that case, the % of the U.S. public overall who now say they are "creationists" would have had to have gone up.

Now that would be truly surprising -- huge news -- because the %s on creationism-vs-evolution haven't changed for decades.

But not surprisingly, Pew reports that "the share of the general public that says that humans have evolved over time is about the same as it was in 2009, when Pew Research last asked the question.":

The same fraction of the U.S. public -- approximately 1/3 -- believes in "naturalistic" evolution today as did then. The 33% who selected the "creationist" response to the bifurcated survey item in 2013 is statistically indistinguishable from the 31% who did in 2009.

So ... if the population frequency of creationism didn't increase, and the proportion of Republican's who now identify as "creationists" did, either creationists are switching to the Republican party or "evolutionists" (theistic or naturalistic) must be switching to Democrat or Independent -- option [B].

But, logically, then, the proportion of "evolutionsists" who are now identifying as either Democrat or as Independent must have risen by an amount corresponding to the increase in "creationists" now identifying as Republican, right?

Nope. Pew says that the division of "opinion among both Democrats and independents has remained about the same":


So if the percentage of Democrats and Independents who identify as creationist has stayed constant, and the proportion of Republicans has increased, [A] --Republicans are "switching" their views on evolution-- must be the answer!

But if the proportion of Republicans who are creationists has significantly increased while the division of "opinion among both Democrats and independents has remained about the same," the total proportion of the population that embraces creationism must be significantly higher. . . . Except that Pew says  "the share of the general public that says that humans have evolved over time is about the same as it was in 2009, when Pew Research last asked the question."

So, something does not compute.

At a minimum, Pew has some 'splainin to do, if in fact it is trying to edify people rather than feed the apptetite of those who make a living exciting fractious group rivalries among culturally diverse citizens.

Has anyone else noticed this?

Right away when I heard about the Pew poll, I turned to the results to see what the explanation was for the interesting -- truly! -- "shift" in Republican view: Were Republicans changing their positions on creationism or creationists changing their party allegiance?

And right away I ran into this logical inconsistency.

Surely, someone will clear this up, I thought.  

But no.  

Just the same predictable, boring "ha ha ha ha!" reaction.

Why let something as silly as logic get in the way of an opportunity to pound one's tribal chest & join in a unifying, polarizing group howl? 

Happy New Year, Liberal Republic of Science ....



"Clueless bumblers": Explaining the "noise" in a polluted science communication environment...

So the question is: what explains the resistance of some individuals to the sort of conformity effects that are the signature of cultural cognition & like forms of motivated reasoning?  

To ground the question, I posed it as a challenge to come up w/ some testable hypothesis that would explain visible "outliers" in a couple of data sets, one that correlated environmental risk perceptions and cultural outlooks and another that correlated right-left political outlooks and "policy preferences" (positions on a set of familiar, highly contested political issues like climate change, gun control, affirmative action, etc.) 

Quite reasonably, the first conjecture -- advanced with palpable ambivalence by @Jen -- was that the "outliers" are people with an independent cast of mind, ones who resist "going with the crowd" and instead form positions on the basis of knowledge of, and reflection on, the evidence.

Well, of course I have measures of "cognitive reflection" and "political knowledge."

The  "cognitive reflection test" (CRT) is considered by many psychologists and behavioral economists to be the "gold standard" for measuring the disposition to use effortful, conscious forms of information processing ("System 2") as opposed to intuitive, heuristic-driven ("system 1") ones.  

If the "outliers" are people disposed to critically interrogate intuitively congenial assessments in light of available information, then we might expect them to have higher CRT scores.

Indeed, consistent with this expectation, several papers (like this one, & also this, & this, & this too)  have now been published that use the negative correlation between CRT and religiosity to support the inference that those who are highly religious are less disposed to engage in the sort of critical reasoning associated with making valid use of empirical evidence. (These studies all seem pretty sound to me; but the reported effects always strike me as quite small & also much less interesting than those associated with the interaction of religiosity & critical reasoning dispositions.)

The standard "political knowledge" test consists of a battery of very elementary civics/current-events questions (e.g., "How long is the term of office for a United States Senator? Is it two years, four years, five years, or six years?"; "Which party currently has the most members in the U.S. Senate?  Is it the Democrats, the Republicans, or neither one?").  

One might think that such questions would have no particular value -- either that "everyone" would know the answers or that in any case they are too simplistic to tap into the mix of motivations and knowledge that one might equate with a "sophisticated" understandings of matters political.  

But in fact, "political knowledge" has shown itself to be a highly discerning measure of the coherence of individuals' policy positions with one another and with their self-reported political outlooks and party attachments.  Use of the measure has played a very very significant role in informing the orthodox political science view that most members of the public are indeed intensely non-political and non-partisan, and hence motivating the project to understand how mass political preferences manage to display the sorts of regularities and order (such as "polarization" on various questions) that are so conspicuous in everyday life.

One answer to this question is that politically unsophisticated types "go with the crowd"-- by using various types of "cues" to orient themselves appropriately in relation to others who they experience some sort of affinity.  

As a result, we might think that the "outliers" -- the individuals who resist forming the "off the rack" clusters of views that are in effect badges of membership in one or another cultural or like affinity group -- would likely be high in political knowledge, and thus less dependent on "group views" to guide them in forming perceptions of risk or positions on largely utilitarian policy questions like whether "concealed carry laws increase crime-- or decrease it."

But as plausible as these conjectures are, they are wrong.  Or in any case, if we use CRT and political knowledge to test the "independence of mind" hypothesis, the data featured in the last post do not support that account of why the outliers are outliers.  On the contrary, those measures strongly support a conjecture that is diametrically opposed to it -- viz., that the outliers are "clueless bumblers" who lack the knowledge & collection of reasoning dispositions necessary to rationally pursue an important element of their own well-being....


This is another scatter plot based on the data reported in the last post to illustrate the correlation between environmental risk perceptions and cultural worldviews.  But now I've color-coded the observations -- the individual study participants-- in a manner that reflects their scores on a "long form" version (10 items rather than 3) of the CRT.

I am a statistical model of a polluted science communication environmentAs can be seen from the color of the observations inside the "outlier circles" (which are position in the same place as last time), the "outliers" are definitely not high in cognitive reflection.  On the contrary, they consist disproportionately of low-scoring respondents.  

High-scoring ones -- those in the 90th percentile and above -- are more likely to be "conformers."  Indeed, this can be seen from the regression lines that I've superimposed on the scatter plot. The effect isn't super strong, but they show that CRT magnifies the polarizing influence of cultural predispositions on environmental risk perceptions (an impact the "statistical significance" of which is reflected in the regression analysis that you can inspect by clicking on the image to the right).

Next, consider this:

Using the data that I reported last time to illustrate the connection between right-left political outlooks and "policy preferences," I've now color-coded the respondents based on their political knowledge scores.  

me too!Again, the "outliers" are not more politically sophisticated but rather considerably less so than the conformers.  The impact of political knowledge in amplifying the fit between political outlooks (measured by a scale that aggregates study particiants' responses to standard liberal-conservative ideology and partisan self-identification measures) & policy preferences is pretty darn pronounced (and measured in this regression).

These results shouldn't be a surprise-- and indeed, @Jen's trepidation in assenting to these ways of testing the "independence of mind" hypothesis reflected her premonition that they would likely be highly unsupportive of it.

On political knowledge, all I've done here is reproduce the conventional political-science wisdom that I referred to earlier.  "Political knowledge" amplifies the coherence of ordinary individuals' policy preferences and their fit with their self-professed political leanings.  So necessarily, those higher in political knowlege will display greater conformity in this regard, and those lower less.

But why exactly? This is an issue on which there is interesting debate among political scientists.

The traditional view (I guess it's that, although the scholars who started down this road were clearly departing from a traditional, and psychologically crude understanding of mass political opinion) is that those higher in "political knowledge" are "better informed" and thus able more reliably to connect their policy views to their values.

But another approach sees political knowledge as merely an indicator of partisanship.  People who are disposed to form highly coherent -- extremely coherent -- policy preferences to gratify their disposition to experience and express a partisan identity are more likely to learn about current events, etc.  

But they aren't necessarily making "better"use of information.  Indeed, they could well be making worse use of it, if the coherence that their policy positions reflect derives from some species of biased assessment of evidence.

This is now a position gaining in strength.  It is reflected in the very interesting & wonderful book The Rationalizing Voter by Taber & Lodge.

But the impact of cognitive reflection in mangifying this form of coherence is not what one would expect under T&L's "rationalizing voter" view.

Without reflecting on the possibility of any alternative, T&L embed politically motivated reasoning in the conventional "system 1/system 2" dual process theory of cognition.  For them, the tendency of partisans to fit evidence to their political predispositions reflects their over-reliance on heuristic-driven and bias-prone "system 1." "Political knowledge" magnifies motivated reasoning because, on their view, it is a measure of partisanship, and thus of the strength of the motivation that is biasing information processing.

If this were correct, however, then we should expect partisans who score higher in CRT to show less conformity or coherence in their views.  Those who score high in CRT are more disposed to use effortful, conscious "System 2" reasoning, which reduces their vulnerability to the cognitive biases that plague system 1 thinking.  If, as T&L posit, politically motivated reasoning is a system-1 form of bias, then its effects ought to abate in those who score highest in CRT.

Or in other words, on T&L's view, our "outliers" should be high in CRT. But they aren't. On the contrary, the outliers have the lowest CRT scores!

But this shouldn't come as a surprise either, at least to the 14 billion readers of this blog.

The reason CRT amplifies cultural cognition is that cultural cognition & like forms of motivated reasoning are not a bias at all. They are elements of information processing that predictably and rationally advance individuals' interests.

What an individual believes about the impact of carbon emissions on global warming, the safety of nuclear power, etc. has zero impact on the risk that person or anyone he or she cares about faces.  That's because the influence that that individual (pretty much any individual) has as consumer, voter, public conversant, etc. is too inconsequential to have any measurable impact on the activities that generate those risks or the adoption of policies intended to mitigate them.

But if an ordinary person makes a mistake about a "fact" that has come to be viewed as a symbol of his or her membership in & loyalty to an important affinity group, then that person's life could be miserable indeed. That person can expect to be viewed with distrust by those he or she depends on, and thus ostracized and denied all manner of benefit, material and emotional.

Perfectly rational for a person in that situation (the situation is not rational--it is collectively irrational; it is not "normal"-- it is "pathological"; it is tragic) to use his or her knowledge and reasoning abilities to give appropriate effect to evidence that promotes formation and persistence in beliefs that express her identity. 

And if he or she is more adept at cognitive reflection or some other element of critical reasoning, then we should expect that person to do an even better job of such fitting.  

This, of course, is the "expressive rationality thesis" that informed the CCP studies on the relationship between cultural cognition and science comprehension.  

The studies consist of observational ones demonstrating that cultural polarization increases as people become more "science literate" & experimental ones showing that the reason is that they are using their critical reasoning dispositions--including cognitive reflection and numeracy--in an opportunistic way that more reliably fits their beliefs to the ones that predominate in their group than to the best available evidence. 

My surmise is that the "political knowledge" battery does measure (even if crudely) elements of knowledge (or at least the disposition to attain it) that individuals need to have in order to form identity-congruent beliefs on disputed issues of risk and like facts.  Political knowledge magnifies coherence in policy preferences, on this view, not because it generates a biasing form of motivation -- the T&L position -- but because rational people can be expected to use their greater knowledge to promote their well-being.

So what about the outliers?

On this account, they are sad, clueless bumblers.  They lack the knowledge and reasoning dispositions to reliably form beliefs that advance their expressive interests.

They aren't reflective and independent thinkers; they are "out to lunch."

And I bet their lives are filled with misery and solitude....

Mine is, too, when I reach this sort of conclusion.

So give me some more hypotheses.

Give me some alternative measures for "independence of mind" and alternative strategies for using them to test whether there might still be some as-yet unidentified element of critical reasoning that resists cultural cognition, or at least its complicity in the effacement of reason associated with a polluted science communication environment.

And better still, use your reason to formulate and test and implement strategies for removing the pathological conditions that divert to such a mean & meaningless end the faculties that make it possible for us to know. 




Can someone explain my noise, please?

Okay, here's a great puzzle.

This can't really be a MAPKIA! because I, at least, am not in a position to frame the question with the precision that the game requires, nor do I anticipate being in a position "tomorrow" or anytime soon to post "the answer."  So I'll treat "answers" as WSMD, JA! entries.

But basically, I want to know what people think explains the "noise" in data where "cultural cognition" or some like conception of motivated reasoning explains a very substantial amount of variance.

To put this in ordinary English (or something closer to that), why do some people with particular cultural or political orientations resist forming the signature risk perceptions associated with their orientations?

@Isabel said she'd like to meet some people like this and talk to them.

Well, I'll show you some people like that.  We can't literally talk to them, because like all CCP study participants, the identities of these ones are unknown to me.  But we can indirectly interrogate them by analyzing the responses they gave to other sorts of questions -- ones that elicited standard demographic data; ones that measured one or another element of "science comprehension" ("cognitive reflection," "numeracy," "science literacy" etc); ones that assess religiosity, etc. -- and by that means try to form a sense of who they are.

Or better, in that way test hypotheses about why some people don't form group-identity-convergent beliefs.  

Here is a scatter plot that arrays about 1000 "egalitarian communitarian" (green) and "hierarchical individualist" (black) outlooks (determined by their score in relation to the mean on the "hierarchy-egalitarian" and "individualist-communitarian" worldview scales) in relation to their environmental risk perceptions, which are measured with an aggregate Likert scale that combines responses to the "industrial strength" risk perception measure as applied to global warming, nuclear power, air pollution, fracking, and second-hand cigarette smoke (Cronbach's alpha = 0.89). 

You can see how strongly correlated the cultural outlooks are with risk perceptions.  

click me ... click me ... click me ...When I regress the environmental risk perception measure on the cultural outlook scales (using the entire N = 1928 sample), I get an "impressively large!" R^2 = 0.45  (to me, any R^2 that is higher than that for viagra use in explaining abatement of "male sexual dysfunction" is "impressively large!"). That means 45% of the variance is accounted for by cultural worldviews -- & necessary that 55% of the variance is still to be "explained."

But here's a more useful way to think of this.  Look at the folks in the dashed red "outlier" circles.  These guys/gals have formed perceptions of risk that are pretty out of keeping with that of the vast majority of those who share their outlooks.

What makes them tick?

Are these folks more "independent"-- or just confused?

Are they more reflective -- or less comprehending?

Are they old? Young? Male? Female? (I'll give you some help: those definitely aren't the answers, at least by themselves; maybe gender & age matter, but if so, then as indicators of some disposition or identity that can be pinned down only with a bunch more indicators.)

The idea here is to come up with a good hypothesis about what explains the outliers.

A "good" hypothesis should reflect a good theory of how people form perceptions of risk.  

But for our purposes, it should also be testable to some extent with data on hand.  Likely the data on hand won't permit "perfect" testing of the hypothesis; indeed, data never really admits of perfect testing!

But the hypotheses that it would be fun to engage here are ones that we can probe at least imperfectly by examining whether there are the sorts of correlations among items in the data set that one would expect to see if a particular hypothesis is correct and not if some alternative hypothesis is.

I've given you some sense of what other sorts of predictors are are in the dataset (& if you are one of the 14 billion regular followers of this blog, you'll be familiar with the sorts of things that usually are included).  

But just go ahead & articulate your hypothesis & specify what sort of testing strategy --i.e., what statistical model -- would give us more confidence than we otherwise would have had that the hypothesis is either correct or incorrect, & I'll work with you to see how close we can get.

I'll then perform analyses to test the "interesting" (as determined by the "expert panel" employed for judging CCP blog contests) hypotheses.

Here: I'll give you another version of the puzzle.

In this scatterplot, I've arrayed about 1600 individuals (from a nationally representative panel, just like the ones in the last scatterplot) by "political outlook" in relation to their scores on a "policy preferences" scale.

The measure for political outlooks is an aggregate Likert scale that combines subjects' responses to a five-point "liberal conservative" ideology measure and a seven-point "party identification" one (Cronbach's alpha = 0.73).  In the scatterplot, indivduals who are below the mean are colored blue, and those above red, consistent with the usual color scheme for "Democrat" vs. "Republican."

The measure for "policy preferences" has been featured previously in a blog that addressed "coherence" of mass political preferences.

It is one of two orthogonal factors extracted from responses to a bunch of items that measured support or opposition to various policies. The "policies" that loaded on this factor included gun control, affirmative action, raising taxes for wealthy people, and carbon-emission restrictions to reduce global warming. The factor was valenced toward "liberal" as opposed to "conservative" positions.

The other factor, btw, was a "libertarian" one that loaded on policies like legalizing marijuana and prostitution (sound familiar?).

So ... what "explains" the individuals in the dashed outlier circles here-- which identify people who have formed policy positions that are out of keeping with the ones that are typical for folks with their professed political outlooks?

click me!!! C'mon!!!!The R^2 on this one is an "impressively large!" 0.56.  

But hey, one person's noise is another person's opportunity to enlarge knowledge.

So go to it! 


Wastebook honorable mentions: more "federally funded" CCP blog posts!

"Doh!" (Click on me!)Okay, okay, I know shouldn't be gloating that my 15x10^3-mins-of-fame, nonfederally-funded "tea party science literacy" post helped reveal the meticulous care with which Sen. Coburn's federally funded staff compiled his annual "Wastebook."

For the truth is, I really dodged a bullet on this one!

Included very near the top of the "honorable mention" appendix for this yr's Wastebook were three additional "federally funded studies" featured in this blog!  

If the diligent member of Coburn's staff who compiled the book had caught his or her innocent, completely understandable error (true, there was nothing in the "tea party science literacy" post that said it was "federally funded," and only someone skipping every other sentence would have missed the statement that the data came from a CCP study of vaccine risk perceptions; but there really should have been a big warning in flashing neon at the top-- "NOT FEDERALLY FUNDED!" My bad!) & included any of these other three, I, rather than Sen. Coburn, Greta Van Sustern & "former Congressman" Allen West, would now be the one who looks like a complete idiot! 

All I can say is, "Phew!"

But in the spirit of full disclosure, here's a brief run down of the disturbingly wasteful CCP blog posts that made the Wastebook honorable mention list:

1. Synbioipad.  A "framing" study designed to see if fusing (literally) synthetic biology (cool math-problem-solving E. coli!)  into a wildly popular Apple product could head off public fear of this new technology (hey-- it worked with the "nanoipad"!).

Cost: $14.32.  

Agency sponsor: Department of Commerce.  

Result: None; subjects failed to complete the study after contracting unanticipated gastrointestinal symptoms that required hospitalization.


2. "Bumblebee--my first drone!"  Experiment to counteract instinctive disgust sensibilities of egalitarian individualists toward drones by disguising them as a delightfully fun children's "toy."  

Cost: $125,000,000.14.  

Agency sponsor: NSA

Result: Complete failure.


3. Macrotechnology risk perceptions.  Exploratory study to determine whether there was anything that white hierarchical individualist males are not afraid of.

Cost: - $13,000,000 (amount of fine imposed on CCP Lab by EPA). 

Agency sponsor: EEOC.

Result: Experimental stimulus ate Akron Ohio



Not very reflective tea-party/Republicans 

These federally funded studies were not on the "cognitive skills of Tea Party members" (they are nowhere mentioned in them):


This blog post is not a federally funded study (it's neither federally funded nor a study):


These tea party/republicans are apparently not very bright (but don't draw any inferences; it's a biased sample!):


But all of this is pretty amazing. Someone should do a study of how so many genuinely reflective people (Rs, Ds, TPs, ECs, HIs, whatever) could become so confused.  NSF could fund it.




What is a "cultural style"? And some thoughts about convergent validity

what do you mean by cultural styles? As a qualitative researcher, that caught my eye! Thanks.

A commenter recently posed this question  in connection with a post from a while back. I thought the question was interesting enough, and the likelihood that others would see it or my response sufficiently remote, that I should give my answer in a new post, which I hope might prompt reflection from others.

My response:

That's a great question!

It goes to what it is that I think is being measured by scales like ours. I've addressed this to some extent before-- e.g., here & here & here & ...

But basically, we can see that on disputed risk issues, positions are not distributed randomly but instead correlated with reocognizable but not directly observable ("latent") group affinities that are themselves associated loosely with a package of individual characteristics and attitudes.

People who share particular group affiniteis, moreover, form clusters of positions across these issues ("earth not heating up" & "concealed  carry laws reduce crime"; "the death penalty doesn't deter murder" & "nuclear wastes can't be stored safety in deep geologic isolation") that can't possibly reflect links in the causal mechanisms involved and instead seem to reflect the identity-expressing equivalence of them.

The point of coming up w/ scales is to sharpen our perception of what these group affinities are & why those who share them see things the way they do -- to explain what's going on, in other words -- & also to enhance our power to predict and form prescriptions.

The term "cultural style" is, for me, a way to describe these affinities. I have adapted it from Gusfield. I & collaborators use the concept and say more about it and how it relates to Gusfield in various places.

“Unlike groups such as religious and ethnic communities[,] they have no church, no political unit, and no associational units which explicitly defend their interests,” but are nevertheless affiliated, in their own self-understandings and in the views of others, by largely convergent worldviews and by common commitments to salient political agendas. 

" 'They "posssess subcultures' " (id.) that
furnish coherent norms for granting and withholding esteem. "Examples of these are cultural generations, such as the traditional and the modern; characterological types, such as 'inner-directed and other-directed'; and reference orientations, such as 'cosmopolitans and locals.'" Many of the most charged social and political issues of the past century can be understood as conflicts between individuals who identify with competing cultural styles and who see their status as bound up with the currency of those styles in society at large.

Dan M. Kahan, The Secret Ambition of Deterrence, 113 Harv. L. Rev. 413, 442 (1999) (quoting Gusfield, who is himself quoting David Reisman, Karl Mannheim & C. Wright Mills-- yow! right after the quoted section, Gusfield discusses as an example Hofstadter's famous "Mugwump style").

Joseph Gusfield -- he rocks!BTW, I regard Gusfield as one of the most brilliant social theorists of our time. It is sad that he is not even more famous. But I suppose lucky, too, for me b/c it means I am able to play a more meaningful role in scholarly discussions by virtue of others not having the advantage of the perspective & insight that comes from reading Gusfield!

I like "cultural style" b/c it helps to reinforce that the orientation in question is relatively loose-- we are talking about a style here; not the sort of fine grained, highly particular set of practices & norms that, say, an anthropologist or sociologist might have in mind as "culture"  -- and also general -- a "style" doesn't reduce in some analytic sense to a set of necessary & sufficient conditions; it is a prototype.

You say you are a qualitative researcher. I take it then that you regard me as a "quantitative" one.  Fair enough.

But in fact, I see myself as just a researcher-- or simply a scholar. I want to understand things, and also to add to scholarly conversation by others who are interested in the same things as a way to reciprocate what I have learned from them.

To do that -- to learn; to add -- I figure out the method most suited to investigating questions of interest to me and invest the effort necessary to be able to use that method properly. Then I just get to it.

Any scholar who thinks that the methods he or she has learned should forever determine the questions he or she should answer rather than vice versa will, at best, soon become boring and, at worst, ultimately become absurd.

Actually, all valid methods, I'm convinced, are empirical in nature, since I don't believe one can actually know anything without being able to make observations that enable valid inferences to be drawn that furnish more reason to credit one account of a phenomenon than another (pending more of the same sorts of evidence, etc.).

I have found the sort of empirical methods that figure in the cultural cognition work very useful for this. And those methods, moreover, have evolved and been refined in various ways to try to meet challenges that we face in seeking to learn/add in the professional student way.

But in fact, I believe that the sorts of ethnographic, historical and related methods that figure in anthropological and sociological accounts and the fact-rich social theorizing that Gusfield has done to be very valid as well.

Indeed, there are few if any hypotheses that we have tested with the sorts of quantitative methods that figure in our cultural cognition work that aren't rooted in insights reflected in these more "qualitative" works.

Gusfield's account of the styles that contended over the issue of temperance--which he identifies as the same ones in conflict over various other issues, including many involving criminal deviancy lawsdrunk driving lawsanti-smoking laws, and other forms of risk regulation--is a source of inspiration for many of our conjectures, as I've indicated.

So is the work of Kristin Luker, whose understanding of the competing egaltiarian & hierarchic styles that impel conflict among women over abortion figured in our study of the white male effect and later in a study that I did of cultural contestation over rape law.

But there are many many other works of this sort that motivate & discipline our studies.

The disciplining consists in the fit between our study results and these accounts.  That correspondence helps to make the case that we really are measuring what we say we are measuring-- or modeling what we say we are modeling.

At the same time, our results give more reason to believe that the qualitative accounts are valid.

For any "qualitative style" (as it were) of empirical investigation, the issue of whether the researcher's own expectations shaped his or her observations rather than vice versa always looms menacingly overhead like a raised sword.

That we are able to build a simple empirical model that displays the characteristics--produces the results-- one would expect if the qualitative researcher's explanation of what's going on is true helps to shield the researcher from this sort of doubt.  I hope qualitiative researchers find value in that!

I am, of course, talking about the idea of convergent validity.

Every empirical method has limits that are in part compensated for by others.  When different approaches all generate the same result, there is more reason to believe not only that that what they are finding is true but that each of the individual approaches used to establish that finding were up to the job.

It's possible that a bunch of imperfect methods (the limitations of which are independent of one another) just all happened to generate the same result. But the more likely explanation is that they converged because they were in fact all managaing to get a decent-sized piece of the truth.

Would you like a more "Bayesian" analogy of how convergent validity validates?

You find something that looks a puzzle piece but aren't sure whether it is.  I find something that looks like a nearly complete puzzle--but also am unsure.  If we meet and discover that the former happens to fit into and seemingly complete the latter, you will have more reason for believing that the putative "puzzle piece" is in fact a puzzle piece. At the same time, I will have more reason for believing that my putative "incomplete puzzle" is truly an incomplete puzzle.  That's because the probability that a thing that isn't a puzzle piece would just happen to fit into a thing that isn't an incomplete puzzle is lower than the probability that the two things truly are "a puzzle piece" and "an incomplete puzzle" respectively.

To me convergent validity is the "gold standard." Or better the remedy for the sort of "gold standard" mentality that manifests itself in a chauvinistic insistence that there is only one genuinely valid one or even a single "best" for empirical investigation of social phenomena.

... Well, I am curious how this strikes you.

Useful? Eclectic? Confused?!



The value of civic science literacy

Gave talk Wednesday at AGU meeting in San Francisco. Slides here. I was on panel w/ a bunch of talented scholars doing really great studies on teaching climate science. The substance of what to teach (primarily in context of undergraduate science courses) was quite interesting but what was really cool was the their (data-filled) account of the "test theory" issues they are attacking in developing valid, reliable, and highly discriminant measures of "climate science literacy" ("earth is heating up," "humans causing," "we're screwed," they recognized, don't reliably measure anything other than the attitude "I care/believe in global warming"). My talk wasn't on how to impart climate science literacy but rather on what needs to be done to assure that a democratic society gets the full value out of having civically science literate citizens: protect the science communication environment-- a matter that making citizens science literate does not itself achieve. (Gave another talk later at The Nature Conservacy's "All-Science" event but will have to report on that "tomorrow.") Here's what I more-or-less remember saying at AGU:

If this were the conversation I'm usually a part of, then I'd likely now be playing the role of heretic.

That discussion isn't about how to teach climate science to college students but rather about how to communicate climate risks to the public.

The climate-risk communication orthodoxy attributes public controversy over global warming to a deficit in the public's comprehension of science. The prescription, on this view, is to improve comprehension—either through better science education or through better public science communication. 

I’ll call this the “civic science literacy” thesis (or CSL).

I’m basically going to stand CSL on its head.

Public controversy, I want to suggest, is not a consequence of a deficit in public science comprehsnion; it is a cause of it. Such controversy is a kind of toxin that disables the normally reliable faculties that ordinary citizens use to recognize valid decision-relevant science.

For that reason I'll call this position the “science communication environment” thesis (or SCE).  The remedy SCE prescribes is to protect the science communication environment from this form of contamination and to repair it when such protective efforts fail.

This account is based, of course, on data—specifically a set of studies designed to examine the relationship between science comprehension and cultural cognition.

“Cultural cognition” refers to the tendency of people to conform their perceptions of risk to ones that predominate in important affinity groups—ones united by shared values, cultural or political. Cultural cognition has  been shown to be an important source of cultural polarization over climate change and various other risks.

In a presentation I made here a couple of years ago, I discussed a study that examined the connection between cultural cognition and science literacy, as measured with the standard NSF Science Indictors battery.In it, we found that polarization measured with reference to cultural values, rather than abating as science literacy increases, grows more intense. 

This isn’t what one would expect if one believed—as is perfectly plausible—that cultural cognition is a consequence of a deficit in science comprehension (the CSL position).

The result suggests instead an alternative hypothesis: that people are using their science comprehension capacity to reinforce their commitment to the positions on risk that predominate in their affinity groups, consistent with cultural cognition.

That hypothesis is one we have since explored in experiments. The experiments are designed to “catch” one or another dimension of science comprehension “in the act” of promoting group-convergent rather than truth- or science-convergent beliefs.

In one, we found evidence that “cognitive reflection”—the disposition to engage in “slow” conscious, analytical reasoning as opposed to “fast” intuitive, heuristic reasoning—has that effect.

But the study I want quickly to summarize for you now involves “numeracy” and cultural cognition. “Numeracy” refers not so much to the ability to do math but to the capacity and disposition to use quantitative information to draw valid causal inferences.

In the study, we instructed experiment subjects to analyze results from an experiment. Researchers tested the effectiveness of a skin rash cream to a “treatment” condition and a “control” condition. They recorded the results in both conditions.  Our study subjects were then supposed to figure out whether treatment with the skin cream was more likely to make the patients’ rash “better” or “worse.”

This is a standard “covariance detection” problem. Most people get the wrong answer because they use a “confirmatory hypothesis” testing strategy: they note that more patients’ rash got better than worse in the treatment; also that more got better in the treatment than in the control; and conclude the cream makes the rash get better.

But this heuristic strategy ignores disconfirming evidence in the form of the ratio of positive to negative outcomes in the two conditions.  Patients using the skin cream were three times more likely to get better than worse; but those using  not using the skin cream were in fact five times more likely to get better. Using the skin cream makes it more likely that that the rash will get worse than not using it.

By manipulating the column headings in the contingency table, we varied whether the data, properly interpreted, supported one result or the other. As one might expect, subjects in both conditions scoring low in numeracy were highly likely to get the wrong answer on this problem, which has been validated as a predictor of this same kind of error in myriad real-world settings. Indeed, subjects weren’t likely to get the “right” answer only if they scored in about the 90th percentile on numeracy.

We assigned two other groups of subjects to conditions in which they were instructed to analyze the same experiment styled as one involving a gun control ban. We again manipulated the column headings.

You can see that the results in the “gun ban” conditions were  are comparable to the ones in the skin-rash treatments. But obviously, it’s noisier.

The reason is cultural cognition.  You can see that in the skin-rash conditions, the relationship between numeracy and getting the right answer was unaffected by right-left political outlooks.

But in the gun-ban conditions, high-numeracy subjects were likely to get the right answer only when the data, properly interpreted, supported the conclusion congenial to their political values.

These are the raw data.  Here are simulations of the predicted probabilities that low- and high-numeracy would get the right answer in the various conditions.  You can see that low-numeracy were partisans were very unlikely to get the right answer and and high-numeracy ones very likely to get it in the skin-rash conditoins—and partisan differences were trivial and nonsignificant.

In the gun-ban conditions, both low- and high-numeracy partisnas were likely to polarize. But the size of the discrepancy in the probability of getting the right answer was between low-numeracy subjects in each condition was much smaller than the size of the discrepancy for high-numeracy ones.

The reason is that the high-numeracy ones but not the low- were able correctly to see when the data supported the view that predominates in their ideological group. If the data properly interpreted did not support that position, however, the high-numeracy subjects used their reasoning capacity perversely—to spring open a confabulatory escape hatch that enabled them to escape the trap of logic.

This sort of effect, if it characterizes how people deal with evidence of a politically controversial empirical issue, will result in the sort of magnification of polarization conditoinal on science literacy that we saw in the climate-change risk perception study.

It should now be apparent why the CSL position is false, and why it’s prescription of improving science comprehension won’t dispel public conflict over decision-relevant science.

The problem reflected in this sort of pattern is not too little rationality, but too much. People are using their science-comprehension capacities opportunistically to fit their risk perceptions to the one that dominates in their group. As they become more science comprehending, then, the problem only gets aggravated.

But here is the critical point: this pattern is not normal. 

The number of science issues on which there is cultural polarization, magnified by science comprehension, is tiny in relation to the number on which there isn’t.

The reason people of diverse values converge on the safety of medical x-rays, the danger of drinking raw milk, the harmlessness of cell-phone radiation etc is not that they comprehend the science involved. Rather it's that they make reliable use of all the cues they have access to on what’s known to science.

Those cues include the views of those who share their outlooks & who are highly proficient in science comprehension.  That's why partisans of even low- to medium-numeracy don't have really bad skin rashes!

This reliable method of discerning what’s known to science breaks down only in the unusual conditions in which positions on some risk issue—like whether the earth is heating up, or whether concealed carry laws increase or decrease violent crime—become recognizable symbols of identity in competing cultural groups. 

When that happens, the stake that people have in forming group-congruent views will dominate the stake they have in forming science-congruent ones. One’s risk from climate change isn’t affected by what one believes about climate change because one’s personal views and behavior won’t make a difference. But make a mistake about the position that marks one out as a loyal member of an important affinity group, and one can end up shunned and ostracized.

One doesn’t have to be a rocket scientist to form and persist in group-congruent views, but if one understands science and is good at scientific reasoning, one can do an even better job at it.

The meanings that make positions on a science-related issue a marker of identity are pollution in the science communication environment.  They disable individuals from making effective use of the social cues that reliably guide diverse citizens to positions consistent with the best available evidence when their science communication environment is not polluted with such meanings.

Accordingly, to dispel controversy over decision-relevant science, we need to protect and repair the science communication environment.  There are different strategies—evidence-based ones—for doing that. I’d divide them into “mitigation” strategies and “adaption” ones.

Last point.  In saying that SCE is right and CSL wrong, I don’t mean to be saying that it is a mistake to improve science comprehension!

On the contrary.  A high degree of civic science literacy is critical to the well-being of democracy.

But in order for a democratic society to realize the benefit of its citizens’ civic science literacy, it is essential to protect its science communication environment form the toxic cultural meanings that effectively disable citizens’ powers of critical reflection.