follow CCP

Recent blog entries
Tuesday
Feb102015

Science of Science Communication 2.0, Session 5.1: the (mis)communication of vaccine-risk perception

It's been a while -- at least 45 mins -- since I've written about vaccine-risk perception and communication.  Well, conveniently, today's session -- #5 -- of  Science of Science Communication course ver 2.0 happens to be on this very topic.  To get the "virtual class" discussion going (real-space class is starting in 60 mins!), consider this case study, which was appended to this week's reading list:

Case Study: Vaccine Risk Communication

1. The Wakefield affair. In 1998, the medical journal the Lancet published a study by a team of researchers led by Andrew Wakefield that purported to establish a connection between the MMR vaccine and autism. Although the study was attacked almost immediately and ultimately was retracted by the journal, the Wakefield paper stoked considerable public anxiety in the U.K., where vaccination rates declined and the incidence of various childhood diseases increased.

2. U.S. Vaccine Risk Anxiety. The U.S. has experienced several pertussis or whooping cough epidemics in recent years as well as recurrent outbreaks of measles, a disease that was deemed eliminated in the U.S. in 2000.  These developments are widely attributed to the growing influence of anti-vaccination groups, high-profile members of which include political figures like Robert Kennedy, Jr. and the Hollywood celebrity Jenny McCarthy, which continue to disseminate the Wakefield study, supplemented with a wide variety of other materials that misrepresent vaccine safety.

Chronicling the misinformation efforts of the activists, public health professionals and journalists warn of a “growing crisis of public confidence” in childhood vaccines. More and more “mainstream parents” are availing themselves of permissive “moral” and “religious” exemptions to universal vaccination, these concerned observers report.  The resulting “erosion in vaccination rates,” a physician who has assumed a high-profile role as an opponent of anti-vaccine activists, national vaccination rates are at risk of dipping below the threshold necessary to preserve herd immunity.

3. The PUAA plan. “Progressives United Against Anti-Science” (PUAA) is a public interest group.  Combatting the anti-vaccine movement is one of its three missions. The other two are correcting the misinformation of climate change deniers and challenging efforts to add the teaching of “creation science” or “intelligent design” to public school curricula.

PUAA has formulated a two-prong plan to counteract the spread of vaccine refusal in the U.S. The first is a multi-million “social marketing campaign” consisting of high-exposure public service announcements, which would variously rebut misinformation about the risks posed by vaccines and seek to excite disapprobation toward parents who decline vaccination for endangering the public health generally.  The second prong of the PUAA plan would involve the promotion of state referenda to repeal “non-medical” exemptions to universal vaccination policies.

To fund its plan, PUAA is seeking a $500 million grant from the General Welfare Foundation (GWF). GWF is among the largest philanthropic NGOs in the U.S.  Promotion of public health is one of its primary interests.

PUAA  has also reached out to both the CDC and the American Medical Association. Because of the CDC’s and AMA’s important institutional roles in the public health establishment, PUAA believes, with good reason, that their communication of support for its plan would significantly increase the prospect of it obtaining funding from GWF.

4.  Issues. Should GWF fund the PUAA plan?  Should the CDC and AMA convey their approval?  Is there enough information to answer these questions?  If not, what additional information would be useful?

Monday
Feb092015

Let it be? Measuring partisan reactions to existing mandatory vaccination laws

In the last week or so, I’ve done somewhere between 2 and 406 blog posts on vaccine risk perceptions (I am indeed bored beyond description; I feel obliged to reciprocate, though, the admirable efforts of others who are trying to shield public discourse from the harm associated with fact-free assertions in this area).

The upshot is that, contrary to the empirically uniformed and reckless blathering of “news” reporters and commentators (not all reporters or commentators are engaged in this behavior!), there is no meaningful public conflict over vaccine safety.

Not only have U.S. vaccination rates held steady at over 90%--the public health target—for all recommended childhood immunizations for over a decade.

But there is also overwhelming consensus in the general population, and within every recognizable political and cultural subsegment of it, that vaccines are safe and make a vital contribution to public health.

But these are characterizations of public risk perceptions

Someone could—a commentator responding to one of my earlier posts did—reasonably ask about whether consensus on vaccine safety translates into consensus in favor of mandatory vaccination laws.

All U.S. states have such laws, requiring vaccination for mumps, measles, and rubella, along with various other childhood diseases, as a condition of school enrollment.  All have so-called “medical exemptions,” for children who have a condition that would make vaccination unsafe, and most “religious” and some “moral” exemptions as well.

Can we say that the same state of consensus exists on this public health regulatory regime?

I’ll show you some data in a sec.

But because this post exceeds what the 14 billion regular readers of this blog know is my usually strictly enforced limit of 250 words, I'll start with this helpful and succinct summary of my own interpretation of them:

1.  Yes, the same consensus supports the current state of the law on mandatory vaccination in the U.S.

2.  However, pursuing legislation to change the status quo—either by eliminating religious or moral exemptions or by eliminating mandatory vaccination laws —risks polarizing the public along familiar political/cultural lines.

3.  Launching childhood vaccines into the reason-eviscerating maelstrom of cultural status conflict that now characterizes issues like climate change, gun control, and the HPV vaccine would itself put this important aspect of our public health system in serious jeopardy. Accordingly, anyone who is considering initiating a campaign to change existing mandatory vaccination laws should (if they care about public health as opposed to making money by being employed to organize such a campaign; marketing and like consulting firms have a huge conflict of interest here) very carefully weigh the risk of that outcome against whatever benefits they might be hoping for in pursuing this course. 

Okay. Here is the evidence.

1. Members of the nationally representative sample that participated in the CCP Vaccine Risk Perceptions and Ad Hoc Risk Communication study were asked what they thought of mandatory vaccination policies.

They responded to a series of questions, which are described in full in the Report,, after first being supplied information (also described there) that was materially identical to the information I supplied above about how those policies work and about the varying forms of exemptions that states permit.  The questions were varied randomly in order.

By an overwhelming margin, the survey participants indicated that they favor existing laws

Specifically, 75% (± 5% at a 0.95 level of confidence) indicated agreed that they “support leaving existing laws on childhood vaccinations as they are.” 

What’s more, this was by far the dominant response across political lines:

The responses to this "let it be" item reflect exactly the same pattern that characterizes the general public’s responses to the study’s vaccine risk-perception items.

Indeed, I myself do not see the response to this item as measuring anything different from what the risk-benefit items measure.

Known as the “affect heuristic,” there is a strong tendency of people to form overall pro- or con- attitudes toward putative risk sources.  Rather than a consequence of their assessment of evidence about those putative risks, their affective orientations are likely to shape how they interpret evidence and form beliefs (Slovic, Peters, Finucane & MacGregor 2005; Loewenstein, Weber, Hsee & Welch 2001).

One can ask them different questions and elicit responses that reflect people’s affective orientations toward one or another risk.

But that’s really all that one is doing with any of these questions: eliciting a generic pro-con attitude.  It is in fact a mistake—one that non-scholarly opinion pollsters invite all the time to get attention or to try to manipulate public impressions of the public’s view on one or another policy issue—to try to take the wording of particular items at face value or to purport to draw inferences from one or another as opposed to all the items considered as a group.

The reactions of members of the public to childhood vaccines displays all the signatures of the affect heuristic.

Moreover, their affective orientation happens to be very positive in the vast bulk of the U.S. population.  The CCP Report estimated that approximately 80% of the population shares it.

Like the risk-benefit questions generally, the CCP study "let it be" item on existing mandatory vaccine laws elicited a positive response of around 80%.

Like the risk-benefit items, there was not much in the way of systematic variance in the "let it be" item—that is, the reasons why somewhere around 20% or 25% of the population didn’t support existing laws did not admit of meaningful explanation by individual demographic or cultural or political characteristics.

Maybe these are the same 20% or so of the population who indicated that “space aliens” or “time travelers” were involved in the disappearance of Malaysian Airlines flight 370 last summer.  No idea.

2.  There is an important “however,” however.

Study participants were also asked to consider items relating to specific proposals to alter existing mandatory vaccination laws.

Their responses to these did not evince support consensus for change of any sort.  Indeed, it revealed the striking absence of any particular consensus:

What's more, variance in the participants’ response did reflect differences in the participants’ political outlooks.  Consider this Figure, which uses a multivariate model to estimate the impact of partisan differences, including identifying as a member of the Tea Party, on the responses to the reform items:

The figure illustrates how much more or less an average "liberal Democrat," "Conservative Republican, non--Tea Party member, or  "Conservative Republican, Tea Party member" is to agree with the indicated reform than is the "average" member of the population--and hence how likely each of the former prototypical partisans is to disagree with each of the others.

These differences are modest in relation to those, say, on climate change.  But they aren’t trivial.

Now what to make of this?

Well, if you are thinking there’s some inconsistency between the participants’ responses to these “change the status quo” items and their responses to the “let it be” item, I agree!

My view is that the “change the status quo” items are not measuring the same thing as the "let it be” item. 

Psychometrically speaking, "change the status quo" items don’t “scale” with the “let it be” item or the general battery of risk-benefit items. That’s is, they don’t display the sort of covariance that one would expect if the items were measures of a common latent or unobserved disposition.

It makes more sense to view the "change the status quo" iems as measuring (in a noisy and attenuated form, particularly in relation, to say, climate change) the general attitudes that subjects of varying political outlooks have toward governmental regulations generally.

My view is that most people generally just don’t give much thought to vaccine laws, which (with the exception of the HPV vaccine; I’ll come back to that) have not been a matter of political contestation in American life.

So if one asks members of the public what they think of universal vaccination laws, they express the same warm fuzzy feeling of contentment and gratitude that characterizes their views toward childhood vaccines generally.  These are the sensibilities, I'm confident, that for over a decade in US has lead 90%+ of parents to get all recommended childhood vaccinations for their kids.

Survey questions about changing those law, starts to trigger affective resonances that relate to peoples’ cultural identities.  Like bad survey questions, those questions aren't measuring what people in society are talking about; but they are measuring something: who they are, culturally or politically speaking.

3.  That's an outcome that ought to cause a sense of tremendous apprehension in anyone who actually values the contribution that childhood vaccines make to public health in this country.

The entanglement of issues of risk with antagonistic cultural meanings is precisely what drives polarization on issues like gun control and climate change, where people’s positions are understood to be badges of membership in and loyalty to opposing groups (Kahan 2015, 2012).

This was exactly the dynamic that generated polarization over the HPV vaccine, the only universally recommended vaccine that isn’t part of the schedule of mandatory, school-enrollment vaccinations across the U.S. states (Kahan et al. 2010).

I’ve written extensively, in this blog and elsewhere (Kahan 2013), about the sad career of the HPV vaccine.

But the basic point is that the HPV vaccine suffered this fate b/c its manufacture, Merck, decided to initiate a controversy-inviting nationwide legislative campaign to add the vaccine to the schedule of required vaccinations that states make a condition of school enrollment.

The company did so through what was initially a covert lobbying campaign that featured Governor “Oops” among others—the disclosure of which amplified the controversy the manufacturer was hoping to sidestep when it thrust the vaccine into the political process.

Merck didn’t have to do this. In the normal course, public health administrators, who are largely insulated from politics, would have almost certainly added the vaccine to their states’ requirements, as they had just finished doing for the HBV (hepatitis-b) vaccine, another STD immunization that was made mandatory for adolescents and then infants.

But the manufacture didn’t want to wait, because in extra time it would have taken to get such approval for a boys & girls vaccine added to the state mandatory lists by administrative action, a competitor, GlaxoSmithKline, would have obtained approval for a rival HPV vaccine and been able to compete for lucrative contracts with state school systems.

The risk that launching the HPV vaccine into the political process would disrupt people's ability to assess the risks and benefits of it was clear at the outset.

But Merck (for pefectly predictable, understandable reasons) decided to take a gamble, motivated by its desire to achieve a decisive commercial advantage in marketing the vaccine.

The company lost. 

We all did.

Should we take a similar gamble by taking action that would place childhood vaccinations into the cross-hairs of partisan politics? 

The CCP Report, which reported data similar to what I’ve just presented, discusses the considerations.  I don’t see any point trying to reformulate the conclusion stated there:

In striking contrast to responses to the other items in the survey, the ones soliciting participants’ positions on proposals to restrict non-medical exemptions were characterized by disagreement. Not only did these items tend to divide the respondents. They divided them on political and cultural lines....

[T]hese results supply reason for circumspection on the issue of exemptions. The power of these items to divide groups already conspicuously arrayed against each other on contested science issues raises the possibility that real-world proposals to restrict universal immunization exemptions could do the same. The study’s experimental component, which found that exposure to the “anti-science” op-ed intensified these divisions, reinforces this concern....

Anyone who dismisses the existence or seriousness of unfounded fears of childhood vaccines would be behaving foolishly. Skilled journalists and others have vividly documented enclaves of concerted resistance to universal immunization programs. Experienced practitioners furnish credible reports of higher numbers of parents seeking counsel and assurance of vaccine safety. And valid measures of vaccination coverage and childhood disease outbreaks confirm that the incidence of such outbreaks is higher in the enclaves in which vaccine coverage falls dangerously short of the high rates of vaccination prevailing at the national level (Atwell et al. 2013; Glanz et al. 2013; Omer et al. 2008).

At the same time, only someone insufficiently attuned to the insights and methods of the science of science communication would infer that this threat to public health warrants a large-scale, sweeping “education” or “marketing” campaign aimed at parents generally or at the public at large. The potentially negative consequences of such a campaign would not be limited to the waste of furnishing assurances of safety to large numbers of people who are in no need of it. High-profile, emphatic assurances of safety themselves tend to generate concern. A broad scale and indiscriminant campaign to communicate vaccine safety—particularly if understood to be motivated by a general decline in vaccination rates—could also furnish a cue that cooperation with universal immunizations programs is low, potentially undermining reciprocal motivations to contribute to the public good of herd immunity. Lastly, such a campaign would create an advocacy climate ripe for the introduction of cultural partisanship and recrimination of the sort known to disable citizens’ capacity to recognize valid decision-relevant science generally .....

The right response to dynamics productive of excess concern over risk is empirically informed risk communication strategiestailored to those specific dynamics. Relevant dynamics in this setting include not only those that motivate enclaves of resistance to universal immunization but also those that figure in the concerns of individual parents seeking counsel, as they ought to, from their families’ pediatricians. Risk communication strategies specifically responsive to those dynamics should be formulated—and they should be tested, both in the course of their development and in their administration, so that those engaged in carrying them out can be confident that they are taking steps that are likely to work and can calibrate their approach as they learn more (Sadaf et al. 2013; Opel et al. 2012).

Again, preliminary research of this sort has commended. Perfection of behavioral-prediction profiles of the sort featured in Opel et al. (2011a, 2011b, 2013b) would not only enable researchers to extend understanding of the sources and consequences of genuine vaccine hesitancy but also to test focused risk-communication strategies on appropriate message recipients.  If made sufficiently precise, screening protocols of this sort would also enable practitioners to accurately identify parents in need of counseling public health officials to identify regions where the extent of hesitancy warrants intervention.

The public health establishment should exercise leadership to make health professionals and other concerned individuals and groups appreciate the distinction between targeted strategies of this sort and the ad hoc forms of risk communication that were the focus of this study.  They should help such groups understand in addition that support for the former does not justify either encouragement or tolerance of the latter.

But anyway, you've got the data now.  Draw your own inferences -- from them, and not from the feral risk communication system that we rely on in this country to the detriment of our public health ....

References

Cultural Cognition Project Lab. Vaccine Risk Perceptions and Ad Hoc Risk Communication: An Emprical Analysis. CCP Risk Studies Report No. 17

Kahan, D. Why we are poles apart on climate change. Nature 488, 255 (2012).

Kahan, D., Braman, D., Cohen, G., Gastil, J. & Slovic, P. Who fears the HPV vaccine, who doesn’t, and why? An experimental study of the mechanisms of cultural cognition. Law Human Behav 34, 501-516 (2010).

Kahan, D.M. A risky science communication environment for vaccines. Science 342, 53-54 (2013).

Kahan, D.M. Climate-Science Communication and the Measurement Problem. Advances in Political Psychology 3 (2015).

Loewenstein, G.F., Weber, E.U., Hsee, C.K. & Welch, N. Risk as feelings. Psychological Bulletin 127, 267-287 (2001).

Slovic, P., Peters, E., Finucane, M.L. & MacGregor, D.G. Affect, Risk, and Decision Making. Health Psychology 24, S35-S40 (2005).

 

 

 

 

Thursday
Feb052015

What does Christie know that the rest of us don't? My guess is nothing (or even less that that actually)

Doing his part to increase the quality of the public debate over the public's non-debate over vaccine safety, NJ Gov'r  Chris Christie delivered a gift to empirically unencumbered pundits by indicating his support (momentarily, at least) for "parental choice" on childhood immunizations.  

News stories the next day were filled with authoritative-sounding insight on what Christie's stance tells us about partisan divisions on universal vaccination.  Most explained  that Christie was trying to "appease" the Republican Party's Tea Party wing.

But there is another possibility: Christie was just making a fool of himself.

Why do I say that?

Well, to start, there is tremendous public support for universal vaccination across all demographic, cultural, religious and other lines in the U.S.

Yes, there are some people who are "anti-vax."  They are outliers in all of the groups that play a recognizable role in contested political life in the U.S.

The media is filled with stories about "growing parental" resentment and "plummeting vaccination" rates.  But those stories are based on the failure of fact checkers to do what they are paid to do.

True, the blogosphere is filled with (contradictory and even comically self-contradictory accounts) of how being "anti-vax" proves this or that cultural or poltical group is "anti-science." The only "polls" those are based on are the ones that fact-free commentators perform on themselves, or possibly the people they meet when they go shopping at whole foods.

But if Christie said he thinks that there should "parental choice," surely that must mean that in fact some significant fraction of the Republican party is anti-vax, right?!  Likely it's those stupid, evil, anti-science Tea Party members!  He must be trying to curry favor with them!

Maybe.  

Or maybe he is just a bonehead.

To test these competing hypotheses, I decided to ... look at some actual evidence!

Based on data from the CCP Vaccine Risk Perceptions and Ad Hoc Risk Communication Report,
the figure illustrates the relationship between partisanship (measured with a scale that combines measures of liberal-conservative ideology and political-party identification) and the perceived benefits and risks of vaccines taking survey respondents' Tea Party membership into account.

Because only around 1% of the relatively liberal, Democratic respondents identified as "Tea Party members," I compared TP and non-TP members only among the relatively conservative, Republican members of the sample, of whom 30% identified as belonging to the TP (where does one sign up, btw? where are the meetings held? seriously!).

Well ... Eighty-four percent of the relative liberal, Democratic respondents and 78% of the conservative, Republican ones agreed that vaccine benefits outweigh their risks.  Boy, that's going to be a real wedge issue in the 2016 election, don't you think?

But wait-- the TP members!  A paltry 74% of them think vaccine benefits outweigh the risks! 

Look, in case you are tempted to try to squeeze a "culture conflict" out of the differences in how overwhelmingly pro-vaccine these groups of citizens are, here's graphic that shows the partisan breakdown on climate change, taking TP membership into account, in this same sample.

See the difference?

No? Well, then check out difference between what a polluted science communication environment and an unpolluted one looks like.

And if that doesn't work, then check out what happened when Glen Beck peed himself with excitement over the size of the difference in the science literacy scores of TP members (the same ones featured here) & the rest of the population.

So either 

(1) the brilliant and brilliantly advised Christie made a super shrewd move in appealing to those particular TP members who are out of step with 3/4 of their fellow TP members, and with the same proportion of non-TP conserv/repubs, thereby yanking the rug out from steam-rolling front-runner Jeb Bush, who is obviously being advised by rank amatures on this issue;

or

(2) Christie is a bonehead.

Call me a "contrarian" (someone did the other day for suggesting that it is actually useful to look at data when trying to interpret public opinion on science issues), but I pick bonehead.


Wednesday
Feb042015

What do polluted & nonpolluted science communication environments *look* like? & how about childhood vaccines?....

This is a common theme on this blog.  Maybe it's the only theme, in fact.

Risk perceptions that display the pathology of identity-protective cognition are pathological, both in being rare and in being inimical to public welfare.

We need to get that.

We  need to to learn how to distinguish risk perceptions that evince this pathology from ones that don't.

We need to use evidence-based methods to identify the dynamics that cause some risk issues, none of which is bound to become this way, to take on the characteristics of this pathology.

And we need to recognize our responsibility to use the knowlege we have of these dynamics to minimize the risk that any particular form of science will become infected with the pathology's reason-effacing characteristics.

Because if we don't do all these things, then we will experience many many avoidable instances of this pathology.  And for that, we'll have only ourselves to blame.

 


These data, btw, come from Climate-Science Communication and the Measurement Problem.

Tuesday
Feb032015

Science of Science Communication 2.0, Session 4.1: Trust in/of science 

I swear this was a huge coincidence but this week's session -- #4 -- in Science of Science Communication course ver 2.0 is ... trust in/of science!

The reading packet, distributed about about a week ago, had excerpts from 2009 Pew Report.

But for sure, live version of the class-- happening in few hrs -- will be focusing on 2015 version. Reason is less b/c of substance of 2015 Report (which is pretty much identical to 2009) than b/c of Pew's disappointing decision to hype the 2015 version in a manner that exploited popular misconceptions about declining status & authority of science in U.S. society.  Why that misconception exists and how various actors use it were among the major themes of the assigned readings!

But since I've now said so much about that, I'll just post some questions & open th floor to the "virtual class" participants to have their say.

But before I do, let me just note that next week the topic is -- (mis)communication of perceptions of vaccine-risk perceptions! We are nothing if not topical in this class!

  1. Do the NSF Indicators’ “public science attitude” items measure “trust” in science or of scientists?

  2. Is the Fiske/Dupree study consistent with the public opinion survey results reflected in the NSF Indicators?

  3. Is the Fiske/Dupree experiment internally valid? Externally?

  4. Did the media appropriately characterize the Fiske/Dupree sudy? If not, why not?

  5. Does Gauchat’s study support the inference that individuals who identify themselves as “conservative” are distrustful of science?   Is the study internally and externally valid?

  6. Did the media appropriately characterize the findings of the Gauchat study? If not, why not?

  7. Does the Jelen & Lockett study support any inferences about the relationship between religiosity and political partisanship, on the one hand, and trust in/of science? Is the study internally and externally valid?

  8. Does disbelief in evolution evince distrust in or of scientists?  Is acceptance of evolution a valid measure of “trust” in or of science? How are these last two questions related?

  9. What explains the perception that there is distrust in or of science, either in the public generally or among particular segments of it?  Does the Kahan study suggest an answer? Is that study valid, internally and externally?

  10. Are conflicts over who trusts or distrusts science something to worry about? Do they affect the quality of the science communication enviornment?

Saturday
Jan312015

Weekend update: Pew's disappointing use of invalid survey methods on GM food risk perceptions

So here’s a follow-up on “grading of Pew's public attitudes toward science report”--& why I awarded it a “C-” in promoting informed public discussion, notwithistanding its earning an “A” in scholarly content (the data separated from the Center’s commentary, particularly the press materials it issued).

This follow-up says a bit more about the unscholarly way Pew handled public opinion on GM food risks.

Some background points:

1. It’s really easy for people to form misimpressions about “public opinion.”

Why? Because, for one thing, figuring out what “people” (who actually usually can’t usefully be analyzed w/o breaking down into groups) “think” about anything is not anything anyone can directly observe; like lots of other complicated processes, it is something we have to try to draw inferences about on the basis of things that we can observe but that are only correlates or proxies of it.

For another, none of us is in the position via our personal, causal observations to collect a valid sample of the sorts of observable correlates or proxies.  We have very limited exposure, reflecting the partiality of our own social networks and experiences, to the ways in which “the public” reveals what it thinks.  And it is in fact a feature of human psychology to overgeneralize from imperfect samples like that & make mistakes as a result.

2. One of the things many many many many people are mistaken about as a result of these difficulties is “public opinion” on GM food risks.  The media is filled with accounts of how anxious people are about GM foods.  That’s just not so: people consume them like mad (70% to 80% of the food for sale in a US supermarket contains GMOs). 

Social science researchers know this & have been engaged in really interesting investigations to explain why this is so, since clearly things could be otherwise: there are environmental risks that irrationally scare the shit out of members of the US public generally (e.g., nuclear waste disposal). Moreover, European public opinion is politically polarized on GM foods, much the way the US is on, say, climate change.  So why not here (Peters et al. 2007; Finucane, M.L. & Holup 2005; Gaskell, Bauer, Durant & Allum 1999)? Fascinating puzzle!

That isn’t to say there isn’t controversy about GM foods in American society. There is: in some sectors of science; in politics, where efforts to regulate GM foods are advanced with persistence by interest groups (organic food companies, small farmers, entrepreneurial environmental groups) & opposed with massive investments by agribusiness; and in very specialized forms of public discourse, mainly on the internet.

Indeed, the misimpression that GM foods are a matter of general public concern exists mainly among people who inhabit these domains, & is fueled both by the vulnerability of those inside them to generalize inappropriately from their own limited experience and by the echo-chamber quality of these enclaves of thought.

3.  The point of empirical public opinion research is to correct the predictable mistakes that arise from dynamics like these.

One way empirical researchers have to tried to do this in the case of GM foods is by showing that in fact members of the public have no idea what GM foods are. 

They fail miserably if you measure their knowledge of GMOs

They also say all kinds of silly things about GM foods that clearly aren’t true: e.g., that they scrupulously avoid eating them and that they believe GM foods are already heavily regulated and subject to labeling requirements (e.g., Hallman et al. 2013).

That people are answering questions in a manner that doesn’t correspond to reality shows that the survey questions themselves are invalid. They are not measuring what people in the world think—b/c people in the world aren’t thinking anything at all about GM foods; they are just eating them. 

The only things the questions are measuring—the only thing they are modeling—is how people react to being asked questions they don’t understand. 

This was a major theme, in fact, of the National Academy of Science’s recent conference on science communication & GMOs.  So was the need to try to get this information across to the public, to correct t the pervasive misimpression that GM foods are in fact a source of public division in the U.S.

So what did Pew do?  It issued survey items that serious social science researchers know are invalid and promoted the results in exactly the way that fosters the misimpression those researchers are trying to correct!

Pew asked members of their general public sample, “Do you think it is generally safe or unsafe to eat genetically modified foods?” 

Thirty-seven percent answered “generally safe,” 57% “generally UNsafe” and 6% “don’t know/Refused.”

Eighty-eight percent of the "scientist" (AAAS member) sample, in contrast, answered "generally safe."

Pew trumpted this 51% difference, making it the major attention-grabber in their media promotional materials and Report Commentary.

This is simply pathetic. 

As an elite scholarly research operation, Pew knows that this survey item did not measure any sort of opinion that exists in the U.S. public. Pew researchers know that members of the public don't know anything about GM foods.  They know the behavior of members of the public in purchasing and consuming tons of food containing GM foods proves their is no meaningful level of concern about the risks of GM foods! 

Indeed, Pew had to know that the responses to their own survey reflected simple confusion on the part of their survey respondents.

Pew couldn't possibly have failed to recognize that because (as eagle-eye blog reader @MW pointed out) another question Pew posed to the respondents was whether “When you are food shopping, how often, if ever, do you LOOK TO SEE if the products are genetically modified?” 

Fifty-percent answered “always or sometimes.” 

This is patently ridiculous, of course, since there is nothing to see in the labels of foods in US grocery stores that indicates whether they contain GMOs. 

This is the sort of question—like the ones that show that the US public believes that there already is GM food labeling in the US, and is generally satisfied with “existing” information on them (Hallman et al. 2013)—that researchers use to show that survey items on GM food risks are not valid: these items are eliciting confusion from people who have no idea what they are being asked.

And here’s another thing: immediately before asking these two questions, Pew used an introductory prompt that stated “Scientists can change the genes in some food crops and farm animals to make them grow faster or bigger and be more resistant to bugs, weeds, and disease.”

That’s a statement that it is quite reasonable to imagine will generate a sense of fear or anxiety in survey takers.  So no surprise that they when one then asks them, “Oh are you worried about this,” and “do you (wisely, of course) check to see if this weird scary thing has been done to your food?!” people answer “oh, yes!”

Even more disturbing, the questeion immediately before that was whether people are worried about pesticides – a topic that will predictably raise risk apprehension level generally and bias upward respondents' perceptions of other putative risk sources in subsequent questions (e.g., Han, Lerner & Keltner 2007).

Sigh.

Bad pollsters use invalid questions on matters of public policy all the time.

They ask members of the American public whether they “support” or “oppose” this or that policy or law that it is clear most Americans have never heard of.  They then report the responses in a manner that implies that the public actually has a view on these things.

Half the respondents in a general population survey won't know-- or even have good enough luck to guess-- the answer to the multiple-choice question "how long is the term of a U.S. Senator?" Only 1/3 of them can name their congressional Representative, and only 1/4 of whom can name both of their Senators.

Are we really supposed to take seriously, then, a poll that tells us 55% of them have an opinion on the “NSA’s telephonic metadata collection policy”?!

Good social science researchers are highly critical of this sort of sensationalist misrepresentation of what is really going on in public discourse (Krosnick, Malhorta & Mittal 2014; Bishop 2005; Shuman 1998).

Pew has been appropriately critical of the use of invalid survey items in the past too, particularly when the practice is resorted by policy advocates, who routinely construct survey items to create the opinion that there is “majority support” for issues people have never heard of (Kohut 2010). 

So why, then, would Pew engage in what certainly looks like exactly this sort of practice here?

* * *

A last point.

Some very sensible correspondents on Twitter (a dreadful forum for meaningful conversation) wondered whether an item like Pew’s, while admittedly invalid as a measure of what members of the public are actually thinking now, might be a good sort of “simulation” of how they might respond if they learned more.

That’s a reasonable question, for sure.

But I think the answer is no

If a major segment of the US public were to become aware of GM foods—what they are, what the evidence is on their risks and benefits—the conditions in which they did so would be rich with informational cues and influences (the identity, e.g., of the messengers, what their peers are saying etc) of the sort that we know have a huge impact on formation of risk perceptions.

It’s just silly to think that the experience of getting telephone call from a faceless pollster asking strange questions about matters on has never considered before can be treated as giving us insight into the reactions such conditions would be likely to produce.

We could try to experimentally simulate what those conditions might be like; indeed, we could try to simulate alternative versions of them, and try to anticipate what effect they might have on opinion formation.

But the idea that the experience of a respondent in a simple opinion survey like Pew’s is a valid model of that process is absurd.  Indeed, that’s one of the things that experimental simulations of how people react to new technologies have shown us.

It's also what real-world experience teaches: just ask the interest groups who sponsored defeated referenda in states they targeted after polls showed 80% support for labeling.

But in any case, if that’s what Pew thought it was doing—simulating how people would think about GM food risks if they were to start thinking of them—they should have said so.  Then people would not have formed a misimpression about what the question was measuring.

Instead, Pew said only that they had done a survey that documents a “gap” between what members of the public think about GM food risks and scientists do.

Their survey items on GM food risks do no such thing.

And that they would claim otherwise, and reinforce rather than correct public misimpressions, is hugely disappointing.

Refs

Bishop, G.F. The illusion of public opinion : fact and artifact in American public opinion polls (Rowman & Littlefield, Lanham, MD, 2005).

Finucane, M.L. & Holup, J.L. Psychosocial and cultural factors affecting the perceived risk of genetically modified food: an overview of the literature. Soc Sci Med 60, 1603-1612 (2005). 

Gaskell, G., Bauer, M.W., Durant, J. & Allum, N.C. Worlds apart? The reception of genetically modified foods in Europe and the US. Science 285, 384-387 (1999).

Hallman, W., Cuite, C. & Morin, X. Public Perceptions of Labeling Genetically Modified Foods. Rutgers School of Environ. Sci. Working Paper 2013-2001, available at http://humeco.rutgers.edu/documents_PDF/news/GMlabelingperceptions.pdf.

Han, S., Lerner, J.S. & Keltner, D. Feelings and Consumer Decision Making: The Appraisal-Tendency Framework. J Consum Psychol 17, 158-168 (2007).

Kohut, A. Views on climate change: What the polls show. N.Y. Times A22 (June 13, 2010), available at http://www.nytimes.com/2010/06/14/opinion/l14climate.html?_r=0

Lerner, J.S., Han, S. & Keltner, D. Feelings and Consumer Decision Making: Extending the Appraisal-Tendency Framework. J Consum Psychol 17, 181-187 (2007).

Krosnick, J.A., Malhotra, N. & Mittal, U. Public Misunderstanding of Political Facts: How Question Wording Affected Estimates of Partisan Differences in Birtherism. Public Opin Quart 78, 147-165 (2014).

Peters, H.P., Lang, J.T., Sawicka, M. & Hallman, W.K. Culture and technological innovation: Impact of institutional trust and appreciation of nature on attitudes towards food biotechnology in the USA and Germany. Int J Public Opin R 19, 191-220 (2007).

Shuman, H. Interpreting the Poll Results Better. Public Perspective 1, 87-88 (1998).

 

Friday
Jan302015

Grading the 2015 version of Pew study of public attitudes toward science

So everybody knows that the Pew Research Center released a cool study yesterday on public attitudes toward science & on differences between public & scientists (or at least AAAS members; it’s worth noting that AAAS membership isn't limited to scientists per se).

It was a follow up to Pew’s classic 2009 study of the same -- & it makes just as huge and valuable a contribution to scholarly understanding as that one, in my view.

Lots of people have said lots of things already & will say even more. But here are a few thoughts:

1. Pew does great work in measuring US public attitudes toward science & scientists.  They ask questions that it is sensible to believe measure general public regard for the enterprise of science, and keep track over time.

When one adds their findings to those collected by the National Science Foundation and the National Opinion Research Center at the University of Chicago, which conducts General Social Survey, source of most of the NSF's annual "Science Indicator" measures, one can really form a good view of how the US public feels about science.  

People should ignore all the bogus studies that administer strange questions to M turk workers -- there are tons of those & they always report really weird, sensational findings.  

Who needs data? It's obvious!2.  This report, like the 2009 one, shows that Americans basically love science.  By overwhelming margins, they report admiration for scientists and positive appraisals of what scientists do.  This is consistent with what the NSF Science Indicators, which are released every year, show too.

3.  Still, there is almost this weird reluctance in the Center's press release and commentary to accept or clearly articulate this conclusion!

It's common wisdom that public disputes over science stem from a “creeping anti-science” sensibility in American society.

Scholars who actually study public attitudes toward science, however, know that that view is unsupported by any convincing, valid data.  Indeed, the Pew and NSF Indicator reports show that there is overwhelming trust-- across all demographic, political, and other types of cultural groups (religious & nonreligious, e.g.).  

The 2009 Report helped to try to correct the “common wisdom” in this regard.

But the 2015 Report seems committed to avoiding any confrontation with this view. Instead, by employing a strategy of silence, inapt juxtaposition, and emphasis of irrelevant data, the Center commentary seems committed to consoling those who hold this fundamentally mistaken understanding of the sources of public conflict over science.

mmmmmmm... science ....It's almost as if Pew feels disappointed to pop the balloon of self-reinforcing popular misunderstanding on this issue with the needle of its own data.

4. Consider the "gap" between scientists & public on evolution.  

Yes, it's there.

But it is well established that public opinion responses to the question “do you believe in human evolution” have zero connection to what people know about either evolutionary science or science in general. 

It's also perfectly clear that this "gap" in public and scientific understandings has nothing to do with public respect for scientists.  

The 2009 Pew Report made that clear, actually, reporting data showing that those who said they "disbelieved in" evolution as well as those who said they "did" both had highly positive views of science's contribution to society.

The Report and Alan Leshner’s commentary for the 2009 Report both emphasized that there was no meaningful differences in that regard between people who said science sometimes conflicts w/ their religious views & those who said it doesn't.

Nothing at all has changed--nothing. But is there anything comparable in this yr's report? Nope!

Leshner himself did write a very thoughtful commentary in Science. 

He's still championing respect for and respectful dialogue with diverse memers of the public: good for him; he's a real science-of-science-communication honey badger!

But even he seemed to think that getting his message across required indulging the "creeping anti-science" meme, warning that "the public's perceptions of scientific expertise and trustworthiness" risk being "compromised whenever information confronts people's personal, personal, or religious views"-- conclusions that actually seem completely contrary to the data presented in both 2009 and 2015.

5. Same w/ the “gap” on climate change.  

It's clear that climate change opinions don't measure either science comprehension, knowledge of climate science in particular, or respect for or attitudes toward science.  

In 2009, Pew wanted people to see, too, that public conflict over climate change did not originate in any disagreement about the value of science or trustworthiness of scientists. It emphasized that both climate-change "believing" & "disblieving" members of the public had the same positive views in this regard.

Not so in the 2015 report, where the "gap" on climate change is repeatedly used to qualify the finding that the public has high regard for science. (Interesting that only 87% of AAAS members indicated they "believe in" AGW; I'm sure they understand the evidence for AGW and even use the evidence "at work.")

What makes this all the more strange is that the 2015 Report recognizes that the public's disagreement over AGW mirrors a public disagreement over what scientific consensus is on this issue (a phenomenon that can be attributed to ideologically biased assessments of evidence on both issues).  

In other words accepters and nonacepters alike believe "science is on their side" -- much the way that nations at war believe that God is....

For sure, the debate is alarming and contrary to enlightened democratic decisionmaking.  

But if anyone thinks the source of the debate is lack of science comprehension on the part of the public or lack of public confidence and trust in science, they are themselves ignoring all the best evidence we have on these issues!

Pew's job is to help remedy this widespread form of science-of-science-communication illiteracy.

6. The data reported on public attitudes on GM foods is super disappointing.  

Social scientists know that surveying the public on issues that it has never heard of generates absolutely meaningless results.

GM food risks are in that category. 

Consider:

American consumers’ knowledge and awareness of GM foods are low. More than half (54%) say they know very little or nothing at all about genetically modified foods, and one in four (25%) say they have never heard of them.

Before introducing the idea of GM foods, the survey participants were asked simply ”What information would you like to see on food labels that is not already on there?” In response, most said that no additional information was needed on food labels. Only 7% of respondents raised GM food labeling on their own. . . .

Only about a quarter (26%) of Americans realize that current regulations do not require GM products to be labeled.

Hallman, W., Cuite, C. & Morin, X. Public Perceptions of Labeling Genetically Modified Foods. Rutgers School of Environ. Sci. Working Paper 2013-2001. 

Americans don't fear GM foods; they eat them.

No meaningful inferences whatsoever can be drawn from the "gap" in attitudes between members of public & scientists on this issue.  

Very un-Pew-like to play to common misunderstandings about this by treating the “gap” between public and scientists on GM as supporting any meaningful inferences about anything.

6.  Also very out of character is Pew's calling attention to minute changes in the overwhelming levels of support for science reflected in particular items:

I'm sure they were just trying to throw a bone to all those who "just know" -- b/c it's so obvious-- that we are living in the "age of denial." But if the latter seize on these changes as meaningful, they'll only be making fools of themselves. 

For perspective, here are comparable data, collected over time, from NSF Indicators (click on them for larger displays).  


That anyone can see in these sort of data evidence for a "creeping anti-science" sensibility in the general US population or any segment of it is astonishing -- something that itself merits investigation by public opinion researchers, like the excellent ones who work for Pew!

But the bottom line is, the job of those researchers isn't to feed these sorts of persistent public misimpressions; it's to correct them!

* * *

How would I grade the Pew Study, then?

“A” for scholarly content.”

“C -” for contribution to informed public understanding.

Overall: “B.”

Thursday
Jan292015

Science of Science Communication 2.0, Session 3.1: Science comprehension: who knows what about what—and how?

Okay, this is "session 3" post for virtual "Science of Science Communication 2.0."  The real space version got snowed out!  

But the discussion I was hoping for (based on these readings) was mainly one on how group affinities contribute to transmission of scientific knowledge.  

"Cultural cognition" is normally associated with idea that such affinities distort such transmission.  

But my hunch is that cultural cognition is in fact integral to citizens' reliable apprehension of what is known to science; that the sorts of pathologies that we see in which people with different cultural identities use their reason to form and persist in opposing views on risks and related facts is a consequence of a polluted science communication environment that disables the normally reliable reasoning strategies people use (including observation of what others who know what's what about what are doing and saying) to figure out what is known...

But I admit to being uncertain about this!  Indeed, I readily admit to being uncertain about everything, including the things I am most confident that I think I understand; certainly I am committed (I hope able) to revise anything I blelieve on the basis of any valid evidence I encounter.

But here I am not even as confident as I'd like to be about the state of the evidence on my conjecture -- that cultural cognition is not a bias, but is integral to the normal process by which diverse people usually converge on the best evidence.  And so I was & remain eager for reflections by others!

Below are the questions I posed to motivate student reading & orient discussion for this session.  Next session, in which we'll be doing "double time" to make up for lost class, will feature trust in/of science...

  1. What is the relationship between the sort of critical reasoning proficiency featured by Baron’s “actively open-minded thinking” and Dewey’s understanding of “scientific thinking”?
  2.  

  3. Is critical reasoning proficiency essential for science comprehension on the part of a non-scientist, either in her capacity as personal decisionmaker, member of civil society, or democratic citizen?
  4.  

  5. Are conflicts over policy-relevant science plausibly attributable to deficits in critical reasoning proficiency?
  6.  

  7. Does the effective use of scientific knowledge by non-scientists—in the various capacities in which their decisions should (by their own lights) be informed by it—depend on their being able to comprehend what it is that science knows?
  8.  

  9. Is it possible for citizens to reliably recognize who knows what is known by science without being able to comprehend what it is those individuals know?  If so, how? Does their ability to do that depend on their possessing the sort of reasoning proficiency emphasized by either Baron or Dewey? If not, what does it depend on?
  10.  

  11. How does Popper’s understanding of the transmission of scientific knowledge relate to Miller’s, Dewey’s, and Baron’s?
  12.  

  13. Do group affinities—ones founded on common outlooks and values—promote transmission of scientific knowledge or inhibit it?  In either case, how?
Wednesday
Jan282015

There is pervasive cultural consensus on the value of childhood vaccines in the U.S.; so why do people *think* that being anti-vaccine reflects any particular cultural predisposition?

A reporter who was covering the current measles outbreak asked me a question about the connection between vaccine hesitancy—the reluctance of parents to get their kids vaccinated—and the contribution cultural or political predispositions make to vaccine risk perceptions. 

In the background of the question were a couple of facts that this reporter gets but that a lot other reporters and people generally don’t: first, that parents in the US along with rest of the general public in this country are overwhelmingly pro-vaccine; and second, that the people who belong to the small segment of the population that is anti-vaccine are big time outliers in all the social groups—cultural, political, religious and so forth—that make up our basic inventory of “who” people are.

Someone didn't get the memo ... click to see whoOn the first point, briefly: Despite the media din to the contrary, the US has enjoyed impressively high childhood  vaccine rates—over 90%, the public health target, for all the recommended universal vaccinations, including MMR—for going on 15 yrs.  The percentage of parents not getting their kids vaccinated has remained below 1% that entire time.

Fortunately, the Wakefield affair, which did have a significant impact on vaccine behavior in the UK (maybe other countries, too, but the truth is, many European countries have lower vaccine rates than they should have had for a long time), didn’t have a comparable effect in the US.

Vac rate trends? Ask the CDC...On the second: As documented in various places including the CCP  Vaccine Risk Perceptions and Ad Hoc Risk Communication study, there is no meaningful correlation between vaccine risk perceptions and the sorts of characteristics that usually indicate membership in one or another cultural group.  

The correlation between such risk perceptions and political outlooks, e.g., is close to nil.

What's the correlation between vaccine risk perceptions and political outlooks? Click & see for self

Likewise, contrary to the empirically uninformed, illiberal, counterproductive "anti-science trope," the cultural groups whose members are divided on climate change and evolution are in fact in overwhelming agreement that vaccine benefits outweigh their risks. 

Still, the reporter wanted to know, given all this, how come it appears to him and others that there is a correlation between anti-vaccine, or concerned about vaccine risks, and a cultural style that is, I guess, left-leaning in politics, anti-industry or –capitalist, highly “naturalist” etc.

What's the relationship between non-acceptance of global warming or human evolution and belief that vaccines are unsafe? Click for answerVery reasonable question.

What’s more, I don’t have an answer that I’m particularly confident in! 

I do have some conjectures, and so I thought I’d share them here & ask others what they thought.

Also, at about the time I was writing this email, Chris Mooney was addressing this same question in thoughtful essay that I encourage others to read—I think my views are pretty close to his!

My thoughts on this question:

I'm really not sure what to say, but my hunch is that there is a huge sampling bias risk here when we try to draw on own experience to figure out vaccine risk perceptions.  

It's clearly the case -- just no arguing w/ it, really! -- that the vast majority of people in US, including parents, are not hostile to but in fact very favorable disposed to childhood vaccinations.  This is true across all the sorts of cultural groups that normally come to mind when we think of risk issues like climate change etc. where there really are very deep & strong cultural divisions.

Yet some people are clearly anti-vaccine. If we see them, what are we to make of them?  

It's hardly a surprise that they will have integrated their views into their cultural understandings generally.  That is, there will be coherence, for them, in their positions on vaccines and their ones on various other issues.  

So if they happen, say, to be the kind of person who has an egalitarian, collectivist style & is anxious about environmental issues & suspicious of corporations and the like, then their positions on vaccines will likely be of a piece with that.

But then if we were to say to ourselves, -- "a ha! Being anti-vaccine coheres with being that sort of person!," we'd be making a mistake.  At a minimum, we'd be making a mistake b/c we'd be neglecting to consider all the people who share that person's cultural style -- and indeed hold the standard collection of risk perceptions that go along with it -- but who don't have anxieties about vaccines!  Those people would outnumber the anti-vaccine mom or moms we are talking about -- by orders of magnitude.

We'd also be at risk of making another mistake.  

That particular anti-vaccine group of moms you ran into -- they might not even be representative of all the other anti-vaccine folks.  Indeed, if you met them at whole foods (I have no idea if this applies to you, but you'll get the idea), then likely your sense of what the anti-vaccine people are like is undercounting all the anti-vaccine people who don't shop there.  They don't shop there b/c doing so would be contrary to their cultural style.  They might be very conservative -- maybe they are religious fundamentalists of one sort or another.  Those might be people you never happen to run into!  As a result, the people who are like that who are anti-vaccine will be missing from your mental census.

Of course, so will all the people "like that" who are not anti-vaccine.  They will, just as in the case of the moms at whole foods, outnumber the anti-vaccine members of their groups by orders of magnitude.  But possibly b/c they are more likely to encounter anti-vaccine types in the community in which they interact w/ people most of the time, they might also have a misimpression that anti-vaccine people actually are more likely to hold values like theirs!

Now one more really really important thing:  I actually am pretty convinced that most of the parents whose kids miss vaccinations are not part of any movement.  

The "movement" is there, but it is small & gets way more attention than it deserves precisely b/c it is loud, in people's faces, and really good at provoking hysterical denunciations of them.  

But more importantly, there's every reason to believe that most of the parents whose kids are not getting vaccinated don't belong to any movement at all.  These are parents who are just nervous, not agitated.

They aren't loud and obnoxious.

They aren't demonstrating or getting in anyone's face.  

 They are not wearing buttons saying "vaccines give kids autism! McCarthy for President   in '68 '16!"

And for that reason it's actually hard to find them--which is very unfortunate b/c almost certainly they could be reassured by a good public health professional trained to give them sensible, evidence-informed risk counseling.

The best work being done on vaccine hesitancy is the research to develop a screening instrument for new parents to identify which ones are likely to end up w/ kids who miss vaccination.  With that sort of instrument in hand, ongoing empirical research to develop an effective risk counseling protocol targeted at these very parents could be carried out much more effectively too.

Rather than propagating the misimpression that a “growing crisis of public confidence” among “a large and growing number” of “otherwise mainstream parents” has generated an “erosion in immunization rates”  in the U.S. (these are in fact demonstrably false claims), those who truly want to make universal vaccination in this country even more effective should be calling for more resources for the scientists doing this excellent and vital research.

It's fine to criticize the small cadre of attention seekers who are spreading misinformation about vaccines. They are a bunch of idiots & a menace etc.  

But engaging in relentless, self-important, attention-grabbing displays of denunciation in return is itself dangerous for the vaccine science communication envirionment, and distracts us from what's really needed: the development of valid, reliable methods that practioners can use to identify the much larger group of parents who are merely anxious and supply them with tailored risk counseling that would give them the same sense of relief & happiness that everyone else gets from knowing that their kids won't get the horrible disease that were our grandparents’, great grandparents’, great-great grandparents’ et al’s biggest nightmare.

Saturday
Jan242015

Weekend update: Are GM foods toxic for the science communication environment? Vice versa? (video lecture & slides)

The National Academy of Sciences Public Interfaces of the Life Sciences program has posted videos from its recent Public Engagement with Gentically Modified Organisms symposium.  It was a really great event with lots of interesting presentation and even more fascinating discussion and debate among the participants.

I gave a presentation on the first day: "Are GM foods toxic for the science communication environment? Vice versa?" (slides here).  Also was on a panel the last day with science communication genius Rick Borchelt,  director of the Office for Communications and Public Affairs Department of Energy; and with Helene Dillard, from UC Davis, and Molly Jahn, from the University of Wisconsin-Madison, both of whom offered harrowing insights into the contentious atmosphere surrounding academic research on GMOs and the difficulties that poses for scientists who want to participate in public deliberations on the subject.

Strongly recommend you watch all the videos if you didn't get a chance to attend or watch live (or even if you did and the alternative is to, say, grade fall semester exams!)

Event highlight: Scheufele demonstrates "performative risk communication" by eating bowl of Frankenberry (he did get sick, but only b/c the cereal had been sitting around since 1973)

Friday
Jan232015

Should we care about the public's *climate science* literacy? What is "ordinary climate science intelligence" *for*?

A friend of mine posed this very appropriate, very basic,  very important question/challenge to me after reading the Measurement Problem:

One thing i was left wondering after reading the paper was "should we care about people's OCSI ["Ordinary Climate Science Intelligence"] scores"? does "climate science comprehension" matter for anything meaningful? should we invest (any) resources in trying to increase people's science comprehension in this arena? and if so, WHY? does it, for example, correlate with people's ability to meaningfully/productively engage in the sorts of collective decision-making and planning processes happening in southeast florida? or do you simply see it as a valid end, not a means to anything on the "confronting climate change" front? i'm not sure what your take is on this, exactly, as it seems at some points that you do advocate communicators learn how to do exactly this (improve comprehension), perhaps by looking to science teachers who have figured out effective strategies in other contexts (evolution); but at other points (i think) you say that climate sci comprehension has nothing/little to do with the cultural polarization that seems to inhibit any large-scale collective response to the issue (which, as you say, should come as little surprise). or perhaps both of these statements are true to your thinking, but then it's less clear why (for what purpose) you advocate the former (coming back to the question of, "what is climate science comprehension FOR in a culturally polarized world?"). 

My answer:

OSCI items & responsive curves-- click it! On Ordinary Climate Science Intelligence (OCSI) assessment scores: I think the scores are useful for exploring questions and testing hypotheses about why there is public conflict over climate change.

E.g., is the source of public conflict on this issue attributable to differences in comprehension of either the rudimentary mechanisms of this climage change? 

Or perhaps to their “unfamiliarity” with the evidence that climate scientists have compiled on the causes and consequences of it?

OCSI can help to answer those questions: because it shows that members of groups polarized over the existence of climate change and the contribution humans are making to it have comparable understandings (and misunderstandings) about those matters, it gives us less reason to credit those explanations than we’d have otherwise.

Indeed, OCSI helps us to see, too, that survey items that assess public “acceptance” of human-caused climate change simply aren’t measuring anything having to do with knowledge of climate science at all.  They have no correlation whatsoever to scores on a rudimentary climate literacy test and instead cohere with—behave exactly like other observable indicators of—their cultural identities.

As I explain in The Measurement Problem, OCSI was meant essentially to be a model of an assessment test, one designed to examine whether it is possible, with appropriately worded items, to disentangle (unconfound) cultural identity & knowledge when measuring how much people know about climate change.

But if we want to measure people's understandings of climate science, then I am sure it would be possible to do better than OCSI!

Your question, I take it, is actually more basic: given what we can see from OCSI, why should even want to measure how much people understand about climate science? Why should we care?

There's no answer, of course, that doesn’t presuppose some sort of normative goal.

The goal of most people who collecte data on the issues we are interested in is simply to "move the needle" of public opinion on "belief in" human-caused climate change.  For them, I guess the results of the paper suggest that they should "not care" whether anyone comprehends anything meaningful about climate change. Because what they “believe” about human-caused climate change turns out not to have anything to do with what they know.

Indeed, the paper shows that, if the goal is simply the instrumental one of generating public engagement with the issue of climate change, advocates should stop obssessively measuring and minutely analyzing the percentage of the public who say they believe in (accept) human-caused climate change

Again, what people say about that is measures only of their cultural identities—who they are.  Their response to “do you believe in human-caused climate change” questions not only don’t measure what they know.  They don’t even measure whether they are worried and concerned about climate change!

The consistently wrong answers most believes & skeptics give about the extent and nature of the dangers posed by climate change (e.g., that it will cause increases in skin cancer, or prevent plant photosynthesis) strongly suggest that believers and skeptics alike (in the general public at least) are very alarmed, as an emotional or affective matter, about the risks human-caused climate change poses.

What those who are trying to mobilize public opinion in this way should be trying to figure out is why their style of advocacy doesn’t tap into this reservoir of concern but instead reliably, predictably, inevitably triggers the identity-protective response that is reflected in the “No, I don’t, you asshole!” answer that 1/2 the US public gives when asked (over & over & over in polls that aren't advancing anyone's understanding of anything at this point) “do you believe in human-caused climate change?” 

But there are bunch of other goals one could have—I’d say, should have—besides the navel-gazing one of “needle moving.” All of them support developing an even better instrument for assessing what ordinary  people know about the science of climate change.

One is to help ordinary members of the public recognize information important to the decisions that they will make as citizens in self-governing communities the welfare of which will be affected by actions they take in relation to a changing climate.

Another would be to create communication materials that make it possible for the relatively small portion of the population who is genuinely curious about what we know about a changing climate to satisfy that interest.

Another would be to educate young people who might, if they are taught well and made excited by what they learn, become either climate scientists or adults who are genuinely curious about what we know and who, in any event, will become people who need to make decisions informed by the best evidence in their own private lives (as, say, property owners or business people; or farmers) or as citizens whose communities will be affected by climate change.

For all of those goals and related ones, then there will be value in having not an "OCSI" but a variety of them suited for the goal at hand.  

As I said in the paper, e.g., I think it is silly to measure whether citizens  know that the North Pole ice cap melting won't cause flooding; it's enough for them to know that melting ice sheets are creating a risk for people as a result of climate change.

They know that!

But if one's goal is to educate young people, "North Pole" might be a pretty good item -- which is to say, it might actually be contributing to measurement of the latent comprehension capacity that educators should be trying to instill.  So there are values in having OCSIs for these purposes that are actually tied to the sort of knowledge and comprehension capacities that it makes sense for those transmitting scientific information to focus on in the context in which they are operating.

The reason it would be nice to have on OCSI for the "curious consumer of science," too, is so that those who are part of the (truly amazing!) profession that is committed to serving him or her can figure out whether their efforts are working as well as they want.  

For all of these actors, there will be domain-specific “OCSIs” better than the experimental one featured in The Measurement Problem.  But I hope that the experimental OCSI can help those developing these practical, real-world OCSIs to see that they can and must “disentangle” identity and knowledge in constructing them. 

Well, those are my thoughts.  What do you think?

Wednesday
Jan212015

Science of Science Communication 2.0, Session 2.1: What is science literacy? And what is it for?

So yesterday was session 2 of the "real space" version of Science Science Communication course ver 2.0.  The topic was public "science literacy" or "comprehension" & whether the NSF Indicators and other standardized assessments are "measuring what counts."  The list of assigned readings are here.

Rather than summarize or sound off (again; I've done 738 posts on this topic since topic since this blog started in 1973), I thought I'd just post some questions & let you-- the 14 billion students who enrolled students in the "virtual" version of course (most of whom registered via Tamar Wilner's site) -- say what you think (& of course, ask and answer different questions if you like).

  1. How important is general scientific knowledge for a general member of the public? Does he or she have to understand particular bodies of science or be able to comprehend scientific evidence to be able reliably to identify and use of scientific knowledge in his or her personal life? In his or her role as a democratic citizen?

  2. Is the NSF science indicators battery a valid assessment of science literacy? What is the battery measuring exactly? And how reliably?

  3. What does Miller’s “civic science literacy” (CSL) measure? Are the elements of knowledge or the dispositions it measures essential for individuals to be able to recognize and use valid scientific knowledge in their lives?  What sort of evidence is there on that question?

  4. Is administering a public “science literacy” test to scientists a useful way to validate such a test?

  5. How does Miller’s CSL  relate to Dewey’s position that “scientific method” just “is thinking”? What might a measure of scientific literacy—or however one might characterize it—look like?  

 

 

Sunday
Jan182015

What does & doesn't count as valid evidence of "ideologically motivated reasoning" & "symmetry" of the same

A friend wrote to me posing a question, implicit in which are a bunch of issues about what counts as valid evidence of motivated reasoning & the symmetry of it across the left-right "spectrum" -- or cultural worldview spectra.  

I've addressed many of these issues, most more than once, in various posts.

 Indeed, I have on many occasions noted that almost everything I say is something I've said before, including that I've already noted on many occasions that everything I have to say is something I seem already to have said before. I think this is just sort of normal actually, when one is engaged in a sort "rolling conversation" w/ a fuzzy set of participants whose aim is reciprocal exchange of ideas for mutual enlightenment, & I should just stop talking about this.

That's the first time I've said that, but I'm sure not the last....  

But in any case, I thought I'd share this particular exchange. Maybe it will be clearer or more accessible than some of the others or simply increase the likelihood that someone who can get value from my views (very possibly by being able to see more clearly why he or she thinks I've made an error!) will find & get value out of these reflections on the nature of what sorts of study designs support inferences on "ideologically motivated reasoning" and asymmetry.

My friend:

I want to say that your research has found that more numerate people are more biased on both ends of the political spectrum, but my recollection is that what you find is actually that more numerate people do not believe more in the reality of climate change.  My question is: Have you looked at the interaction – e.g., done a median split on numeracy and then compared the polarization graphs between the numerate and innumerate?

Me:

I don't think what you've asked me to show you can support the inference that any particular form of reasoning proficiency ("science literacy," "numeracy," "cognitive reflection" etc.)  magnifies ideologically motivated reasoning "symmetrically" (let's say) with respect to ideology. 

But I'll show you what you asked to see first, before explaining why.

A. "Looking at" the magnification of polarization conditional on science comprehension

There are a variety of ways to graphically display what you are asking for--a view, essentially, of the differential in polarization at different levels of reasoning proficiency.

I think what I've done below -- splitting the "ideology" predictor & doing a lowess for each 1/2 of the sample separately in relationship to science comprehension -- is the best, or better in any case than splitting both continuous measures and giving you two pairs of %'s (for left-leaning & right-leaning at "below" & "above" avg); this way you get the information benefit of the continuous science-comprehension measure.

 

These are the same data from your slide 7.

With the lowess, one can see pretty readily that the gap between "left-" & "right-leaning" respondents gets progressively larger from -1 SD (16th percentile) to +1 SD (84th) on OSI_2.0 & then pretty much levels out.

(As you know, "OSI_2.0" is a 1-dimensional science-comprehension measure that consists of "science literacy," numeracy & cognitive reflection items. It was formed using a 2PL Item response theory model. For details, see 'Ordinary Science Intelligence': A Science Comprehension Measure for Use in the Study of Science Communication, with Notes on 'Belief in' Evolution and Climate Change.). 

B. But what are we looking at? 

So if one is trying to get the practical point across, this justifies saying things like, "polarization tends to increase, not abate, as individual with opposing political or cultural outlooks become more proficient in making sense of scientific information" etc.

But one can't on this basis infer that motivated reasoning is being magnified conditional on reasoning proficiency-- or as you put it, that "more numerate people are more biased on both ends of the political spectrum."

The question-- is human-caused climate change occurring?-- is a factual one that presumably has a correct answer. Thus, one "side" -- "liberals" or "conservatives" -- is presumably becoming more likely to get the correct answer as reasoning proficiency increases.

It is thus possible that motivated reasoning is increasing conditional on reasoning proficiency for the side that is becoming more likely to get the "wrong answer" but dissipating conditional on reasoning proficiency for the side that is becoming more likely to get the right answer!

That inference isn't logically compelled, of course.  If one is predisposed to believe something that happens to be true, then motivated reasoning will increase the likelihood of "getting the right answer."

But in that case, your getting the right answer won't prove you are smart; it will show only that you were lucky, at least on that particular issue.

The point, though, is that the evidence we are looking at above is equally consistent with the inference that motivated reasoning is being magnified by enhanced reasoning proficiency and the inference that ideologically motivated reasoning is "asymmetric" with respect to ideology. 

C. Observing what we really are trying to figure out

There is, I think, only one way to determine whether greater polarization conditional on greater reasoning proficiency is being caused by an ideologically symmetric (more or less) magnification of motivated reasoning: by looking at how people reason independently of whether they are getting the right answer.

What we need to see is how biased or unbiased the reasoning of those on both "sides" is as each side's members display greater reasoning proficiency.

I'll show you results from two studies that bear on this.

1. Motivated system 2 reasoning 

In the first (Kahan 2013), subjects evaluated "evidence" of the validity of the cognitive reflection test as a measure of "reflective & open-minded" reasoning. The experimental manipulation was the representation that those who score higher are more likely or instead less likely to accept evidence of human-caused global warming.

One might like to use a valid test of reflective reasoning (particularly an objective, performance based one like CRT, say, as opposed to the self-reporting ones like "need for cognition" that a fair number of researchers persist in using despite their dubious validity) to test the oft-asserted claim that "right-leaning" individuals are more "dogmatic" and "closed minded" etc. than "left-leaning" ones.

But if one is moved to selectively credit or discredit evidence of the validity of an open-mindedness test based on whether it supports the conclusion that those who share one's own ideology are "more open-minded and reflective" than are one's ideological opposites, one's own ideologically motivated reasoning will prevent one from being able to carry out such a test.

Indeed, if both "left-leaning" and "right-leaning" individuals display this effect, then we can conclude that both side are disposed to be unreflective & closed-minded in their thinking on this very issue.

That's what we see, of course:

If you can't make this out, click for enlarged version (and then by all means book an appointment with an optometrist!)

How likely subjects were to credit the evidence that the CRT was "valid" in this study was conditional on the interaction of experimental treatment and ideology: that is, the more conservative one becomes, the more likely one is to conclude the CRT is valid in the "believers score lower" condition and invalid in "skeptics score lower" one; vice versa as people become more liberal.

This inference doesn't depend on the CRT being an "open-mindedness" measure: the design would have worked just as well if the measure whose validity was being assessed was a fictional "open-mindedness" one.

Nevertheless, the effect observed is conditional on the subjects' CRT scores: the higher their score, the more likely they were to display this bias.

Accordingly, we can see that the form of critical-reasoning proficiency measured by CRT is indeed magnifying ideologically biased information processing.

2. Motivated numeracy

In the second study (Kahan et al. 2013), subjects performed a standard "covariance detection" problem.

The problem is diagnostic of a tendency to over-rely on heuristic reasoning of the sort that involves trying to infer causation from examining  the differential between the number of "got better" & "got worse" in only the fictional-study "treatment" condition.  That approach generates the wrong answer.  

As expected, in the control condition of our study (effectiveness of a new skin-rash treatment), there was a very strong correlation between correctly answering the problem and numeracy (indeed, one has to be at about the 90th percentile before there is even a 70% chance of getting it right).

But in the treatment condition of our study -- in which the outcome of a "gun control ban" experiment was manipulated-- subjects higher in numeracy were more likely to get the correct answer only if the data, properly interpreted, supported the position consistent with their own ideology.

So again, we see that critical reasoning is being used opportunistically: evidence is being assessed correctly depending on whether it supports an ideologically congenial outcome.

We see, too, that critical reasoning is magnifying this effect; subjects who are higher in numeracy are reliably summoning their greater proficiency to resist heuristic reasoning only when heuristic reasoning generates an uncongenial answer.

And finally, we are seeing that the effect is symmetric. Or in any case, we see it in both the left and right. (If you want to discuss whether it is "bigger" in the right, I'm happy to go into that; the answer is actually "no"-- although many people perceive that)!

3. Lesson: Manipulate motivating stake, measure "likelihood ratio"

Note that in both of these experiments, the "high proficiency" are "more polarized" than their "low proficiency" counterparts. They are "better," more reliable, in fitting the evidence to their ideological predispositions.

If this is how people process information in the world, then we will see, in an observational study, that those higher in proficiency are more polarized. We'll be able to infer, moreover, that this is the result of the magnification of biased information processing & not a result of one "side" getting the "right answer."

Indeed, the who point of the experimental design was to unconfounding quality of reasoning from "right answer." This was done, in effect, by manipulating the motivational stake of the subjects to credit one and the same piece of evidence.

In Bayesian terms, we are demonstrating that subjects opportunistically adjust the likelihood ratio in response to an identity-protective motivation unrelated to assessing the truth-value weight of the new information (Kahan 2015).

This is a more general point: studies that purport to show "motivated reasoning" or "biased assimilation" by looking at the equivalent of "posterior probabilities" are almost always invalid. They are consistent with the inference that "one side is biased," or even that "neither side is," because differences in opinion after review of evidence is consistent with different priors or pre-treatment effects (consideration of the evidence in question or its equivalent) before the study. One should manipulate the stake the subjects have in the outcome & assess how that effects the likelihood ratio assigned to one and the same piece of evidence -- conceptually speaking (Kahan 2013).

References

Saturday
Jan172015

Weekend update: More on HPV vaccine disaster (Science of Science Communication 2.0, Session 1)

Tamar Wilner has a great "reaction paper" response to the Session 1 materials on the "HPV vaccine disaster."  She asks, provocatively, whether the "media ... manufacture[d] the HPV vaccine controversy" and then answers, moderately, "well, probably not" while flagging difficult issues for science-communication journalists about their role in covering science issues that become connected to cultural controversy.

Read Tamar's post for yourselves-- & comment, if you have anything to add.

I've posted my comment, both below, as a comment, and on the Session 1.1 post, and at Tamar's site.  I'm gonig to close comments here so that the discussion doesn't get too fractured.

Read the rest at Tamar Wilner's site!

Thursday
Jan152015

National Academy Conference on Science Communication & GMOs

This is where I am today.

Tune in via webcast!

Tuesday
Jan132015

Science of Science Communication 2.0, Session 1.1: The HPV Vaccine Disaster

page 1 of the course materials-- click it!Okay, so here is the first post for the "virtual" Science of Science Communication course 2.0. Actually, I'll be teaching/learning/attending the first "real world" session in about 30 mins.  

Today's readings are in the nature of a "case study" of the introduction of the HPV vaccine in the U.S. & its status today. The material below is in the nature of a "set up" for discussion, which I encourage people to have whereever they want but also in the comments section.  

I'm designating this "Session 1.1" in anticipation that I might myself post something-- in the nature of a "follow up" -- in which case I'll designate "Session 1.2."  Or maybe I'll do something else, who knows.

BTW check out this super cool & generous invitation if you are looking for "virtual" classmates & course materials!!

The introduction of the HPV vaccine in the US-- or the cost of being innocent & ignorant of the science of science communication. . . .

1. Merck’s application for “fast-track” review of female-only shot. It is late 2005. Merck, the manufacturer of the HPV vaccine Gardasil, has applies for “fast-track” FDA approval of a female-only shot.

HPV—the human papilloma virus—is a sexually transmitted pathogen. Exposure is widespread: some 45% of women in their early 20s have been infected. A comparable percentage of men almost certainly have been, too, although there is at this time no effective test for males.

HPV causes genital warts in some but not all infected individuals.

It is also the sole cause of cervical cancer. A diseases that can normally be detected at an early stage by a routine pap smear and thereafter successfully treated, cervical cancer nevertheless claims the lives of 3,000 women per year in the U.S. (many more in undeveloped countries that lack effective public health systems).

Clinical tests show that Gardasil confers immunity against 70% of the strains of HPV that can cause cervical cancer. This evidence furnishes logical reason to believe that widespread immunization would reduce cervical cancer rates, although the vaccine is in fact too new, and experience with it in nations where it is already in use too limited, for that proposition to have been empirically tested.

The role of HPV in causing cervical cancer is the basis for Merck’s application for “fast track” review, which is available only for drugs that furnish an “unmet medical need” for treatment or prevention of a “serious disease.” The link to cervical cancer is also why the “fast track” application is for a female-only shot: genital warts are not considered a “serious disease,” and while HPV might cause oral or anal cancer in men, there is at this time insufficient evidence to be sure.

If put on the “fast track,” Gardasil will likely be approved for use for women within six months. The FDA review process would otherwise be expected to take three additional years. Within that time frame, in fact, the FDA is likely to approve for males and females both Gardasil and Cervarix, an HPV vaccine manufactured by GlaxoSmithKline and already approved for use in Europe.

2. Health risks? Clinical trials suggest no reason to believe Gardasil poses a risk of dangerous side-effects. Some critics question the quality of this evidence, however, noting the recent withdrawal of Merck’s anti-inflammatory Vioxx based on evidence, known but not initially acknowledged by Merck, that showed the drug increased the risk of heart attacks and strokes.

Other critics suggest that widespread HPV immunization could have perverse behavioral consequences. To be effective, immunization should occur during adolescence, before an individual is likely to have become exposed to the disease through sexual activity. Some groups, including social conservative and religious ones, have voiced concern that immunization will generate a sense of false security in teenage girls, who will therefore be more likely to engage in unprotected sex, exposing themselves to a higher risk of pregnancy or other STDs. There is currently no evidence one way or the other on whether HPV immunization of adolescent girls would have any such effects.

3. The proposed legislative initiative. In addition to seeking fast-track approval of Gardasil, Merck is known to be organizing a nationwide lobbying campaign aimed at securing legislation adding the HPV vaccination to the schedule of universal childhood immunizations treated as a condition of public-school enrollment.

As part of this effort, Merck has reached out to women’s health advocacy groups. These groups strongly support making the HPV vaccine available in the U.S. Merck has proposed that these groups play a lead role in the company’s lobbying campaign, which would be funded by Merck. Merck is also understood to be searching for social conservatives to participate in the campaign.

4. Physicians’ views. There is every reason to believe physicians will view the availability of an HPV vaccination as a very positive development. No major U.S. medical association, however, has taken a position on either Merck’s fast-track proposal or on adding the HPV vaccine to states’ school-enrollment immunization schedules.

At least some physicians, however, have voiced criticism of how the vaccine is being introduced. They assert that Merck’s fast-track application and its planned nationwide legislative campaign are economically motivated: Merck’s goal, they have argued (in various fora, including medical journal commentaries), is to establish a dominant position in the market before the FDA approves of GlaxoSmithKline’s rival Cervarix vaccine. Whatever public health benefits might be associated with accelerating the speed with which Gardasil is approved and HPV vaccine added to universal vaccination schedules, these commentators have warned, will be offset by the increased risk of a political backlash.

5. Political controversy? At this point, there is no meaningful dispute over Gardasil. Indeed, only a minute fraction of the U.S. population has ever heard of the vaccine or even HPV for that matter.

Nevertheless, the prospect of controversy has already been anticipated in the national media. A government-mandated STD shot for adolescent girls, these sources predict, is certain to provoke confrontation between women’s rights groups and religious and social conservatives.

Aside from some women’s’ health groups, the only other advocacy group to address the HPV vaccine is the Family Research Council. Committed to protecting religious values in American life, FRC has played a major role in opposing public-school instruction on birth control. The FRC has stated that it does not oppose—indeed, “welcomes”—the introduction of the HPV vaccine, but views state-mandated vaccination as interfering with parental control of their children’s’ health and their sexual behavior.

5. The HBV vaccine. The HPV vaccine would not be the first STD immunization to be placed on states’ school-enrollment vaccination schedules. A decade ago the FDA approved the HBV vaccine for hepatitis-b, a sexually transmitted disease that causes a lethal form of liver cancer. The CDC quickly recommended that the vaccine, which had been approved for both males and females, be added to the list of universal childhood immunizations. Within several years, almost every state had added the HBV vaccine to its mandatory-immunization schedule via regulations issued by state public health officials, the conventional—and politically low-profile process—for updating such provisions. The addition of the HBV vaccine to the state schedules generated no particular controversy, and the nationwide vaccination rate for HBV, like other childhood immunizations, has consistently been well over 90%.

6. “Public acceptance” research. Public health researchers have conducted studies specifically aimed at assessing the public acceptability of an adolescent HPV shot. These studies, which consist of surveys of parents with adolescent children, uniformly report that parents say they are unfamiliar with the HPV vaccine but will have their children immunized if their pediatricians recommends doing so.

Issues. Should the FDA grant Merck’s application for fast-track review? Should Merck withdraw it? Should women’s’ advocacy groups agree to participate in the company’s nationwide legislative campaign? What position, if any, should medical professional associations take? Is the position of social conservative groups like the FRC relevant to these questions?

Thursday
Jan082015

The "disentanglement research program": a fragment

From something I'm working on -- & closely related to what is described here, of course.

The "disentanglement project": an empirical research program

“Evolution” refers not only to a scientifically grounded account of the natural history of life on earth but also to a symbol in relationship to which people's stances signify membership in one or another cultural group.  The confounding of the former and the latter are at the root of a cluster of related societal problems. One is simply how to measure individual comprehension of evolutionary science and science generally. Another is how to impart collective knowledge on terms that avoid needlessly conditioning its acquisition on an abandonment or denigration of cultural commitments collateral to science.  And a final problem is how to protect the enterprise of acquiring, assessing, and transmitting knowledge from becoming a focal point for cultural status competition corrosive of the reciprocal benefits that science and liberal democratic governance naturally confer on one another.  This paper discuss the “disentanglement project,” an empirical research program aimed at identifying an integrated set of practices for unconfounding the status of evolution as a token of collective knowledge and a symbol of cultural identity within the institutions of the liberal state. 

Wednesday
Jan072015

So you want to meet the 'Pakistani Dr'? Just pay a visit to the Kentucky Farmer

I now realize that a lot of people think that Hameed’s Pakistani Dr—who without apparent self-contradiction “disbelieves” in evolution “at home” but “believes” in it at work—is a mystery the solution to which must have something to do with his living in Pakistan (or at least having grown up and gone to school there before moving to the US to practice medicine (Everhart & Hameed (2013)).

That’s a big mistake! 

Indeed, in my view it gets things exactly backwards: what makes the Pakistani Dr so intriguing, & important, is that he is the solution to mysteries about the psychology of a lot of people born & bred right here in the U.S. of A!

One place where you can find a lot of Pakistani Drs, e.g., is in the South & Midwest, where their occupation of choice is farming.

Public opinion studies consistently find that farmers are deeply skeptical of climate change (e.g., Prokopy et al. 2014).

Which is to say, when you ask them if they believe human fossil-fuel burning is heating up the planet, they say, “Heck no! Don’t give me that Al Gore bull shit!”

But that’s what happens, you see, if you ask them about what they believe “at home.” 

If you ask them what they believe “at work,” where they must make practical decisions based on the best available evidence, then you are likely to get a completely different answer!

Or so a group of researchers recently reported in an amazingly cool study published in the Journal of Agricultural and Applied Economics (Rejesus, Hensley, Mithcell, Coble & Knight 2013).

Analyzing the results of an N = 1380 USDA-conducted survey of farmers in Mississippi, North Carolina, Texas, and Wisconsin, RHMCK reported that less than 50% in each state agreed with the statement, “I believe human activities are causing changes in the earth climate.”

Indeed, only a minority—around a quarter of the respondents in Mississippi, Texas, and Wisconsin; a bit over a third in North Carolina—indicated that they “believe climate change has been scientifically proven” at all.

But when these same respondents answered questions relating to how climate change would affect farmers, only a small minority expressed any doubt whatsoever that the impact would be considerable.

For example,

nearly 60% of producers in Mississippi and Texas, states where scientific proof of climate change is typically not agreed to, believe there will be some change in crop mix resulting from climate change.

Majorities in Mississippi (55%) and North Carolina (56%) also indicated that it was likely that, in response to climate change, farmers in their state would be buying more crop insurance to protect them from the increased variability in yields associated with a higher incidence of extreme weather events.

Of course, you can insure yourself from risks only if the benefits exceed the expected costs of enduring them. A lot of farmers think that farming won't be profitable be in the future-- thanks to climate change.

In North Carolina (57%) and Texas (51%), a majority of the respondents indicated that they thought it was either “likely” or “extremely likely” that climate change would force some farmers out of business.

In none of the states did anything even close to a majority indicate that they thought it was either "unlikely” or “extremely unlikely" that farmers would resort to greater crop rotation, increased insurance coverage, or simply quitting the business altogether in response to climate change.

Obviously, some fraction of the positive responses to these questions came from the minority of farmers in these states who indicated that they do believe climate change is "scientifically proven." 

But it turns out the views of “believers” and “disbelievers” on these matters didn’t vary by much.

  • Likely that farmers will resort to crop diversification as a result of climate change
    Believers: 51% agree 
    Disbelievers: 47% agree

  • Likely that farmers will be driven out of business by climate change:
    Believers: 50% agree
    Nobelievers: 47% agree

  • Likely that farmers will acquire greater crop insurance protection to deal with climate change:
    Believers: 56% agree
    Nonbelievers: 45%, agree

These self-report data, moreover, match  up quite well with behavioral data, which show that climate-skeptical farmers are already adopting practices (like no-till planting, new patterns of crop rotation, adjustments in growing season projections) in anticipation of climate impacts.

Business actors, moreover, are rushing in to profit from the willingness of farmers to pay for services and technologies that will help them weather climate change. Just ask Monsanto, which is perfectly happy to proclaim its belief in climate change, how excited farmers are about its climate-change resistant GM crops, as well as the company’s new business ventures in supplying climate data and climate-change crop insurance.

How to make sense of this?

 The most straightforward answer is the one set forth in the Measurement Problem (Kahan in press): whether people say they “believe” or “disbelieve in” human-caused climate change is not a valid measure of what they know about climate science; rather it is simply an indicator ofidentity on a par with people’s responses to items that solicit their cultural values, their right-left political outlooks, their religiosity or whathaveyou.

Farmers who express their cultural identity by saying they “disbelieve in” human-caused climate change actually do know a lot about it—much more, probably, than the average person who says he or she does “believe in” climate change but who it turns out is highly likely to think that global warming is caused by sulfur emissions and will stifle photosynthesis in plants.

What do "believers" & "nonbelievers" in human-caused climate change know about climate science? Not much! Click & see for self.In the Measurement Problem study, I used a climate-literacy assessment instrument the items of which were carefully calibrated to disentangle or unconfound "identity" and "knowledge."

To me, the RHMCK results suggest that one can unconfounded "identity" and "knowledge" in an equivalent way with items that, unlike the cultural-identity-eliciting "do you believe in climate change" item, effectively assessed what farmers understood the evidence of climate change to signify for their vocation.

Cool, okay.

But the much more difficult question is—what exactly is going on in the heads of those farmers who clearly comprehend the evidence but who say they “don’t believe in” climate change?

This is exactly what the Pakistani Dr has been trying, so patiently, to help us figure out!

If he hadn't been so persistent in trying to pierce through the dense armor of my incomprehension, I would have had nothing more to say than what I just did—viz., that what a farmer in Mississippi, Texas, North Carolina, or Wisconsin says he “believes” about climate change measures something entirely different from what he “knows” about it.

But now, thanks to what the Dr has taught me, I have a hunch that the “climate change” that that farmer doesn’t “believe in” & the "climate change” he does “believe in” are, as the Dr would say, "entirely different things!"

“Climate change,” certainly, can be defined with reference solely to a state of affairs, or the evidence for it.

But as an object of belief or knowledge, climate change can’t be defined that way.

It’s just plain weird, really, to imagine that if we could somehow take a person, unscrew the lid of his mind, turn him upside down, and shake him a bit, a bunch of discrete “beliefs” would fall onto the ground in front of us. 

What we believe or know—the objects of those intentional states—don’t have any existence independently of what we do with them.  The kinds of things we do, moreover, are multiple and diverse—and correspond to the multiple and diverse roles our integrated identities comprise.

The Pakistani Dr is an oncologist and a proud member of a science-trained profession.  His belief in evolution enables him to be those things.

He is also a devout Muslim.  His disbelief of evolution enables him to be that—when being that is what he is doing.

There’s no conflict!, he keeps insisting. The evolution he “accepts” and the evolution he “rejects” are entirely different things—because the things he is doing with those intentional states are entirely different, and, fortunately for him, perfectly compatible with each other in the life he leads.

This is Scott Travis, the Kentucky farmer. Click to have a conversation. He can teach you something.Well, for the Kentucky (Mississippi/Texas/North Carolina/Wisconsin/Indiana  etc.) farmer, there are two climate changes: the one he rejects to protect his standing in a particular cultural community engaged in an ugly status competition with another whose members’ “belief in” climate change serves the same function; and the one the Kentucky farmer accepts in the course of using his reason to negotiate the challenges of his vocation.

Sadly, the Kentucky farmer lives in a society that makes reconciling the diverse roles that he occupies—the different things he is enabled to do—by “believing in” one “climate change” and “disbelieving in” another much less straightforward, routine--boring even--than what the Pakistani Dr does when he accepts one evolution and reject another.

This is a big problem.  Not just for the Kentucky farmer but for all those who live in the society so many of whose members find what the Kentucky farmer is doing with his reason not only incomprehensible but simply intolerable. 

References

Everhart, D. & Hameed, S. Muslims and evolution: a study of Pakistani physicians in the United States. Evo Edu Outreach 6, 1-8 (2013).

Kahan, D. Climate Science Communication and the Measurement Problem. Advances in Pol. Psych (in press).

Prokopy, L.S., Morton, L.W., Arbuckle, J.G., Mase, A.S. & Wilke, A. Agricultural stakeholder views on climate change: Implications for conducting research and outreach. Bulletin of the American Meteorological Society  (2014).

Rejesus, R.M., Mutuc-Hensley, M., Mitchell, P.D., Coble, K.H. & Knight, T.O. US Agricultural Producer Perceptions of Climate Change. Journal of Agricultural and Applied Economics 45 (2013).

 

 

Tuesday
Jan062015

"Science of Science Communication" course, version 2.0

This semester I will be teaching my "science of science communication" course for 2nd time.  

I got my act together this time, too, and had the course, which is a Psychology Dept graduate offering, cross-listed in the School of Public Health, the School of Forestry and Environmental Studies, plus the Law School.  The value of the "science of science communication," it seems to me, depends entirely on the function it can perform in integrating the production of scientific knowledge & science-informed policymaking, on the one hand, with scientific knowledge of the processes by which people come to know what is known by science, on the other. So obviously, offerings like this shouldn't be "in the course catalog" of only decision-science or communication-science disciplines.... 

Anyway, like last time, I'm going to see if I can offer a "virtual" counterpart of the course via this blog.

I'll post course materials, as they become available, here.  Unfortunately, I can't post the readings themselves, since access to portions of them is restricted to users covered by one or another of Yale University's site licenses or subscriptions to various commercial content providers. But I will post the reading lists & various "open access" materials.

After each "real space" session, though, I'll post some sort of synopsis or argument or whatever as a "starter" for discussion.  People can weigh in based on their access to that, plus whatever else they can get their hands on -- including materials other than those assigned to students enrolled in the course at Yale!

This worked pretty well last time, except I wasn't as conscientious as I should have been in posting "starters."

This time I'll do better!

Below I've posted the "course catalog" description of the course, plus the "manifesto" that introduces the course requirements & topics etc.

 

The Science of Science Communication, PSYC 601b, FES 862b, HPM 601, LAW 21141. The simple dissemination of valid scientific knowledge does not guarantee it will be recognized by non-experts to whom it is of consequence. The science of science communication is an emerging, multidisciplinary field that investigates the processes that enable ordinary citizens to form beliefs consistent with the best available scientific evidence, the conditions that impede the formation of such beliefs, and the strategies that can be employed to avoid or ameliorate such conditions. This seminar surveys, and makes a modest attempt to systematize, the growing body of work in this area. Special attention is paid to identifying the distinctive communication dynamics of the diverse contexts in which non-experts engage scientific information, including electoral politics, governmental policy making, and personal health decision making.

* * *

1. Overview. The most effective way to communicate the nature of this course is to identify its motivation.  We live in a place and at a time in which we have ready access to information—scientific information—of unprecedented value to our individual and collective welfare. But the proportion of this information that is effectively used—by individuals and by society—is shockingly small. The evidence for this conclusion is reflected in the manifestly awful decisions people make, and outcomes they suffer as a result, in their personal health and financial planning. It is reflected too not only in the failure of governmental institutions to utilize the best available scientific evidence that bears on the safety, security, and prosperity of its members, but in the inability of citizens and their representatives even to agree on what that evidence is or what it signifies for the policy tradeoffs acting on it necessarily entails.

This course is about remedying this state of affairs. Its premise is that the effective transmission of consequential scientific knowledge to deliberating individuals and groups is itself a matter that admits of, and indeed demands, scientific study.  The use of empirical methods is necessary to generate an understanding of the social and psychological dynamics that govern how people (members of the public, but experts too) come to know what is known to science. Such methods are also necessary to comprehend the social and political dynamics that determine whether the best evidence we have on how to communicate science becomes integrated into how we do science and how we make decisions, individual and collective, that are or should be informed by science.

Likely you get this already: but this course is not simply about how scientists can avoid speaking in jargony language when addressing the public or how journalists can communicate technical matters in comprehensible ways without mangling the facts.  Those are only two of many science communication” problems, and as important as they are, they are likely not the ones in most urgent need of study (I myself think science journalists have their craft well in hand, but we’ll get to this in time).  Indeed, in addition to dispelling (assaulting) the fallacy that science communication is not a matter that requires its own science, this course will self-consciously attack the notion that the sort of scientific insight necessary to guide science communication is unitary, or uniform across contexts—as if the same techniques that might help a modestly numerate individual understand the probabilistic elements of a decision to undergo a risky medical procedure were exactly the same ones needed to dispel polarization over climate science! We will try to individuate the separate domains in which a science of science communication is needed, and take stock of what is known, and what isn’t but needs to be, in each.

The primary aim of the course comprises these matters; a secondary aim is to acquire a facility with the empirical methods on which the science of science communication depends.  You will not have to do empirical analyses of any particular sort in this class. But you will have to make sense of many kinds.  No matter what your primary area of study is—even if it is one that doesn’t involve empirical methods—you can do this.  If you don’t yet understand that, then perhaps that is the most important thing you will learn in the course. Accordingly, while we will not approach study of empirical methods in a methodical way, we will always engage critically the sorts of methods that are being used in the studies we examine, and I from time to time will supplement readings with more general ones relating to methods.  Mainly, though, I will try to enable you to see (by seeing yourself and others doing it) that apprehending the significance of empirical work depends on recognizing when and how inferences can be drawn from observation: if you know that, you can learn whatever more is necessary to appreciate how particular empirical methods contribute to insight; if you don’t know that, nothing you understand about methods will furnish you with reliable guidance (just watch how much foolishness empirical methods separated from reflective, grounded inference can involve).

Friday
Jan022015

Humans using statistical models are embarrassingly bad at predicting Supreme Court decisions....

Part of Lexy's brain. From Ruger et al. (2004)Demoralizingly (for some people; I don't mind!), computers have defeated us humans in highly discerning contests of intellectual acuity such as chess and Jeopardy.

But what about prediction of Supreme Court decisions?  Can we still at least claim superiority there?

Well, you tell me.

In 2002-03, a group of scholars organized a contest between a computer and a diverse group of human "experts" drawn from private practice and the academy (Ruger, Kim, Martin & Quinn 2004).

Political scientists have actually been toiling for quite a number of years to develop predictive models for the Supreme Court.  The premise of their models is that the Court’s decisionmaking can be explained by “ideological” variables (Edwards & Livermore 2008).

In the contest, the computer competitor, Lexy (let’s call it), was programmed using the field’s state-of-the-art model, which in effect tries to predict the Court's decisions based on a combination of variables relating to the nature of the case and the parties, on the one hand, and the ideological affinity of individual Justices as reflected by covariance in their votes, on the other.

For this reason, the contest could have been seen (often is described) as one that tested the political scientists’ “ideology thesis” against “formal legal reasoning.” 

But in fact, that's a silly characterization, since the informed professional judgment of genuine Supreme Court experts would certainly reflect the significance of "Justice ideology" along with all the other influences on the Court’s decisionmaking (Margolis 1987, 1996; Llewellyn, 1960).

In any case, Lexy trounced those playing the role of “experts” in this contest.  The political scientists' model correctly predicted the outcome in 75% of the decisions, while the experts collectively managed only 59% correct . . . .

The result was widely heralded as a triumph for both  algorithmic decisionmaking procedures over expert judgment & for the political scientists’ “ideology thesis."

But here’s the problem: while Lexy “did significantly better at predicting outcomes than did the experts” (Ruger et al. 2004, p. 1152), Lexy did not perform significantly better than chance!

The Supreme Court’s docket is discretionary: parties who’ve lost in lower courts petition for review, and the Court decides whether to hear their cases.

It rejects the vast majority of review petitions—96% of the ones on the "paid" docket and 99% of those on the “in forma pauperis” docket, in which the petitioner (usually a self-represented prisoner) has requested waiver of the filing fee on grounds of economic hardship.

Not surprisingly, the Court is much more likely to accept for review a case in which it thinks the lower court has reached the wrong result.

Hence, the Court is far more likely to reverse than to affirm the lower court decision. It is not unusual for the Court to reverse in 70% of the cases it hears in a Term (Hofer 2010).  The average Supreme Court decision, in other words, is no coin toss!

Under these circumstances, the way to test the predictive value of a statistical model is to ask how much better someone using the model would have done than someone uniformly picking the most likely outcome--here, reversal-- in all cases (Scott 1997, pp. 107-08; Pampel 2000, p. 51).

In the year in which Lexy squared off against the experts, the Court heard only 68 cases.  It reversed in 72% of them. 

Thus, a non-expert who knew nothing more than that the Supreme Court reverses in a substantial majority of its cases, and who simply picked "reverese" in every case, would have correctly predicted 72% of the outcomes.  The margin between her performance and Lexy's 75% -- a grand total of two fewer correct predictions -- doesn't differ significantly (p = 0.58) or practically from zero. 

A practical person, then, wouldn't bother to use Lexy instead of just uniformly predicting "reverse."

None of the “holy cow!” write-ups on Lexy’s triumph—which continue to this day, over a decade after the contest—mentioned that the algorithm used by Lexy had no genuine predictive value.

But to be fair, the researchers didn't mention that either.

They noted that the the Supreme Court had reversed in 72% of its cases only in footnote 82 of their 60-page, 122-footnote paper. And even there they didn't acknowledge that the predictive superiority of their model over the "72% accuracy rate" one would have achieved by simply picking "reverse" in all cases was equivalent—practically and statistically—to chance.  

Instead, they observed that the Court had in some "recent" Terms reversed less than 70% of the granted cases. The previous Terms were in fact the the source of the researchers' "training data"--the cases they used to construct their statistical model. They don't say, but one has to believe that their "model" did a lot better than 75% accuracy when it was "fit" retrospectively to those Terms' cases-- or else the researchers would surely have tinkered with its parameters all the more.  But that the resulting model performed no better than chance (i.e, than someone uniformly picking "reverse," the most likely result in the training data) when applied prospectively to a new sample is a resounding verdict of “useless” for the algorithm the researchers derived by those means.

Sure, the “experts” did even worse, and that is embarrassing: it means they'd have done better by not "thinking" and instead just picking "reverse" in all cases, the strategy a non-expert possessed only of knowledge of the Court's lopsided proclivity to overturn lower court decisions would have selected.

But the "experts'" performance—a testament perhaps to their hubristic overconfidence in their abilities but also largely to the inclusion of many law professors who had no general specialty in Supreme Court decisionmaking —doesn’t detract from the conclusion that the statistical model they were up against was a complete failure.

What’s more, I don’t think there’s anything for Lexy or computers generally to feel embarrassed about in this matter. After all, a computer didn’t program Lexy; a group of humans did.

The only thing being tested in that regard was the adequacy of the “ideology thesis”-prediction model developed by political scientists.

That the researchers who published this study either didn’t get or didn’t care that this model was shown to perform no better than chance makes them the ones who have the most to be embarrassed about.

References 

Edwards, H.T. & Livermore, M.A. Pitfalls of empirical studies that attempt to understand the factors affecting appellate decisionmaking. Duke LJ 58, 1895-1989 (2008).

Hofer R. Supreme Court Reversal Rates: Evaluating the Federal Courts of Appeals, Landslide 2, 10 (2010). 

Llewellyn, K.N. The Common Law Tradition: Deciding Appeals (1960).


Long, J.S. Regression models for categorical and limited dependent variables (Sage Publications, Thousand Oaks, 1997).

Margolis, H. Dealing with risk : why the public and the experts disagree on environmental issues (University of Chicago Press, Chicago, IL, 1996).

Margolis, H. Patterns, Thinking, and Cognition (1987).

Pampel, F.C. Logistic regression : a primer (Sage Publications, Thousand Oaks, Calif., 2000).

Ruger, T.W., Kim, P.T., Martin, A.D. & Quinn, K.M. The Supreme Court Forecasting Project: Legal and Political Science Approaches to Predicting Supreme Court Decisionmaking. Columbia Law Rev 104, 1150-1210 (2004).