Weekend update: Pew's disappointing use of invalid survey methods on GM food risk perceptions - Cultural Cognition of Health

So here’s a follow-up on “grading of Pew’s public attitudes toward science report” –& why I awarded it a “ C- ” in promoting informed public discussion, notwithistanding its earning an “ A ” in scholarly content (the data separated from the Center’s commentary, particularly the press materials it issued).

This follow-up says a bit more about the unscholarly way Pew handled public opinion on GM food risks.

1. It’s really easy for people to form misimpressions about “public opinion.”

Why? Because, for one thing, figuring out what “people” (who actually usually can’t usefully be analyzed w/o being broken down into groups) “think” about anything is not anything anyone can directly observe; like lots of other complicated processes, it is something we have to try to draw inferences about on the basis of things that we can observe but that are only correlates of, or proxies for, it.

For another, none of us is in the position via our personal, casual observations to collect a valid sample of the sorts of observable correlates or proxies .  We have very limited exposure, reflecting the partiality of our own social networks and experiences, to the ways in which “the public” reveals what it thinks.  And it is in fact a feature of human psychology to overgeneralize from imperfect samples like that & make mistakes as a result.

2. One of the things many many many many people are mistaken about as a result of these difficulties is “public opinion” on GM food risks.  The media is filled with accounts of how anxious people are about GM foods. That’s just not so : people consume them like mad (70% to 80% of the food for sale in a US supermarket contains GMOs).

Social science researchers know this & have been engaged in really interesting investigations to explain why this is so, since clearly things could be otherwise: there are environmental risks that irrationally scare the shit out of members of the US public generally (e.g., nuclear waste disposal). Moreover, European public opinion is politically polarized on GM foods, much the way the US is on, say, climate change.  So why not here (Peters et al. 2007; Finucane, M.L. & Holup 2005; Gaskell, Bauer, Durant & Allum 1999)? Fascinating puzzle!

That isn’t to say there isn’t controversy about GM foods in American society. There is: in some sectors of science; in politics, where efforts to regulate GM foods are advanced with persistence by interest groups (organic food companies, small farmers, entrepreneurial environmental groups) & opposed with massive investments by agribusiness; and in very specialized forms of public discourse, mainly on the internet.

Indeed, the misimpression that GM foods are a matter of general public concern exists mainly among people who inhabit these domains, & is fueled both by the vulnerability of those inside them to generalize inappropriately from their own limited experience and by the echo-chamber quality of these enclaves of thought.

3.  The point of empirical public opinion research is to correct the predictable mistakes that arise from dynamics like these.

One way empirical researchers have to tried to do this in the case of GM foods is by showing that in fact members of the public have no idea what GM foods are.

They fail miserably if you measure their knowledge of GMOs.

They also say all kinds of silly things about GM foods that clearly aren’t true: e.g., that they scrupulously avoid eating them and that they believe GM foods are already heavily regulated and subject to labeling requirements (e.g., Hallman et al. 2013).

That people are answering questions in a manner that doesn’t correspond to reality shows that the survey questions themselves are invalid . They are not measuring what people in the world think—b/c people in the world (i.e., United States) aren’t thinking anything at all about GM foods; they are just eating them.

The only things the questions are measuring—the only thing they are modeling—is how people react to being asked questions they don’t understand.

This was a major theme, in fact, of the National Academy of Science’s recent conference on science communication & GMOs.  So was the need to try to get this information across to the public, to correct t the pervasive misimpression that GM foods are in fact a source of public division in the U.S.

So what did Pew do? It issued survey items that serious social science researchers know are invalid and promoted the results in exactly the way that fosters the misimpression those researchers are trying to correct!

Pew asked members of their general public sample, “Do you think it is generally safe or unsafe to eat genetically modified foods?”

Thirty-seven percent answered “generally safe,” 57% “generally UNsafe” and 6% “don’t know/Refused.”

Eighty-eight percent of the “scientist” (AAAS member) sample, in contrast, answered “generally safe.”