Serious problems with "the strongest evidence to date" on consensus messaging ... - Cultural Cognition of Health

van der Linden, Leiserowitz, Feinberg & Maibach (2015) posted the data from their study purporting to show that subjects exposed to a scientific-consensus message “increased” their “key beliefs about climate change” and “in turn” their “support for public action” to mitigate it.

Christening this dynamic the “gateway belief” model, VLFM touted their results as  “the strongest evidence to date” that “consensus messaging”— social-marketing campaigns that communicate scientific consensus on human-caused global warming—“is consequential.”

At the time they published the paper, I was critical because of the opacity of the paper’s discussion of its methods and the sparseness of the reporting of its results, which in any case seemed underwhelming—not nearly strong enough to support the strength of the inferences the authors were drawing.

But it turns out the paper has problems much more fundamental than that.

I reanalyzed the data, which VLFM posted in March, a little over a year after publication,  in conformity with the “open data” policy of PLOS ONE , the journal in which the article appeared.

As I describe in my reanalysis, VLFM fail to report key study data necessary to evaluate their study hypotheses and announced conclusions.

Their experiment involved measuring the “before-and-after” responses of subjects who received a “consensus message”—one that advised them that “97% of climate scientists have concluded that human-caused climate change is happening”—and those who read only “distractor” news stories on things like a forthcoming animated Star Wars cartoon series.

In such a design, one compares the “before-after” response of the “treated” group to the “control,” to determine if the “treatment”—here the consensus message—had an effect that differed significantly from the control placebo. Indeed, VLFM explicitly state that their analyses “compared” the response of the consensus-message and control-group subjects

But it turns out that the only comparison VLFM made was between the groups’ respective estimates of the percentage of climate-change scientists who subscribe to the consensus position. Subjects who read a statement that “97% of climate scientists have concluded that climate-change is happening” increased theirs more than did subjects who viewed only a distractor news story.

But remarkably VLFM nowhere report comparisons of the two groups’ post-message responses to items measuring any of the beliefs and attitudes for which they conclude perceived scientific consensus as a critical “gateway” .

Readers including myself, initially, thought that such comparisons were being reported in a table of “differences” in “Pre-” and “Post-test Means” included in the article.

But when I analyzed the VLFM data, I realized that, with the exception of the difference in “estimated scientific consensus,” all the “pre-” and “post-test” means in the table had combined the responses of consensus-message and control-group subjects.

There was no comparison of the pre- and post-message responses of the two group of subjects; no analysis of whether their responses differed –the key information necessary to assess the impact of being exposed to a consensus message.

Part of what made this even harder to discern is that VLFM presented a complicated “path diagram” that can be read to imply that exposure to a consensus message initiated a “cascade” (their words) of differences in before-and-after responses, ultimately leading to “increased support for public action”—their announced conclusion.

But this model also doesn’t compare the responses of consensus-message and control-group subjects on any study measure except the one soliciting their estimates of the “percentage of scientists [who] have concluded that human-caused climate change is happening.”

That variable is the only one connected by an arrow to the “treatment”–exposure to a consensus message.

As I explain in the paper, none of the other paths in the model distinguishes between the responses of subjects “treated” with a consensus message and those who got the “placebo” distractor news story. Accordingly, the “significant” coefficients in the path diagram reflect nothing more than correlations between variables one would expect to be highly correlated given the coherence of people’s beliefs and attitudes on climate change generally.

In the paper, I report the data necessary to genuinely compare the responses of the consensus-message and control-group subjects.

It turns out that, subjects exposed to a consensus message didn’t change their “belief in climate change” or their “support for public action to mitigate it” to an extent that significantly differed, statistically or practically, from the extent to which control subjects changed theirs .

Indeed, the modal and median effects of being exposed to the consensus message on the 101-point scales used by VLFM to measure “belief in climate change” and “support for action” to mitigate it were both zero –i.e., no difference in “after” or “before” responses to these  study measures.