follow CCP

Recent blog entries
popular papers

Science Curiosity and Political Information Processing

What Is the "Science of Science Communication"?

Climate-Science Communication and the Measurement Problem

Ideology, Motivated Cognition, and Cognitive Reflection: An Experimental Study

'Ideology' or 'Situation Sense'? An Experimental Investigation of Motivated Reasoning and Professional Judgment

A Risky Science Communication Environment for Vaccines

Motivated Numeracy and Enlightened Self-Government

Making Climate Science Communication Evidence-based—All the Way Down 

Neutral Principles, Motivated Cognition, and Some Problems for Constitutional Law 

Cultural Cognition of Scientific Consensus
 

The Tragedy of the Risk-Perception Commons: Science Literacy and Climate Change

"They Saw a Protest": Cognitive Illiberalism and the Speech-Conduct Distinction 

Geoengineering and the Science Communication Environment: a Cross-Cultural Experiment

Fixing the Communications Failure

Why We Are Poles Apart on Climate Change

The Cognitively Illiberal State 

Who Fears the HPV Vaccine, Who Doesn't, and Why? An Experimental Study

Cultural Cognition of the Risks and Benefits of Nanotechnology

Whose Eyes Are You Going to Believe? An Empirical Examination of Scott v. Harris

Cultural Cognition and Public Policy

Culture, Cognition, and Consent: Who Perceives What, and Why, in "Acquaintance Rape" Cases

Culture and Identity-Protective Cognition: Explaining the White Male Effect

Fear of Democracy: A Cultural Evaluation of Sunstein on Risk

Cultural Cognition as a Conception of the Cultural Theory of Risk

« More evidence of AOT's failure to counteract politically motivated reasoning | Main | In Cambridge, MA w/ nothing to do this afternoon? Come see cool panel discussion »
Sunday
Oct292017

Weekend update--it's baaaaaack! Our paper explaining why N=55, 95% liberal, is not a valid sample for "replicating" our "motivated numeracy" study

After a brief hiatus (primarily so we could reanalyze the data after using multiple imputation to handle missing data), our working paper responding to Ballarini & Sloman (2017) is back up at SSRN

As you likely will recall, B&S reported their "failure to replicate" our motivated numeracy study. Our response points out that B&S's N=55 student sample, which was 95% liberal (not a joke), had inadequate statistical power to replicate our study, which in addition to employing a design very different from B&S's used a large (N = 1100), nationally representative sample.

In addition to our paper, you can (re)read Mark Brandt's very reflective blog post on our paper and B&S's.

I'm still baffled about B&S's motivations for making such a weakly supported claim.  Very weird . . . .

 

 

 

PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (10)

Dan,

"I'm still baffled..." Have you asked Sloman? If not, why not?

I know what you said (Sloman should know better), but I'm still betting it was just a case of giving an undergrad a first chance, even though the study was so weak. I'm not saying this is a good excuse, but it might explain the behavior.

October 29, 2017 | Unregistered CommenterJonathan

@Jonathan

No comment.

For now.

October 29, 2017 | Registered CommenterDan Kahan

Undergrads need practice writing and submitting papers if they want to go into research, and they have limited resources for financing big studies. Given the usual quality of undergrad student projects, I'm actually quite impressed that she produced something of sufficient quality for a journal to publish! Not many undergrads manage to do that. But I'd not see it is a major refutation of Dan's study, more as an exercise in paper-writing. The subject/conclusion is fairly irrelevant for that purpose.

I suppose it's also a compliment that Dan took the work seriously enough to write a response paper!

For what it's worth, I've heard the same thing happens in other research areas, like medicine. I recall reading an article complaining about the plague of data-dredge papers making the "X causes cancer!" claims beloved by the tabloid press, saying that the problem was it took a new researcher five to ten years to get enough experience to produce worthwhile interesting and original research, but to keep their job they needed to publish lots of papers from the get-go. (Publish or perish.) So what happened was they got given lots of data-dredge exercises to generate "significant" results they could publish, and which then some other student researcher could produce another paper refuting it. Experienced medical researchers in the field soon learn to recognise and ignore them. This sounds like the same thing.

It's one of the many reasons why simply being published in the peer-reviewed journal literature should hold no authority. The 'publish or perish' culture on the part of research funders is what leads to this sort of distortion, in conformity with Goodhart's Law.

October 29, 2017 | Unregistered CommenterNiV

" I'm still baffled about B&S's motivations for making such a weakly supported claim "

Whatever their motivation, which others speculate may be merely lack of experience, is it not more worrying that " B&S's N=55 student sample, which was 95% liberal (not a joke) " could in fact result in publication?

October 29, 2017 | Unregistered CommenterAndy West

@Andy-- it was published only in a conference proceedings compendium. But not surprisingly,, other scholars have noted its existence & cited it etc.

October 30, 2017 | Registered CommenterDan Kahan

@Dan - I see, thanks.

October 30, 2017 | Unregistered CommenterAndy West

@NiV-- I suspect the undergrad had little or nothing to do with the characterization of her & Sloman's paper as a "nonrepication" of our "Motivated Numeracy" study.

October 30, 2017 | Registered CommenterDan Kahan

Coincidental link drop:
https://theconversation.com/expertise-in-sciences-and-the-decision-of-what-is-publishable-a-noble-yet-endangered-task-86339

October 30, 2017 | Unregistered CommenterJonathan

"But not surprisingly,, other scholars have noted its existence & cited it etc."

Just fellow Yalies Pennycook and Rand's one paper on fake news!

My conspiracy detector, rebounding from the disappointing JFK papers release, is getting fidgety...

October 30, 2017 | Unregistered CommenterJonathan

"Whatever their motivation, which others speculate may be merely lack of experience, is it not more worrying that "B&S's N=55 student sample, which was 95% liberal (not a joke) " could in fact result in publication?"

The Mann Hockeystick paper got published in Nature, and it was a lot worse! This sort of thing is all too common.

And conference proceedings are not generally very fussy. They're usually more interested in attendee fees.

" But not surprisingly,, other scholars have noted its existence & cited it etc."

Actually, I find that a little more surprising (although to be honest, not very). Surely they should have noticed the issues themselves? They went and cited a low-grade conference paper written by an undergrad, without checking the calculations?! Tut!

Perhaps there was some motivated reasoning on the part of those other scholars? They're more inclined to check stuff they don't like or don't agree with? You could perhaps call that an additional confirmation of the claim!

"I suspect the undergrad had little or nothing to do with the characterization of her & Sloman's paper as a "nonrepication" of our "Motivated Numeracy" study."

I'd certainly expect so!

October 30, 2017 | Unregistered CommenterNiV

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>