follow CCP

Recent blog entries
popular papers

Science Curiosity and Political Information Processing

What Is the "Science of Science Communication"?

Climate-Science Communication and the Measurement Problem

Ideology, Motivated Cognition, and Cognitive Reflection: An Experimental Study

'Ideology' or 'Situation Sense'? An Experimental Investigation of Motivated Reasoning and Professional Judgment

A Risky Science Communication Environment for Vaccines

Motivated Numeracy and Enlightened Self-Government

Making Climate Science Communication Evidence-based—All the Way Down 

Neutral Principles, Motivated Cognition, and Some Problems for Constitutional Law 

Cultural Cognition of Scientific Consensus
 

The Tragedy of the Risk-Perception Commons: Science Literacy and Climate Change

"They Saw a Protest": Cognitive Illiberalism and the Speech-Conduct Distinction 

Geoengineering and the Science Communication Environment: a Cross-Cultural Experiment

Fixing the Communications Failure

Why We Are Poles Apart on Climate Change

The Cognitively Illiberal State 

Who Fears the HPV Vaccine, Who Doesn't, and Why? An Experimental Study

Cultural Cognition of the Risks and Benefits of Nanotechnology

Whose Eyes Are You Going to Believe? An Empirical Examination of Scott v. Harris

Cultural Cognition and Public Policy

Culture, Cognition, and Consent: Who Perceives What, and Why, in "Acquaintance Rape" Cases

Culture and Identity-Protective Cognition: Explaining the White Male Effect

Fear of Democracy: A Cultural Evaluation of Sunstein on Risk

Cultural Cognition as a Conception of the Cultural Theory of Risk

« In Cambridge, MA w/ nothing to do this afternoon? Come see cool panel discussion | Main | Are smart people ruining our democracy? What about curious ones? ... You tell me! »
Thursday
Oct192017

How & how not to do replications--guest post by someone who knows what he is talking about

Getting the Most Out of Replication Studies

by Mark Brandt

Ok. At this point, I think most people know that replications are important and necessary for science to proceed. This is what tells us if a finding is robust to different samples, different lab groups, and minor differences in procedure. If a finding is found, but never replicated is it really a finding? Most working scientists would say no (I hope).

But not all replications are created equal. What makes a convincing replication? A few years ago with a lot of help from collaborators we sat down to figure it out (at least for now; see the open access paper). A convincing replication is rigorously conducted by independent researchers, but there are also another 5 ingredients.  

1. Carefully defining the effects and methods that the researcher intends to replicate: If you don’t know what effect you are exactly trying to replicate, it is difficult to carefully plan the study and evaluate the replication attempt. This ingredient determines nearly all that follow.

2. Following as exactly as possible the methods of the original study (including participant recruitment, instructions, stimuli, measures, procedures, and analyses): The closer the replication is to the original attempt, the easier it is to infer if the original finding is confirmed (or not). Although replications that are less close or even just conceptually similar help establish the generalizability of an effect (see this nice paper), the differences make it impossible to tell if differences in results are due to the instability of the underlying effect or to differences in the design.

3. Having high statistical power: Statistical power is basically an indicator of whether your study has a chance of detecting the effect you plan to study. Statisticians will give you more precise definitions and some branches of statistics (e.g., Bayesian) don’t really have the concept. Putting these things aside, the general idea is that you should be able to collect enough data to have precise enough estimates to make strong conclusions about the effect you’re interested in. In most of the domains I work in, power is most easily increased by including more people in the sample; however, it’s also possible to increase power by increasing the number of observations in other ways (e.g., using a within-subjects design with multiple observations per person). The best way to ensure high statistical power in a replication will depend on the precise design of the original study.

4. Making complete details about the replication available, so that interested experts can fully evaluate the replication attempt (or attempt another replication themselves): To best evaluate whether a replication is a close replication attempt, it is useful to make all of the details available for external evaluation. This transparency can illuminate potential problems with either the replication attempt or the original study (or both). It is also beneficial to pre-register the replication study, including the criteria that will be used to evaluate the replication attempt.

5. Evaluating replication results, and comparing them critically to the results of the original study: Don’t just put the results out there. Interpret them too! How are the results similar to the original study and how are they different? Are they statistically similar or different? And what could possibly explain the differences? How to evaluate replication results has become its own industry, with a lot of food for thought (see this paper).

This is all fine, you might say. But how does this work in practice? Well, for one thing we’ve developed a form to help people plan and pre-register replication results. It’s available in our paper, its available here (and in French!), and its built into the Open Science Framework. It’s also useful to examine how it doesn’t work in practice.

Here we turn to a paper that Ballarini and Sloman (B&S) presented at the meeting of the Cognitive Science Society (paper is here). B&S were testing out a debiasing strategy and in that context state that they “failed to replicate Kahan et al.’s ‘motivated numeracy effect’.” To evaluate this claim we need to know what the motivated numeracy effect is and if the B&S study is a convincing replication of it.

A quick summary of the original Kahan et al paper (paper is here): a large, representative sample of Americans evaluated a math problem incorrectly when it conflicted with their prior beliefs and this was the case primarily for people high in numeracy (the people who are good at math). The design is entirely between subjects, with participants completing a scale of political beliefs, a numeracy scale, and a word problem that did or did not conflict with their beliefs. There is more to the paper; go read it.

B&S wanted to see how they could debias people within the context of the Kahan paradigm by presenting people with competing interpretations of the data in the math problem. They found that highly numerate people were more likely to adjust their interpretation based on this competing information. This is interesting. They also did not find any evidence that highly numerate people are more likely to misinterpret a belief contradicting math problem.

It is important to state that this study was conducted by independent scholars and appears to be conducted rigorously. This is a step in the right direction as it provides evidence relevant to the motivated numeracy effect that is independent of the Kahan et al group.  But did they fail to replicate?

It is actually hard to say. The first problem is that B&S used a within-subjects paradigm where participants repeatedly received math problems of the sorts used by Kahan (and a few other types). This is different than the between-subjects design of the original study and so a problem with Ingredient #2. Although within- and between-subject designs can tap into similar processes, it is up to these replication authors to show that this procedural change does not affect the psychological processes at work.

But I do not think this is the biggest problem; if it’s powerful then the motivated numeracy effect should be able to overcome some of these design changes.

The second and more consequential problem is that whereas the original study used a very large sample (N = 1111) representative of Americans, B&S use a small sample (N = 66) of students (that is further reduced for procedural reasons). This smaller sample of students makes it less likely that they will have participants with diverse political views (1% were conservative) and a range of numeracy scores. In designs with measured predictors it is necessary to have adequate range or else there won’t be enough people who are truly low numerate or conservative to test hypotheses about these subpopulations.

The small sample size also it makes it impossible to confidently estimate the size and the direction of these effects (a problem with Ingredient #3). B&S point to the within-subjects part of their design as evidence of its statistical power, but that part of the design does not address the low power for the between-subjects part of the design. That is, although they might have the necessary power to detect differences between the math problems (the within part of the design), they do not have enough people to make strong inferences about the between part of the design (numeracy and politics).

So, at the end of this, what does the B&S study tell us about the motivated numeracy effect? Not much. The sample isn’t big enough or diverse enough for these research questions (and the difference in design is an additional complication). If B&S are just interested in the debiasing aspect, then I think that these data are useful, but they should not be framed as a replication of Kahan et al; the study is not set up to convincingly replicate the motivated numeracy effect. To their credit, B&S are more circumspect in interpreting the replication aspect of their study in the discussion (in contrast to their summary in the abstract). Hopefully most readers will go beyond the abstract…

Why do I care and why should you? Replications are important, but poor replications, just like poor original studies, pollute the literature. I don’t want to discourage people from replicating Kahan et al’s work, but when it is replicated it is important for researchers to carefully recreate the conditions of the study so that we can be confident in the evidence obtained in the study. A representative sample of America is expensive, but there are other ways of recruiting participants with diverse political backgrounds (e.g., collect data from other university campuses). We need a literature of high quality studies so that we can make informed theoretical and practical decisions. Without this it will be difficult to know where to begin.

Self-replicating otters!

PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (20)

link drop:
https://www.nytimes.com/2017/10/18/magazine/when-the-revolution-came-for-amy-cuddy.html

October 24, 2017 | Unregistered CommenterJonathan

Jonathan -

I'm guessing that you've seen this?

https://fivethirtyeight.com/features/failure-is-moving-science-forward/

IIRC, Dan thought it a highly flawed article, although when asked why didn't offer an explanation...

October 24, 2017 | Unregistered CommenterJoshua

Joshua,

Yes, I saw that.

October 24, 2017 | Unregistered CommenterJonathan

Jonathan -

Very interesting article. Perhaps most interesting, IMO, is Gelman saying this:

“I don’t like interpersonal conflict,” he said.

It's an interesting aspect of this whole situation that he can believes that running a blog that stimulates and tolerates personal antipathy is not interpersonal conflict.

I keep trying to remind myself to beware the tendency towards thinking that current incidents describe some kind of meaningful, longitudinal arc of change - but it is really hard for me to fight a conclusion that the Internet is creating a new world.

October 24, 2017 | Unregistered CommenterJoshua

link drop - a consideration of cognition on climate change from the collective action/commons perspective, favoring technological and market solutions:
http://www.law.fsu.edu/docs/default-source/journals/jluel/current-issue/jluelv32n2_(01)-rosearticle_082117.pdf?sfvrsn=4

October 24, 2017 | Unregistered CommenterJonathan

Contrarian link drop - honesty isn't the best science policy?:
https://phys.org/news/2017-10-honest-scientists.html
http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0186049

October 25, 2017 | Unregistered CommenterJonathan

Just curious, but is anyone else watching the Monsanto dicamba controversy?:

https://www.nytimes.com/2017/09/21/business/monsanto-dicamba-weed-killer.html
http://www.npr.org/sections/thesalt/2017/10/26/559733837/monsanto-and-the-weed-scientists-not-a-love-story

Of course, this speaks to the trust-in-academic-science vs. trust-in-corporate-science issue we've discussed before. But, what I think is even more interesting is how the convoluted incentive network involved appears to be making for volatile group-identity motivation.

October 26, 2017 | Unregistered CommenterJonathan

Jonathan -

Thanks for those links. I found this follow-on link pretty interesting also.

http://www.npr.org/templates/transcript/transcript.php?storyId=531272125

"...the convoluted incentive network involved appears to be making for volatile group-identity motivation."

Where in those stories do you see group-identity motivation? I see it as a rather unusual situation where people locate on the issue matrix largely independent of identity. Farmers locating on both sides of the issue. Scientists also. Even politicians who are usually pretty much single-minded about regulation. Even Trump's EPA to some extent, for god's sake. Near as I can tell, the only stakeholder group that is monolithic in viewpoint is the seed-producing industry.

October 26, 2017 | Unregistered CommenterJoshua

"Where in those stories do you see group-identity motivation?"

The group-identity I was referring to are pro-GMO pro-herbicide pro-Monsanto farmers. The convoluted incentive is: Monsanto is blaming these customers for the bad impact of dicamba on neighboring farms (either due to bad spray practice, or to use of the prior volatile dicamba formulation). This might be causing these farmers to reconsider their group allegiance, and perhaps flip sides on the GMO/herbicide issue completely. Also, would they end up trusting academic scientists more than corporate scientists?

"Farmers locating on both sides of the issue." - yes, but in this case, some appear to be switching sides. It's not 2 static groups: GMO/herbicide-using vs. non-GMO/organic farmers. It's some GMO/herbicide-using farmers beginning to think that maybe they're allied with the wrong group.

Most of the motivated cognition group identity issues we've discussed have the property that the effects are hard to experience or control directly. They're either population-wide, or delayed, or distant, or involve complex interactions that are easily doubted, etc.. This one appears to be different.

October 26, 2017 | Unregistered CommenterJonathan

Jonathan -

This might be causing these farmers to reconsider their group allegiance, and perhaps flip sides on the GMO/herbicide issue completely.

Let me try some thoughts out.

I guess the question is whether there was ever an ideological identity-associated allegiance, which superseded the practical issues at hand - being economic impact/benefit - and then which flipped in this situation. I tend to doubt that was the case, unlike what we see with other issues where it is harder to see the direct line of economic impact related to orientation on the issue. There is always a grabbag of identity-oriented influences available on any issue, and in various specific contexts the salience of any particular identity-oriented influence as opposed to another can vary.

Because say, with climate change, the economic impact is so complex and multi-faceted, it is easy for political identity-orientation to become more salient when orientation on the issue is being determined. Farmers aren't identifying as farmers or Republicans when they develop their views on GMOs, but as individuals who have a clear economic incentive. Thus, as the economic impact changes, changing orientation on the issue follows. Because their economic interest identity supersedes other identity influences, their views follow accordingly.

It is rather like Republicans for whom Republican identity is particularly important; with that being the case, they can easily flip in their views on an insurance mandate, to go from thinking that a mandate is a basic obligation of personal responsibility to thinking it is the epitome of "tyranny." If the economic impact of an insurance mandate were direct and unambiguous, then it would have a greater likelihood of being a significant competition to the strength of their ideological orientation influence w/r/t that issue. Of course, there would probably be an ideological itentity association between farmers and views on GMOs, in that most farmers are probably pro GMO and Republican, but I don't think that association explains much there. Indeed, the evidence of a strong ideological identity influence on views about GMOs seems to be weak in general.


Also, would they end up trusting academic scientists more than corporate scientists?

I doubt it. Seem to me more likely that they would carve this situation out as an exception, in the same way that "knowing disbelief" works for the Pakistani doctor. They would find some way, although not really in a deliberate fashion, to not incorporate this situation into a larger ideological/political framework, where academic scientists are socialist lefties, and not to be trusted. IMO, people have little difficulty in rationalizing away logical conflicts so as to confirm previously established biases.

October 26, 2017 | Unregistered CommenterJoshua

yes, but in this case, some appear to be switching sides.

Not if the side that they were always on was defined by economic interest. As "farmers" they may have switched sides, but as people who are clearly aligned with economic interest, they haven't switched sides. They have remained on the side which benefits them most, economically. To the point where one farmer shoots another because they have competing economic interests.

It's not 2 static groups: GMO/herbicide-using vs. non-GMO/organic farmers.

So I think that employs a mistaken definition of the "groups." The non-organic, GMO using farmers haven't become organic farmers, but non-organic, GMO using farmers who are suffering a direct and clear economic impact from the use of GMOs. They aren't going to stop using Roundup-resistant seeds because of opposition to GMOs, but because they aren't effective any longer.

It's some GMO/herbicide-using farmers beginning to think that maybe they're allied with the wrong group.

Again, I don't see it that way. I don't think that they are identifying as members of "groups" in the way that you are delineating the "groups."

Most of the motivated cognition group identity issues we've discussed have the property that the effects are hard to experience or control directly. They're either population-wide, or delayed, or distant, or involve complex interactions that are easily doubted, etc..

Yes, I agree. I think that is relevant here.

This one appears to be different.

I think in the sense that economic impact is rather unambiguous, not in some sense that people are switching sides, where the sides are defined by some kind of ideological identity orientation.

October 26, 2017 | Unregistered CommenterJoshua

link drop:
https://www.the-tls.co.uk/articles/public/the-end-of-an-error-peer-review/

October 27, 2017 | Unregistered CommenterJonathan

@Jonathan & @Joshua-- cool links! thanks

October 27, 2017 | Registered CommenterDan Kahan

Jonathan's da man.

October 27, 2017 | Unregistered CommenterJoshua

"Jonathan's da man." Being retired helps.

BTW - the contrary perspective on market solutions for climate change (one of the link drops above):
https://theconversation.com/why-we-cant-rely-on-corporations-to-save-us-from-climate-change-86309

October 27, 2017 | Unregistered CommenterJonathan

Science communication link for Dan - the problem with documentaries:
https://www.scientificamerican.com/article/when-gmos-are-the-movie-star

October 27, 2017 | Unregistered CommenterJonathan

Jonathan,

This is epitomised by British entrepreneur Richard Branson, who has claimed that “our only hope to stop climate change is for industry to make money from it”.

Well yes. Quite. I've already explained how to do that. Nobody is interested, of course, because "stopping climate change" was never the real aim.

October 28, 2017 | Unregistered CommenterNiV

"stopping climate change" was never the real aim.

I just thank my lucky stars that we have NiV around to skeptasplain people's actual intent. If only I had his ability to divine others' motives.

October 28, 2017 | Unregistered CommenterJoshua

"If only I had his ability to divine others' motives."

They've made their motives pretty obvious.

But you're being far too modest! You can do it too!

Farmers aren't identifying as farmers or Republicans when they develop their views on GMOs, but as individuals who have a clear economic incentive. Thus, as the economic impact changes, changing orientation on the issue follows.

Their motive for changing views is their economic interest.

It is rather like Republicans for whom Republican identity is particularly important; with that being the case, they can easily flip in their views on an insurance mandate, to go from thinking that a mandate is a basic obligation of personal responsibility to thinking it is the epitome of "tyranny."

Their motive is their Republican identity.

Seem to me more likely that they would carve this situation out as an exception, in the same way that "knowing disbelief" works for the Pakistani doctor. They would find some way, although not really in a deliberate fashion, to not incorporate this situation into a larger ideological/political framework, where academic scientists are socialist lefties, and not to be trusted. IMO, people have little difficulty in rationalizing away logical conflicts so as to confirm previously established biases.

Their motive is to rationalise previously established biases.

The non-organic, GMO using farmers haven't become organic farmers, but non-organic, GMO using farmers who are suffering a direct and clear economic impact from the use of GMOs. They aren't going to stop using Roundup-resistant seeds because of opposition to GMOs, but because they aren't effective any longer.

Their motive for stopping use of Roundup-ready isn't GMO opposition, but that they're not effective.

The ability to deduce other people's motives is a common human characteristic, like walking upright or using language, and we all do it without really thinking about it, or sometimes even realising we're doing it. Given that you must know this, (you're not stupid), and that you do it yourself all the time, I can only deduce that there's some other meaning or intent behind your repeated sceptical comments on my ability to deduce motive. It's left as an "exercise for the student" to figure out what that is. (Or to figure out what *I'm* going to think it is... :-))

But the idea that there might be people who can't do it would make for an interesting cop show script...
Detective Inspector Cluebat [puts on sunglasses, significant pause]: "Cracking the case was easy once I'd shown he was the only one to have means, motive, and opportunity."
Sergeant Joshua: "If only I had his ability to divine others' motives."
:-)

October 29, 2017 | Unregistered CommenterNiV

more on dicamba:
https://www.reuters.com/article/us-usa-pesticides-complaints/monsanto-basf-weed-killers-strain-u-s-states-with-damage-complaints-idUSKBN1D14N0

Evidently, not a small problem.

November 1, 2017 | Unregistered CommenterJonathan

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>