follow CCP

Recent blog entries
« New paper: Ideology, Motivated Reasoning, and Cognitive Reflection: An Experimental Study | Main | "I endorse this message ...": video of lecture bits »
Wednesday
Nov282012

Some opinionated reflections on design of motivated reasoning experiments

Am tuning up a working paper--"Ideology, Cognitive Reflection, & Motivated Reasoning" -- that reports experiment results relating to "ideological symmetry" of motivated cognition as well as the relationship between motivated cognition & dual-process reasoning theories. Probably post in day or 2 or 3...

But here's a piece of the paper. It comes out of the methods section & addresses issues relating to design for motivated-reasoning experiments:

Testing for vulnerability to motivated reasoning is not straightforward. Simply asking individuals whether they would change their mind if shown contrary evidence, e.g., is inconclusive, because motivated reasoning is unconscious and thus not reliably observed or checked through introspection.

Nor is it satisfactory simply to measure reasoning dispositions or styles—whether by an objective performance test, such as CRT, or by a subjective self-evaluation one, like Need For Cognition. None of these tests has been validated as a predictor of motivated cognition. Indeed, early work in dual-process reasoning theory—research predating Kahneman’s “System 1”/“System 2” framework—supported the conclusion that motivated reasoning can bias higher-level or “systematic” information processing as well as lower-level, heuristic processing (Chen, Duckworth & Chaiken 1999; Giner-Sorolla & Chaiken 1997).

For these reasons, experimental study is more satisfactory. Nevertheless, proper experiment design can be complicated too.

One common design involves furnishing subjects who disagree on some issue (e.g., climate change or the death penalty) with balanced information and measuring whether they change their positions. The inference that they are engaged in motivated reasoning if they do not, however, is open to dispute. For one thing, the subjects might have previously encountered equivalent information outside the context of the experiment; being exposed to the same information again would not furnish them with reason to alter their positions no matter how open-mindedly they assessed it. Alternatively, subjects on both sides of the issue might have given open-minded consideration to the evidence—indeed, even given it exactly the same weight—but still failed to “change their minds” or to reach a shared conclusion because of how strongly opposed their prior beliefs were before the experiment.

Variants of this design that assess whether subjects of opposing ideologies change their positions when afforded with counter-attitudinal information on different issues are even more suspect. In those instances, it will not only be unclear whether subjects who stuck to their guns failed to afford the information open-minded consideration. It will also be unclear whether the counter-attitudinal information supplied respectively to the opposing sides was comparable in strength, thereby defeating any inference about the two groups’ relative disposition to engage in motivated reasoning.

It is possible to avoid these difficulties with an experimental manipulation aimed at changing the motivational stake subjects have in crediting a single piece of evidence. In Bayesian terms, the researcher should be measuring neither subjects' priors nor their posteriors but instead their likelihood ratios--to determine whether subjects will opportunistically adjust the significance they assign to information in a manner that promotes some interest or goal collateral to making an accurate judgment.

For example, subjects of diverse ideologies can be instructed to determine whether a demonstrator in a video—represented in one condition as an “anti-abortion protestor” and in another an “gay-rights protestor”—“blocked” or “screamed in the face” of a pedestrian trying to enter a building. If the perceptions of subjects vary in a manner that reflects the congeniality of the protestors’ message to the subjects’ ideologies, that would be convincing evidence of motivated reasoning. If the film of the protestors’ behavior is itself evidence relevant to some other issue—whether, say, the protestors broke a law against use of “coercion” or “intimidation”—then the impact of ideologically motivated reasoning will necessarily be biasing subjects’ assessment of that issue in directions congenial to their ideologies (Kahan, Hoffman, Evans, Braman & Rachlinski 2012).

In such a design, moreover, it is the subjects’ ideologies rather than their priors that are being used to predict their assessments of evidence conditional on the experimental manipulation. This element of the design bolsters the inference that the effect was generated by ideological motivation rather than a generic form of confirmation bias (Kahan, Jenkins-Smith & Braman 2011).

Such a design also enables straightforward testing of any hypothesized asymmetry in motivated reasoning among subjects of opposing ideologies. The corroboration of motivated reasoning in this design consists of the interaction between the experimental manipulation and subjects’ ideology: the direction or magnitude of the weight assigned to the evidence must be found to be conditional on the manipulation, which determined the congruence or noncongruence of the evidence with subjects’ ideologies. The hypothesis that this effect will be asymmetric—that it will, say, be greater among more conservative than liberal subjects, as RHB would assert—is equivalent to predicting that the size of the interaction will vary conditional on ideology. Such a hypothesis can be tested by examining whether a polynomial model—one that posits a “curved” rather than a “linear” effect—confirms that the magnitude of the interaction varies in the manner predicted and furnishes a bitter fit than a model that treats such an effect as uniform across ideology (Cohen, Cohen, West & Aiken 2003).

PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (2)

Just a clarification -- any given test of ideological asymmetry would be testing whether there is asymmetry in that particular context, and not more generally, correct? In other words, if you find that people on the left have roughly the same opinions about protesters whether their signs are pro- or anti- choice, while those on the right vary significantly depending on the content of the signs, this tells you about asymmetry in this particular domain. However, the possibility would still exist that there is an asymmetry on this issue, but if another issue were substituted the asymmetry would moderate or reverse.

Going back to the original "They Saw a Game" model -- to test ideological differences, it could actually be interesting to substitute a non-partisan target, like a sports affiliation. Any differences in motivated reasoning based on ideology could not be a function of an asymmetry in how important the issue was to partisans on either side (unless the left or right were bigger fans of the team in question, but that could be equated/controlled for).

November 28, 2012 | Unregistered CommenterDave Nussbaum

@Dave:
1. Yes on clarification in theory -- but is this really how things work? There is an issue that *one* group is passionate about but not other? Isn't it intrinsic to phenomenon of partisanship that it is reciprocated? Why become worked up if the other side *isn't*?! Now the only way to figure out answer to that question is, of coruse, to actually test it. Casual empiricism is fine for this too. What would you say these issues are? And also, when someone who really believes in symmetry not happily wager his or her kids' college savings if someone w/ a valid study design for ideologically motivated reasoning showed up & wanted to bet there'd be asymmetry? I can't think of a study I've ever worked on where I would think, if I saw asymmetry, "well, maybe that's one of those issues where one side gets much more worked up...." I'd say, shit-- I was wrong!

2. Almost same answer on this point about Saw a Game, actually. I think in a sense what you are thinking goes to the heart of it. As I understand you, you are saying, let's see if we can come up nonpartisan issues to test whether one or another ideological group is more disposed to motivated reasoning. Do that & I will be truly amazed & have only the consolation of being rescued from ignorance to console me for being so wrong. I really don't think what you describe is possible. It would be possible only if "motivated reasoning" were a sort of generic personality or cognitive reasoning trait--one that the poor bastards who suffer from it just lapse into whenever they are supposed to be considering evidence. I think ideologically motivated cognition is connected to the psycholgical need that people -- all of them -- have to be connected to groups. The only occasion, then, for ideologically motivated reasoning will be ones that have some sort of ideological significance. If you show me that being a Republican or a Democrat predicts that someone will show more motivated reasoning about something unrelated to being a republican or democrat-- & I'm pretty damn close to surrendering. Actually, Chris Mooney tried a study like that iin his book! It failed; & he honestly, in front of all his readers (it's right there in the book) scratched his head & said, huh, maybe I'm wrong... He should keep trying actually; one study not working has LR = 1.00000000000000001 on the hypothesis. But I do predict no experiment of that sort will ever work!

November 29, 2012 | Registered CommenterDan Kahan

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>