follow CCP

Recent blog entries
popular papers

Science Curiosity and Political Information Processing

What Is the "Science of Science Communication"?

Climate-Science Communication and the Measurement Problem

Ideology, Motivated Cognition, and Cognitive Reflection: An Experimental Study

'Ideology' or 'Situation Sense'? An Experimental Investigation of Motivated Reasoning and Professional Judgment

A Risky Science Communication Environment for Vaccines

Motivated Numeracy and Enlightened Self-Government

Making Climate Science Communication Evidence-based—All the Way Down 

Neutral Principles, Motivated Cognition, and Some Problems for Constitutional Law 

Cultural Cognition of Scientific Consensus

The Tragedy of the Risk-Perception Commons: Science Literacy and Climate Change

"They Saw a Protest": Cognitive Illiberalism and the Speech-Conduct Distinction 

Geoengineering and the Science Communication Environment: a Cross-Cultural Experiment

Fixing the Communications Failure

Why We Are Poles Apart on Climate Change

The Cognitively Illiberal State 

Who Fears the HPV Vaccine, Who Doesn't, and Why? An Experimental Study

Cultural Cognition of the Risks and Benefits of Nanotechnology

Whose Eyes Are You Going to Believe? An Empirical Examination of Scott v. Harris

Cultural Cognition and Public Policy

Culture, Cognition, and Consent: Who Perceives What, and Why, in "Acquaintance Rape" Cases

Culture and Identity-Protective Cognition: Explaining the White Male Effect

Fear of Democracy: A Cultural Evaluation of Sunstein on Risk

Cultural Cognition as a Conception of the Cultural Theory of Risk

« More discussion of SPBMC ... | Main | Unconfounding knowledge from cultural identity--as big a challenge for measuring the climate-science literacy of middle schoolers as grown ups »

What comes first--misinformation or the motivation to believe it? Some reflections on study design

 Unlike our myriad competitors, the CCP blog now & again gets genuine experts to come in & address complicated stuff that these commentators actually know something about. We've been criticized for this, but sometimes I'm too busy to write myself & have no choice.  Anyway, the following is an expert guest post from a commentator making his second "guest" appearance.  Kevin Arceneaux's last essay, Partisan Media Are Not Destroying America (while subsequently disproven by events), was the most popular post ever on this blog, being read by an estimated 19.3 billion readers.  Now he's back to address related issues on study design and causal inference in assessments of the impact of partisan news coverage on public opinion. Arceneaux is the author, with Martin Johnsonof the acclaimed Changing Minds, Changing Channels (Univ. Chicago Press 2013).

Kevin Acreneaux:

News and entertainment media have the dubious distinction of serving as both a whipping boy and a potential savior.  They are often treated as the source of many social ills.  Beauty magazines perpetuate unhealthy body images; political advertisements inveigle; partisan news programs mislead and confuse (especially if we happen to disagree with them).  We also imagine that their power can be put to good use.  Media can serve as a catalyst for positive change, however defined.

As seductive as these narratives are, the problem is that they are difficult to evaluate empirically.  How could this be?  In modern advanced democracies, like the United States, we are surrounded by media.  Traditional forms of mass media – newspapers, magazines, radio, television – operate along side newer forms of interactive media on the Internet.  Can’t we just observe how people respond to all these forms of news media?

We can certainly observe what people consume and what they do, but we can’t always infer the effects of media consumption on their behavior.  Observational research is inherently beset by many threats to causal inference, and the current media environment only makes it worse.  The study of media effects could easily be the poster child for the dictum that correlation does not necessarily imply causation.

The biggest hurdle to divining the effects of media from observational research is the fact that people, by and large, choose what to consume.  For instance, we know that conservatives say that they consume conservative media at higher rates than other Americans.  But because conservatives are consciously choosing to view conservative media and construct conservative networks on social media, it is difficult to sort out how much of their conservatism come from their personal predispositions and how much of it comes from the messages that they encounter.

To muddy the waters further, the ability to select among news and entertainment alternatives creates incentives for media producers to fashion content that will appeal to particular segments of the population.  To take a current day example, Fox News has received its fair share of criticism for how it has covered the threats posed by the Islamic State of Iraq and Syria (ISIS) and the Ebola epidemic.  It is easy to accuse Fox News and other media outlets of whipping up hysteria, but we must also entertain the possibility that they are just giving their viewers what they think they want.  People who are chronically worried about threats need a place to turn to for answers and outlets like Fox News are happy to oblige.

From the standpoint of causal inference, it is difficult to pinpoint the effects of Fox News, because people who aren’t predisposed to be worried about Ebola are happily consuming different media content and if, for some reason, they happened across Fox News coverage of the Ebola epidemic, they may find in more amusing than worrying.

The problem here is so bad that statisticians refer to it as the Fundamental Problem of Causal Inference.  In a nutshell, the only way we can really now the effect of media content to observe two states of the world: one where the person consumes it and one where the person does not.  Of course, that’s impossible.  The only way forward for intrepid researchers is to figure out how to construct a comparable group of people.  For example, people who are just like the ones who watch Fox News but who do not.  That is easier said than done.

Fancy statistical models that try to address the problem by accounting for people’s viewing preferences (i.e., “control variables”) can actually cause more harm than good.  At the very least, this approach rests on the strong assumption that one has accounted for everything, and we can never know if we have.

Another approach that fares a little better is observing the same people overtime.  In doing so, we can get a before and after take on their behavior.  Yet this approach also makes strong assumptions, too, and as Tobias Konitzer points out in a recent conference paper even if we can make those assumptions, we need lots of observations across time.  Panel surveys are rare and long-running panel surveys, even rarer.

Many scholars, including myself, have pointed to randomized experiments as a way forward.  Experiments use random assignment to construct comparable groups of individuals.  Some people are exposed to media content while others are not.  Because people were assigned to groups at random, we know that they should have similar tastes and similar responses.  So, if we see one group behaving differently than another, we can more credibly infer that the difference was caused by the treatment that we administered.

While randomized experiments do allow us to say with more confidence that exposure to, say, partisan news content causes people to do X, Y, or Z, it is also not a panacea.  For one, experimentalists generally construct comparable groups and then ask people to do things that they would not always do or expose them to things that they may not have encountered but for the intervention of the researcher.  Consequently, we cannot be certain that they would not behave differently if the treatment had unfolded through natural means.  Field experiments and natural “experiments” (i.e., observational designs that have plausibly exogenous treatments) do better on this score, but they are often difficult to employ.

Another limitation is that experiments are not particularly good at measuring the cumulative effect of media exposure, but rather at pinpointing the effect of a particular intervention.  So, the upshot here should be familiar: nothing is perfect and there is no silver bullet. It may be trite, but it is true.  We learn the most through the triangulation of methods. 

PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (6)

In our understanding of how brains work and learn, everything is a cumulative effect. In reading media today and reacting to it, we unconsciously channel echoes from long ago, half remembered memories of parents, churches, cultures, and even the bully in the seventh grade. It is unclear how to incorporate these experiences into a well designed experiment.

October 28, 2014 | Unregistered CommenterEric Fairfield

"It is easy to accuse Fox News and other media outlets of whipping up hysteria, but we must also entertain the possibility that they are just giving their viewers what they think they want. People who are chronically worried about threats need a place to turn to for answers and outlets like Fox News are happy to oblige."

And as we always have to remind ourselves, whether it is considered a threat or not is culturally determined, and we are all subject to our own culture, so the judgement of the other news channels that it is mere hysteria may likewise be "just giving viewers what they think they want" - Democrat viewers in this case.

People who feel a need to be chronically worried about threats need only turn on the news to be told about global warming, overpopulation, pollution, extinction, resources running out, the seas filling with plastic, nuclear waste, polar bears and walruses and frogs and bees and snails and trees, ... My favourite was the impending invasion of vampire moths, closely followed by Ted Turner's 'cannibal' scenario.

People who are chronically worried about threats? I think you may need to look in a mirror!

As is all too usual with 'experts' in the predominantly liberal social sciences, the implications of Dan's symmetry result are being ignored, and the research is working from a set of assumptions founded in a particular cultural worldview. Why ask only whether it is Fox News that makes people conservative - why not ask whether it is CNN and MSNBC that are making people liberal? Isn't this more like the 'Republican Brain" hypothesis?

If you really want to know, though, the people to study would be those young adults who have grown up in a household where the parents are of one tribe, and only watch TV conforming to their beliefs, and yet the children have somehow grown up to be of the other tribe. When did they first realise that they "weren't like the others"? Was it something they read, or someone they met, or did they once stop up late one Halloween night and watch Rush Limbaugh during the witching hour for a dare, and now levitate and vomit pea soup?

No, but seriously - how would you go about applying the symmetry principle to modify this line of research (or social sciences generally) in order to reduce the potential for subconscious bias? Do you agree that it is a useful thing to do? What does 'motivated reasoning' theory predict you'll do?

October 28, 2014 | Unregistered CommenterNiV

Gonna repost my comment from Kevin's last guest post (w/ a slight edit):

It seems to me that polarized media often get blamed for polarizing the public - but what is lost is that to the extent that the media are polarized, it is because media polarization reflects the existing polarization of the audience.

I also think it is fascinating that both "sides" in many debates, and notably the climate wars , are absolutely convinced that the support for the opposing views is attributable to what the "MSM" do or don't do. Why is that fascinating? Because each side is absolutely convinced of the existence of an exact opposite effect; i.e., "skeptics" are convinced that MSM shills are the reason why so much of the public is concerned about climate change and "realists" are convinced that MSM shills are the reason why so much of the public is not concerned about climate change. Of course, the exact same dynamic takes place with how partisans view the effect of the media w/ so many other issues that are politically polarized as well.

The "MSM Is to blame" narrative fits, precisely, the need for being the "victim" - which in turn, IMO, fits precisely the basic mechanism of cultural cognition; i.e., being a victim of the other side is just about the best way for someone to feel vindicated in their identification. It fits right in line with the identity-protective and identity-defensive behaviors that are so inextricably linked to motivated reasoning.

October 29, 2014 | Unregistered CommenterJoshua

Dear Dr. Kahan:

Do you ever write about the war on IQ science? People like James D. Watson, Jason Richwine, and (to a partial extent) Larry Summers lose their jobs just for publicly stating what psychometricians know from an immense amount of research over the decades.

November 3, 2014 | Unregistered CommenterSteve Sailer

Dear @Steve:

No, I haven't. I agree this is interesting issue. (Didn't Chris Mooney just do something on this? I'm behind)

PS Kahan

November 4, 2014 | Registered CommenterDan Kahan

==> "Do you ever write about the war on IQ science? "

The "war on IQ science? Why does different people having different views on IQ science translate into a "war on IQ science." Is it anything like the "war on Christmas?"

==> "Larry Summers lose their jobs just for publicly stating what psychometricians know from an immense amount of research over the decades."

Larry Summers:

My January remarks substantially understated the impact of socialization and discrimination, including implicit attitudes — patterns of thought to which all of us are unconsciously subject,” he said. “The issue of gender difference is far more complex than comes through in my comments, and my remarks about variability went beyond what the research has established.”

November 4, 2014 | Unregistered CommenterJoshua

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>