follow CCP

Recent blog entries
popular papers

Science Curiosity and Political Information Processing

What Is the "Science of Science Communication"?

Climate-Science Communication and the Measurement Problem

Ideology, Motivated Cognition, and Cognitive Reflection: An Experimental Study

'Ideology' or 'Situation Sense'? An Experimental Investigation of Motivated Reasoning and Professional Judgment

A Risky Science Communication Environment for Vaccines

Motivated Numeracy and Enlightened Self-Government

Making Climate Science Communication Evidence-based—All the Way Down 

Neutral Principles, Motivated Cognition, and Some Problems for Constitutional Law 

Cultural Cognition of Scientific Consensus
 

The Tragedy of the Risk-Perception Commons: Science Literacy and Climate Change

"They Saw a Protest": Cognitive Illiberalism and the Speech-Conduct Distinction 

Geoengineering and the Science Communication Environment: a Cross-Cultural Experiment

Fixing the Communications Failure

Why We Are Poles Apart on Climate Change

The Cognitively Illiberal State 

Who Fears the HPV Vaccine, Who Doesn't, and Why? An Experimental Study

Cultural Cognition of the Risks and Benefits of Nanotechnology

Whose Eyes Are You Going to Believe? An Empirical Examination of Scott v. Harris

Cultural Cognition and Public Policy

Culture, Cognition, and Consent: Who Perceives What, and Why, in "Acquaintance Rape" Cases

Culture and Identity-Protective Cognition: Explaining the White Male Effect

Fear of Democracy: A Cultural Evaluation of Sunstein on Risk

Cultural Cognition as a Conception of the Cultural Theory of Risk

« Lack of discriminant validity saga #9312 (or "Let's just make this blog into a blog on Gelman's blog!," episode #612) | Main | Cross-cultural cultural cognition's latest conquest: Slovakia! »
Wednesday
Sep272017

Is fit-statistic anarchy the answer to tyranny of the p-value?

Now we are getting somewhere!

But note how much weight this proposal places on (or how much confidence it expresses in) the inferential literacy of referees. If we cut social science loose from the p-value in favor of the gestalt judgment of reviewers and editors, what's to prevent a dictatorship of confirmation bias?

PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (23)

Dan:

You write, "If we cut social science loose from the p-value in favor of the gestalt judgment of reviewers and editors . . . " But, for better or worse, journals are already using subjective gestalt judgment (not to mention flat-out politics) when deciding what papers to publish. The current rule for journal editors is: (a) require p less than 0.05, (b) do whatever the hell you want. I'd like to remove step (a) and require some sort of justification for step (b). Currently, I fear that the existence of step (a) makes the Susan Fiskes of the world think they don't need to do step (b) at all! I have no problem with journal editors using null hypothesis significance tests as part of their decisions. I think the lexicographic rule is ridiculous.

September 27, 2017 | Unregistered CommenterAndrew Gelman

@Andrew--

Point conceded. And as I think you know, I do support your program over the others. In my mind, at least, an "anarchy of reason" is very attractive in theory.

Still, I do think that at a minimum there will be a bloody transition as referees & editors whose style of reviewing doesn't include genuine engagement with question "what can we infer & how strongly" struggle to make sense of their new mandate to actually *think* about what a study adds to the weight of the evidence in favor of the study hypothesis in relation to its rivals. It might be that some "weight of the evidence" statistics, used heuristically & not as screens, could help them to get what this form of reasoning entails (the p-value clearly didn't).

I suspect the transition won't be completed until the next generation of social scientists or even the one after that matures.

Another thought: is there a journal that you think is already using something close to your proposed review process? HOw is it working out for them?

September 27, 2017 | Registered CommenterDan Kahan

link drop "Challenging people’s political views and values makes them think even harder and produce better arguments to defend themselves":
http://bit.ly/2jNT9Ng

We know from many studies that motivated reasoners push back again information that conflicts with their original beliefs, but we did not know what this did to the cognitive processes that are engaged. We designed an experiment where we varied the direction of the information (incongruent vs. congruent relative to the participant’s own ideological leanings) shared about a political candidate’s policy positions. We find that while exposure to this incongruent information often does little to change the minds of motivated reasoners, they do think harder about the issue, generate more responses to it when asked, and that those responses are more complex than they otherwise would be.

That, plus something I happened to read the other day here:

https://www.edge.org/conversation/tania_lombrozo-learning-by-thinking

seems like it could provide some insight into the mechanism of backfire. The backfirer first constructs a superior counterargument, then learns something new from that counterargument.

September 27, 2017 | Unregistered CommenterJonathan

Also from the same source, evidence of asymmetry in news viewing behavior:
http://bit.ly/2nFHcqo

Overall, subjects did not engage in selective exposure – except those who were partial to the Republican candidate and who were in the high-choice, negative ads condition...

September 27, 2017 | Unregistered CommenterJonathan

@Jonathan -- sounds like someone discovering confabulation & concluding it can be treated as a useful exercise in critical reasoning...

September 27, 2017 | Registered CommenterDan Kahan

Dan,

Does that mean your opinion is that more complex arguments arrived at after cognitive effort are necessarily evidence of confabulation, or just that they didn't sufficiently remove the possibility that they are measuring confabulation instead of after-the-fact novel and rational cognition?

BTW - I can't see their paper due to paywall, so all I have is the blog entry to go on. So, I can't see how they operationalize "integrative complexity" etc.

I am reading this now: http://stuttercut.org/detective/lombrozo.pdf

September 27, 2017 | Unregistered CommenterJonathan

@Jonathan-- thanks. 2nd I think is closer. There are some great studies that expose confabulation & that sound a lot like the design of this one

September 27, 2017 | Registered CommenterDan Kahan

Dan,

They do find that politically more sophisticated subjects do this more. If it's confabulation, why does it correlate with politically sophistication? I would expect the less sophisticated subjects to confabulate more.

But - yes - it would be nice to know if their test measures accuracy of the more complex explanations, instead of just their complexity.

However - even if it is confabulation - might the mechanism Lombrozo points out be at work regardless? Would seem to be even worse if it could work despite the explanations being confabulated. One could backfire themselves into Wonderland.

September 27, 2017 | Unregistered CommenterJonathan

From http://stuttercut.org/detective/lombrozo.pdf:

Whether these effects of explanation reflect a naı̈ve epistemology or a legitimate and necessary use of prior knowledge is controversial. But because explanations embody prior beliefs, they have an undisputed danger: when generated from true beliefs, explanations provide an invaluable source of constraint; when generated from false beliefs, explanations can perpetuate inaccuracy.

September 27, 2017 | Unregistered CommenterJonathan

That need for evidence paper I cited back on 9/19, https://doi.org/10.1371/journal.pone.0184733, just got some popsci treatment:
https://phys.org/news/2017-09-americans-gut-true.html

which contains a nice shout-out to Dan's work:

We aren't the only ones who have observed a pattern like this. Another recent study shows that people who exhibit higher scientific curiosity also tend to adopt more accurate beliefs about politically charged science topics, such as fracking and global warming.

So, even though it's in PLOS, should we not at least respect it some?

September 28, 2017 | Unregistered CommenterJonathan

About that demonic "adorable" cat - maybe essentialism there too?:
https://phys.org/news/2017-09-science-people-animals-couldnt.html

September 29, 2017 | Unregistered CommenterJonathan

"shows that people who exhibit higher scientific curiosity also tend to adopt more accurate beliefs"

"Accurate"? That's been shown?

September 29, 2017 | Unregistered CommenterNiV

NiV,

On "accurate" - maybe a better way to say it is that SC folks display what appears to be Bayesian adjustments in belief without requiring extra variables that flip the valence of those adjustments.

It would be interesting to see if SC folks can be made to acquire such extra variables. Are they averse to such things? Are they unable to handle the extra complexity? Are they endlessly gullible? Are they just (un)lucky?

One hypothesis I have is that they underweigh their own priors. Which would make them gullible.

However, an alternate hypothesis I have is that they underweigh (with respect to others) the probability of malicious agency. Which would make them gullible, but only in cases where the conspiracy is true. As with Hanlon's Razor.

September 29, 2017 | Unregistered CommenterJonathan

"SC folks display what appears to be Bayesian adjustments in belief without requiring extra variables that flip the valence of those adjustments"

All it says is that they make belief adjustments using variables that don't include the political one. That says nothing at all about whether there are any other extra valence-flipping variables involved.

"However, an alternate hypothesis I have is that they underweigh (with respect to others) the probability of malicious agency. Which would make them gullible, but only in cases where the conspiracy is true."

So why would they think fracking was dangerous if that was the case? The main arguments against it seem to be wrapped around a conspiracy on the part of mining companies to make profit at the expense of public safety. If you don't distrust the mining engineers, why would you distrust what they say about safety?

I don't understand what mechanism you have in mind.

September 30, 2017 | Unregistered CommenterNiV

"So why would they think fracking was dangerous if that was the case? The main arguments against it seem to be wrapped around a conspiracy on the part of mining companies to make profit at the expense of public safety. If you don't distrust the mining engineers, why would you distrust what they say about safety?"

We discussed that if they are getting most of their exposure from popsci, then that's the message they see and would respond to. I don't think it is a conspiracy message - it satisfy's Hanlon's Razor - the mining engineers could just not know what impact they are having, and/or be in denial.

September 30, 2017 | Unregistered CommenterJonathan

it satisfy's - wow - I invented the 3rd person possessive verb tense ;)

September 30, 2017 | Unregistered CommenterJonathan

"We discussed that if they are getting most of their exposure from popsci, then that's the message they see and would respond to."

Ah. You meant "alternative" in the sense of there being multiple effects/mechanisms of which this is another one, rather than an alternative explanation for a single effect. I understand now.

"wow - I invented the 3rd person possessive verb tense"

Nice! :-)

September 30, 2017 | Unregistered CommenterNiV

How 'bout that - Nate Silver used Hanlon's Razor at about the same time I did:

https://fivethirtyeight.com/features/the-media-needs-to-stop-rationalizing-president-trumps-behavior/

Must be a conspiracy... ;)

September 30, 2017 | Unregistered CommenterJonathan

"How 'bout that - Nate Silver used Hanlon's Razor at about the same time I did:"

It's an entertaining article! I tend to agree with Nate that people are over-analysing. (And in particular, analysing with left-wing assumptions and interpretations, which nearly always miss the mark). To answer his question, I think it's actually both. The contents of the tweets are simply what Trump thinks, as an immediate gut reaction and without any deep analysis, spin, or manipulation. The strategy of putting out such tweets is a deliberate strategy - presenting the image of an honest, straight-forward straight-talking guy, just like all your friends down at the bar. He's fallible and sometimes shoots his mouth off and puts his foot in it, but that's a breath of fresh air after all the polished, politically-correct, mealy-mouthed, carefully tailored "statements" that most politicians put out. People on his side like that, and don't take what he says too seriously.

And most of the examples where he got into "trouble" (from the perspective of his opponents) were cases where various people entered the public spotlight to attack him, his policies, or his political following, often in fairly extreme terms, and he simply attacked back. From the left-wing perspective, that's not allowed. Either the attacks on him were "justified" and therefore don't constitute a problem, or were from cultural "sacred cows" - people or groups that you are not allowed to criticise in polite (i.e. left-wing) circles.

I think that ignoring those rules is also a deliberate strategy. The media seeks to control politicians by controlling their language - when people say of certain things "you can't say that!", you're into an Orwellian world of newspeak. So by saying it, and then carrying on saying it after the inevitable firestorm, he's effectively saying: "Yes, we can say that! These cows are not sacred. You can no longer control what we say." He's moving the "Overton window" on free speech back. He's (maybe, just about) got the power as President to get away with it.

In my view, it would probably be a far more significant achievement of his Presidency, if he can do it, than any of the stuff about repealing Obamacare or building walls. But as for the tweets themselves, they're deliberately not calculated political manipulations - they're what he really thinks.

On Puerto Rico, he was just annoyed that after going to a lot of effort and spending a lot of money trying to help someone, instead of gratitude he got a lot of political attacks on his Presidency and the competence of his people. He found that rude and annoying, and said so. On the NFL thing, a bunch of left-wingers decided to hijack a common patriotic ceremony to gain maximum publicity for a divisive political protest, that in effect criticised all the whites in America, and the ones who follow Trump in particular. He told them off for it. On Charlottesville, a set of Communist thugs launched an illegal and violent attack on a legal and authorised free speech rally (admittedly including groups with some fairly unpleasant views) to shut down their free speech rights, and a lone nutter with anger-management issues attacking back, with lethal consequences. His initial view, that there was fault on both sides, might be politically incorrect, but was factually entirely correct. And his base know it, and love him because he was willing to say it.

Itll be fascinating to see what happens next! We live in interesting times...

October 1, 2017 | Unregistered CommenterNiV

NiV - if I may, a quick correction: it was not the Charlottesville driver who caused the "lethal consequences". The parents of Heather Heyer cite the coroner's report, which specifies cause of death was heart attack with no sign of trauma caused by a car accident.
https://www.nbcnews.com/video/heather-heyer-s-mom-delivers-message-about-karma-to-white-nationalists-1028340803735

October 1, 2017 | Unregistered CommenterEcoute Sauvage

Consistent with the above conspiracist narrative, whereby lizard aliens are controlling my brain along with those of widely followed pollsters, Pew has this today:

http://www.journalism.org/2017/10/02/covering-president-trump-in-a-polarized-media-environment/

Well, OK - so not exactly the same thing (no direct Hanlon's Razor usage, and more in line with this blog's topic) - but that's only because they don't want to be easily detected...

October 2, 2017 | Unregistered CommenterJonathan

"Right wing news sources are more likely to agree with Trump" shock! :-)

October 3, 2017 | Unregistered CommenterNiV

Jonathan -

Thanks for that "learning by thinking" link. I've been traveling a lot and haven't had time to digest it completely yet, but from what I've seen so far I found it very interesting....


As an educator, I find the connection between "explaining" and learning to be quite foundational. Some of my views on that issue are an outgrowth of my training in Montessori education - the first formal study I did of educational methodology - and more specifically the Montessori concept of a "Three period lesson." I look at the "three period lesson" as a kind of analogy that can be extended for a more generalized epistemological model, where learning involves "explaining" in a broad sense - where developing conceptual "schema" or "neural networks" may often be crucial component to "meta-cognitive" understanding, and where "explaining" is a way to realize those "schema" or "neural networks."

http://montessoritraining.blogspot.com/2007/09/montessori-method-three-period-lesson.html

After I've looked at it more I might comment on that...but in the meantime, more closely related to Dan's blog...I think that there is an important connection to "motivated reasoning" and "cultural cognition."

IMO, a key factor in addressing those phenomena is "perspective taking." IOW, IMO, to really understand someone else's view, one needs to learn about it, and it in order to learn about it, it becomes important be able to "explain" it in a truly accurate (not fallacious) manner. IMO, the ability to learn about and "explain" someone else's view is largely a function of intent - I think that we are all heavily motivated by identity defense mechanisms, but that maybe the influence of those mechanisms can be mitigated by a variety of tools. The scientific method is one of those tools, and maybe an explicit focus on differentiating positions from interests, and on truly leaning about and "explaining" alternative perspectives (where positions can be identified independent of the "interests" of the learner) is another such tool.

October 5, 2017 | Unregistered CommenterJoshua

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>