follow CCP

Recent blog entries
popular papers

Science Curiosity and Political Information Processing

What Is the "Science of Science Communication"?

Climate-Science Communication and the Measurement Problem

Ideology, Motivated Cognition, and Cognitive Reflection: An Experimental Study

'Ideology' or 'Situation Sense'? An Experimental Investigation of Motivated Reasoning and Professional Judgment

A Risky Science Communication Environment for Vaccines

Motivated Numeracy and Enlightened Self-Government

Making Climate Science Communication Evidence-based—All the Way Down 

Neutral Principles, Motivated Cognition, and Some Problems for Constitutional Law 

Cultural Cognition of Scientific Consensus
 

The Tragedy of the Risk-Perception Commons: Science Literacy and Climate Change

"They Saw a Protest": Cognitive Illiberalism and the Speech-Conduct Distinction 

Geoengineering and the Science Communication Environment: a Cross-Cultural Experiment

Fixing the Communications Failure

Why We Are Poles Apart on Climate Change

The Cognitively Illiberal State 

Who Fears the HPV Vaccine, Who Doesn't, and Why? An Experimental Study

Cultural Cognition of the Risks and Benefits of Nanotechnology

Whose Eyes Are You Going to Believe? An Empirical Examination of Scott v. Harris

Cultural Cognition and Public Policy

Culture, Cognition, and Consent: Who Perceives What, and Why, in "Acquaintance Rape" Cases

Culture and Identity-Protective Cognition: Explaining the White Male Effect

Fear of Democracy: A Cultural Evaluation of Sunstein on Risk

Cultural Cognition as a Conception of the Cultural Theory of Risk

« Which provides more information--probability density distribution or scatter plot with locally weighted regression line? Which is easiest to comprehend? | Main | Travel journal entries »
Sunday
Jun182017

Weekend update: bias & unreliability in peer review

Here's a classic on biases of peer reviewers:

The experiment results are pretty cool. But one conclusion I view as even more alarming than the reviewers' confirmation bias was Mahoney's finding that there's very little "inter-rater reliability" among peer reviewers, something that I think people who regularly review for & publish in decision-science journals get to observe first hand. Another instance of the self-measurement paradox, this finding ought to make those in the business pretty nervous about the power of pre-publication peer review to winnow the wheat from the chaff.

One thing  I hadn't had occasion to notice or think about previously is the citation history:

 

Two possibilities occur to me.  One is that electronic search has made it easier for authors to find this paper.  

The other is that the topic of confirmation bias among reviewers has become much more topical as challenges to the integrity & effectiveness of pre-publication peer review have become more common.  

One way to sort those out would be to see if articles of other sorts--ones unrelated to cognitive bias of scientists as reviewers -- display this same pattern, in which case possibility one would seem the stronger explanation. One could also look to see if other older articles on reviewer confirmatory bias (e.g., Koehler 1993) have recent-citation upsurge that goes against usual steady decline in citation of papers.

For what it is worth, with virtually no investigation on my part, I'm willing to bet, oh, $10,000 that papers inquiring into reviewer bias have increased a lot in last few years in response to anxiety that there are group-conformity influences in the study of politically charged topics (e.g., climate change).

But if anyone else wants to share some back-of-envelope data analysis here, go for it.

Be interesting too to get some color on the sort of thinker Mahoney, who died at age 60 in 2006, was.

BTW, one of the funniest/scariest things in Mahoney's paper is his quotation of this bit of feedback on the study method, which in part involved having some reviewers read papers w/ just methods & w/o results.

"Personally, I don't see how anyone can write the Introduction and Method without first having the Results" (p. 172), the subject stated. 

It's less surprising than that an empirical researcher would feel that way than that he or she would so unself-consciously admit to a style of study presentation that is based on post-hoc story-telling.

References

Koehler, J.J. The Influence of Prior Beliefs on Scientific Judgments of Evidence Quality. Org. Behavior & Human Decision Processes 56, 28-55 (1993).


Mahoney, M. & DeMonbreun, B. Psychology of the scientist: An analysis of problem-solving bias. Cogn Ther Res 1, 229-238 (1977).  

PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (27)

Dan -

Do you always complete your introductions before you have results from your experiments?

June 18, 2017 | Unregistered CommenterJoshua

@Joshua-- I don't write introductions before I have results. But I do write up a memo on what hypotheses are, on why & what observations in the study will support inferences on the hypotheses, and on what the analytic strategy will be. In collaborative work, those are major things to work through before we start to collect data; if solo, I show to other researchers to see if they agree design is suited to objective before I start data collection.

June 18, 2017 | Registered CommenterDan Kahan

"But one conclusion I view as even more alarming than the reviewers' confirmation bias was Mahoney's finding that there's very little "inter-rater reliability" among peer reviewers, something that I think people who regularly review for & publish in decision-science journals get to observe first hand. Another instance of the self-measurement paradox, this finding ought to make those in the business pretty nervous about the power of pre-publication peer review to winnow the wheat from the chaff."

It's a point I've been arguing for a number of years. The function of pre-publication peer review is a lot more limited than people think it is. "Peer review" in the scientific sense is done post-publication, when other scientists reading the journal try to replicate, debunk, or extend the result. The only function of pre-publication review is to collate papers by topic and filter out papers plausible enough for it to be worth other scientists' time to review. Does the paper present enough evidence to be taken seriously, enough detail to attempt replication, enough information on the precautions taken against error to know where to start looking for holes, and is it novel, significant, and interesting to the research community the journal serves.

It frequently consists of a few hours of unpaid time from a couple of experts in the subject. That's a long way from sufficient to thoroughly test a scientific claim. The amazing bit to me is not that half of all peer-reviewed journal articles should turn out to be wrong, but that any scientist familiar with the process could possibly have thought that they weren't.

The classic 'gold standard' of 'accepted science' was never the peer-reviewed journals, it was supposedly the textbooks.

But because it's so funny, I'll link again to How to Publish a Scientific Comment in 123 Easy Steps.

June 18, 2017 | Unregistered CommenterNiV

A lot of the science that manages it out to the public arena as "Scientific Progress" may be rooted in academic basic research, but owes much to industrial R&D, and is often not effectively followed up by governmental regulation or further academic or other independent research into impacts. And much industrial work is carried out as corporate restricted information.

All of this is heavily influenced by the tendency of scientists to act as cozy little clubs. The American Chemical Society for example.

See: Inventor hero was a one-man environmental disaster

Thomas Midgley is a former ACS President (1944).

Tetraethyl lead and Freon. Patents were key considerations. Almost all of the initial (but decades long) research into health effects were beholden to the corporations involved.

"Midgley had "more impact on the atmosphere than any other organism in Earth's history""

https://www.newscientist.com/article/mg23431290-800-inventor-hero-was-a-oneman-environmental-disaster/

June 18, 2017 | Unregistered CommenterGaythia Weis

"Tetraethyl lead and Freon. Patents were key considerations. Almost all of the initial (but decades long) research into health effects were beholden to the corporations involved."

Yes, of course. But don't forget the other side of the trade-off. Tetra-ethyl lead in fuel made cheaper motoring possible, lifting millions out of poverty sooner, as they were no longer tied to jobs they could walk to; and Freon made cheap non-toxic refrigeration possible (it was a considerable safety improvement over the ammonia that was previously used), with vast health effects in the reduction of food poisoning, the transport of fresh food long distances bringing more, better, and cheaper food into reach of the poor, along with a host of other industrial applications that made life better and safer for everyone. Patents are valueless if nobody wants or needs the product - the money made is a direct measure of their positive contribution to the good of society. Every new and beneficial technology for lifting people out of poverty and suffering has its own set of costs and risks.

Industries should, of course, be held accountable for the proven damage they do, even if unintentional. That's precisely what should motivate them to do the safety research before releasing a new product on the market - for self-protection if nothing else. (Industrial scientists are human, and just as moral as anyone else.) But we should not entirely discount the good they do in the process as well, when setting policy on regulation.

Single-issue fanatics always highlight one side of the trade-off and entirely neglect or deny the other. It's bad science, and leads to bad policy.

June 18, 2017 | Unregistered CommenterNiV

link drop:
https://ssrn.com/abstract=2973892

June 18, 2017 | Unregistered CommenterJonathan

NiV, you may not have read the article, since it was behind a paywall, but the point was that there WERE alternatives to tetraethyl lead, TEL, ones that Midgley discovered (143 of them) and thus his employer, GM was or should have been aware of. The difference was that TEL was patentable, and cheaper than most alternatives. (Given that, of course, the effects of lead poisoning were not factored in.)

There were warnings by outside experts that TEL was highly poisonous. The article states that workers at the TEL factory suffered bouts of violent paranoia and were hauled away in straitjackets. Midgley himself was treated for lead poisoning. Outside experts tried to raise the alarm. For example the article notes that it was in 1923 that William Clark at the US Public Health Service warned that lead oxide dust would build up along busy roads. And the year after that that a Yale toxicologist warned that "the development of lead poisoning will come on so insidiously that leaded gasoline will be in nearly universal use...before the public and the government become aware of it". TEL was marketed as "Ethyl" omitting the word lead. With mounds of advertising. And many industry funding studies "proving" safety.

This was also the era in which GM bought out Los Angeles electric "red car" system and ran it into the ground. Thereby facilitating the freeway driven suburban sprall LA is known for. And a need for GM cars.

Business is in the business of making short term profits and leveraging those profits forward into market domination and even more profits.

There are alternative pathways to Scientific Progress. Ones with better balance.

June 18, 2017 | Unregistered CommenterGaythia Weis

The way that US corporations do not manage risk, but rather duck and pass the consequences on:

Extraction Oil, founded in 2012, has assumed ownership of Colorado oil and gas mineral rights once held by larger firms with greater net worth. They are frequently operating in densely populated areas where former farm and ranch land is now overrun by suburban development.

What they have to say about how they manage risk:

"A substantial portion of our reserves are located in urban portions of the DJ Basin, which could disproportionately expose us to operational and regulatory risk in that area. "

"Does the driller have insurance for unforeseen events? "We maintain insurance against some risks associated with above or underground contamination that may occur as a result of our exploration and production activities. However, this insurance is limited to activities at the well site and there can be no assurance that this insurance will continue to be commercially available or that this insurance will be available at premium levels that justify its purchase by us. The occurrence of a significant event that is not fully insured or indemnified against could have a materially adverse effect on our financial condition and operations. Further, we have no coverage for gradual, long-term pollution events." Makes sense: the taxpayers cover those costs via superfund cleanup. And in another section: ". . .pollution and environmental risks generally are not fully insurable.""

http://www.dailycamera.com/guest-opinions/ci_31070997/merrily-mazza-lies-communities-truth-investors

June 19, 2017 | Unregistered CommenterGaythia Weis

Who was looking for a separation of mainstream Republicans from Tea Party Republicans on climate change?:
http://www.sciencedirect.com/science/article/pii/S2214629616302973

"The Tea Party political movement began in the United States in 2009, driven by opposition to the presidency of Barack Obama. Prior research has shown that Tea Party Republicans differ from mainstream Republicans on a range of attitudes, such as belief in climate change and attitudes towards immigrants. However, it is unknown if Tea Party affiliation is consequential for energy policy. In this paper, we use an array of dependent variables related to a number of different energy policy options and find substantial differences between Tea Party Republicans and mainstream Republicans. In particular, Tea Party Republicans are much more supportive of increasing energy supply via fossil fuel extraction and less supportive of regulating power plants than mainstream Republicans. We also find that political affiliation is moderated by education, in which more educated Tea Party supporters are more resistant to energy regulation and more supportive of fossil fuels than Tea Party supporters with less education. Implications for future research and the energy policy regime in the United States are discussed."

However, I can't find a non-paywall version.

June 19, 2017 | Unregistered CommenterJonathan

Jonathan -

I wrote a few comments discussing how Pubs are closer to Indies and maybe even Dems on some issues than they are to Tea Partiers.

We also find that political affiliation is moderated by education, in which more educated Tea Party supporters are more resistant to energy regulation and more supportive of fossil fuels than Tea Party supporters with less education.

Interesting. Especially since in the last election education level don't Pubs was strongly inversely associated with support for Trump relative to the relationship between education level among Pubs and support for Romney.

Seems an odd inconsistency, IMO.

June 19, 2017 | Unregistered CommenterJoshua

Also interesting in that it seems to suggest a confounder to the "science comprehension"/polarization on climate change association.

June 19, 2017 | Unregistered CommenterJoshua

"Personally, I don't see how anyone can write the Introduction and Method without first having the Results" (p. 172), the subject stated.

It's less surprising than that an empirical researcher would feel that way than that he or she would so unself-consciously admit to a style of study presentation that is based on post-hoc story-telling.

Is it merely a style of study presentation, or is it a philosophy of science?

June 20, 2017 | Unregistered Commenterdypoon

http://thehill.com/blogs/pundits-blog/lobbying-world/322181-just-the-facts-information-access-can-shrink-political

"The key lesson is that, in order for information to decrease polarization, it should be presented in the plainest, most impartial way possible. People are more open to update their views when they aren’t preoccupied with defending them."

The (paywalled) study is:
http://journals.sagepub.com/doi/abs/10.1177/1948550616687126

June 20, 2017 | Unregistered CommenterJonathan

This is interesting....speaks to how identity affiliation (and identity-influenced cognition) trumps (pun intended) other "motivations" in voters' decision making.

Of course, these taxes fund specific benefits, including the expansion of Medicaid and the tax credit that helps lower-income Americans buy insurance via the private health insurance marketplace. In the table below, we show the difference between the share of people in each state who receive the tax credits to buy insurance and the margin of voters the Democratic party won in the 2016 presidential election. Almost all the states where more residents pay the tax than receive the credit are blue, while most of the states where more people receive the credit than pay the tax are red.

My bold


https://fivethirtyeight.com/features/republicans-obamacare-repeal-would-cut-taxes-but-mostly-in-blue-states/

June 20, 2017 | Unregistered CommenterJoshua

Reading to the end, I see the authors made a similar point:

Since our analysis suggests that Obamacare taxes are disproportionately paid by blue states and disproportionately benefit red states, in an important sense, Republican senators are lining up to vote against their states’ interests. Of course, the ACA taxes were originally enacted by Democrats, so prioritizing ideological and partisan commitments over the state-level costs and benefits is by no means unique to the GOP. But this trend does highlight a key fact about contemporary policymaking: Senators evaluating big-ticket bills like the ACA and the House’s American Health Care Act are looking at them through the lenses of partisanship and ideology much more than through the lens of how the bills will affect their states.

June 20, 2017 | Unregistered CommenterJoshua

Joshua,

That's just more of the old red-state blue-state paradox, first pointed out by Pat Moynihan in the 70's.

https://en.wikipedia.org/wiki/Federal_taxation_and_spending_by_state

"In other words, Democratic-leaning states tend to be net contributors to the federal budget while Republican-leaning states are more often net recipients of federal spending."

June 21, 2017 | Unregistered CommenterJonathan

Dan is on to a key datum with his bias in publication statistics. Consider this sad example from Tech Review:

https://www.technologyreview.com/s/608089/climate-change-has-made-heat-waves-much-more-deadly-mainly-for-the-poor/

"..........A study published in Science Advances shows that as average summer temperatures in India rose 0.5 oC from 1960 until 2009, the chance of a heat wave causing more than 100 deaths more than doubled. As the authors write, their work is far from theoretical: heat waves in 2010, 2013, and 2015 each killed thousands of people across the country....."

Ah yes, but from 1960 to 2010 the population of India More Than Tripled >>
http://data.worldbank.org/country/india
>> so on a per capita basis deaths from heat waves Fell By Two Thirds, They Did NOT "more than double".

It gets worse - further in the same article, we read:
"......Richer people and countries will be better equipped to adapt. And in a cruel twist, some of those nations, particularly in northern Europe, might actually get a boost in GDP as their cold climates get increasingly balmy...."

That's emphatically NOT the case. Elementary hydrodynamic analysis will show that as polar ice melts, increasing flow of lower-saline Labrador current southwards, the effect will be to weaken the Gulf Stream, which is what makes northern Europe habitable in the first place.may already be happening, since the jet stream is already shifting, causing incredible cold spots in locations that had generally been cold anyway, like Siberia, and hot spots elsewhere.

Might the Indians fare better if they stopped poisoning their waters with every pollutant, bacterium, germ known to humanity - and even contributing new ones, like the Delhi virus? Even rural groundwater is now laced with arsenic and assorted fertilizer chemicals, and the situation in lakes and rivers - aka tannery runoff, dead body repository, and general sewage and garbage disposal - is too well documented to require elaboration. Instead, they are advised to install solar panels for powering air conditioners, for which they expect to send us a bill.

June 21, 2017 | Unregistered CommenterEcoute Sauvage

Jonathan -

=={ That's just more of the old red-state blue-state paradox, first pointed out by Pat Moynihan in the 70's. }==

Yes, the pattern whereby red states depend more on federal subsidization is an ironic twist that has been around for a while. But somehow, with the health insurance policy polarization, the contradiction between ideological partisanship and economic patterns seemed more stark to me. Maybe that's just because it is more under the microscope right now.

June 21, 2017 | Unregistered CommenterJoshua

PS The Australians, who DID install their solar panels, suffer from heat waves as their grid fails on a regular basis.
http://www.popsci.com/deadly-heatwave-climate-change/#page-2
Most public buildings now have standby diesel generators......

June 21, 2017 | Unregistered CommenterEcoute Sauvage

For those Jonathan Haidt fans out there. Discusses Heterodox Academy, campus culture wars, Haidt's take on the election.

June 21, 2017 | Unregistered CommenterJonathan

Jonathan -

Thanks for that link. I have a lot of respect for Haidt's approach to research...but have been thinking lately that he's jumped the shark with his identification of a "victimhood culture" with a locus on college campuses (as if perfecting victimhood to an art form hasn't been around for a long time, reaching a magnitude at places like Fox News that dwarfs in scale what we see on college campuses). It's nice to see that Haidt is at least considering the impact of his advocacy in a fuller context...although I'm still skeptical...he seems rather dug in. Not to say that there aren't examples of legitimate problems to be found - but it seems rather absurd, to me, to argue that there is some larger, long term "signal" of greater intolerance in our society that has resulted from intolerance on college campuses, when we consider massive social developments such as people feeling freer to be open about their sexuality.

In October, Heterodox Academy released a Guide to Colleges that rates campuses on whether they’re conducive to free speech and diversity of thought. The ratings are based on a combination of factors, including whether the college has endorsed the University of Chicago’s principles on free expression. It also takes into account rankings from FIRE and from the Intercollegiate Studies Institute, a conservative group founded in 1953 by, among others, William F. Buckley Jr.

Here's an idea: since Haidt is so convinced that the "victimhood culture" on college campuses and the accompanying "coddling" of American minds does so much harm...it would be interesting to see whether or not some psychological profiling might find that Brown students are systematically differentiated from U of C students on scales of all the supposed lines of impact - such as hypersensitivity and inability to deal with adversity - after having moved on from their safespace college environment.

My guess is that no such differentiation would be found. Just my guess is that there would be no such differentiation to be found if we had those profiles of the general public from decades ago to compare to profiles of the general public today. Everyone loves a slippery slope, and people have been decrying the decline of "kids today" since we lived in caves, I would guess.

June 21, 2017 | Unregistered CommenterJoshua

Reverting back to the topic of this blog, I think that this paper raises some key issues regarding scientific peer review and the involvement of corporations:

http://retractionwatch.com/2017/06/21/journal-alerts-readers-technical-criticism-crispr-study/

More background information on CRISPR here:

https://www.amazon.com/Crack-Creation-Editing-Unthinkable-Evolution/dp/0544716949/ref=sr_1_1?ie=UTF8&qid=1498072059&sr=8-1&keywords=crack+in+creation

June 21, 2017 | Unregistered CommenterGaythia Weis

link drop:
https://link.springer.com/article/10.1007%2Fs10584-017-1993-1

"Indeed, respondents who were asked about climate change responded “Yes” (definitely or somewhat) more often (85.8%) than respondents who were asked about global warming (80.9%), an effect observed for Republicans (74.4 vs. 65.5%) but not Democrats (94% in both conditions). "

June 21, 2017 | Unregistered CommenterJonathan

Did the EU just sanction the base rate fallacy?:

http://www.courthousenews.com/eu-relaxes-evidence-standard-vaccine-liability-case/

"In the case at hand, sufficient proof would include “the temporal proximity between the administering of a vaccine and the occurrence of a disease and the lack of personal and familial history of that disease, together with the existence of a significant number of reported cases of the disease occurring following such vaccines being administered,” according to the ruling."

June 21, 2017 | Unregistered CommenterJonathan

"For those Jonathan Haidt fans out there. Discusses Heterodox Academy, campus culture wars, Haidt's take on the election."

The take on the election was particularly interesting.

Then I realized that it’s a new game. It’s one thing to be provocative when all the powers controlling universities are on the left," he says. "But now that the presidency and the Department of Education are controlled by the right, the dangers are very different."

He fretted over the implications of the election for Heterodox Academy, posting his thoughts in a letter to the group’s membership in January. "HxA must proceed with caution," he wrote. "In a time of such powerful and understandable passions, it will be harder for HxA to make the case that wisdom is to be found on all sides, and from the conflict of viewpoints." He added, "It will be easier for us to anger and alienate potential supporters."

Why are they different? If control of universities by the right is dangerous, why is it not equally dangerous to have them controlled by the left? If control by the right inflames passions and causes the anger and alienation of supporters on the left, why wouldn't control by the left cause anger and alienation of supporters on the right?

Where does the asymmetry come from?

"...but have been thinking lately that he's jumped the shark with his identification of a "victimhood culture" with a locus on college campuses..."

You would, culturally.

"but it seems rather absurd, to me, to argue that there is some larger, long term "signal" of greater intolerance in our society that has resulted from intolerance on college campuses, when we consider massive social developments such as people feeling freer to be open about their sexuality."

It's simply a replacement of targets. Previously, society was intolerant of homosexuality and homophobes felt free to express their feelings. Now society is intolerant of homophobes and homosexuals are free to express their feelings. Same difference.

"I think that this paper raises some key issues regarding scientific peer review and the involvement of corporations:"

Does it? That sounds a lot like the ad hominem Appeal to Motive fallacy. I agree - it's worrying that it should be appearing so openly in the debate about scientific peer review. That was what you meant, yes?

"respondents who were asked about climate change responded “Yes” (definitely or somewhat) more often (85.8%) than respondents who were asked about global warming (80.9%)"

The only surprising thing about this is that they'd be surprised. "Climate change" means all changes in climate, including natural ones. Ice ages, the holocene optimum, the Younger Dryas, the Minoan Warm Period, the Roman Warm Period, the Medieval Warm Period, the Green Sahara, the American Megadroughts, and so on. Do you believe in ice ages? Yes. Do you believe in Al Gore's Imminent Apocalypse? No. Why would anyone be surprised? Unless they didn't know what the debate was about. Or perhaps the paper's primary authors were all Democrats, for who there's no difference in meaning?

"Did the EU just sanction the base rate fallacy?"

Post hoc ergo propter hoc.

June 22, 2017 | Unregistered CommenterNiV

The problem is not the reviewers but the editors. The review guidelines are absurd as judgments on results are not only allowed but encouraged. This makes no sense whatsoever. Reviewers should be exclusively concerned with the methodology employed so as to ensure, within reason, that a given study is reproducible.

June 22, 2017 | Unregistered CommenterFlanagan

@Flanagan -- I agree!

June 23, 2017 | Registered CommenterDan Kahan

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>