follow CCP

Recent blog entries
popular papers

Science Curiosity and Political Information Processing

What Is the "Science of Science Communication"?

Climate-Science Communication and the Measurement Problem

Ideology, Motivated Cognition, and Cognitive Reflection: An Experimental Study

'Ideology' or 'Situation Sense'? An Experimental Investigation of Motivated Reasoning and Professional Judgment

A Risky Science Communication Environment for Vaccines

Motivated Numeracy and Enlightened Self-Government

Making Climate Science Communication Evidence-based—All the Way Down 

Neutral Principles, Motivated Cognition, and Some Problems for Constitutional Law 

Cultural Cognition of Scientific Consensus
 

The Tragedy of the Risk-Perception Commons: Science Literacy and Climate Change

"They Saw a Protest": Cognitive Illiberalism and the Speech-Conduct Distinction 

Geoengineering and the Science Communication Environment: a Cross-Cultural Experiment

Fixing the Communications Failure

Why We Are Poles Apart on Climate Change

The Cognitively Illiberal State 

Who Fears the HPV Vaccine, Who Doesn't, and Why? An Experimental Study

Cultural Cognition of the Risks and Benefits of Nanotechnology

Whose Eyes Are You Going to Believe? An Empirical Examination of Scott v. Harris

Cultural Cognition and Public Policy

Culture, Cognition, and Consent: Who Perceives What, and Why, in "Acquaintance Rape" Cases

Culture and Identity-Protective Cognition: Explaining the White Male Effect

Fear of Democracy: A Cultural Evaluation of Sunstein on Risk

Cultural Cognition as a Conception of the Cultural Theory of Risk

« Hey, everybody--come to this cool " 'Hot hand fallacy' fallacy" workshop! | Main | Fortifying #scicomm craft norms with empirical inquiry-- a snippet »
Wednesday
Jun062018

Shut up & update! . . . a snippet

Also something I've been working on . . . .

1. “Evidence” vs.”truth”—the law’s position. The distinction between “evidence for” a proposition and the “truth of” it is inscribed in the legal mind through professional training and experience.

Rule 401 of the Federal Rules of Evidence defines “relevance” as “any tendency” of an item of proof “to make a fact … of consequence” to the litigation either “more or less probable” in the estimation of the factfinder. In Bayesian terms, this position is equivalent to saying that an item of proof is “relevant” (and hence presumptively admissible; see Fed. R. Evid. 402) if, in relation to competing factual allegations, the likelihood ratio associated with that evidence is either less than or greater than 1 (Lempert 1977).  

Folksy idioms—e.g., “a brick is not a wall” [Rule 401, advisory committee notes])—are used to teach prospective lawyers that this “liberal” standard of admissibility does not depend on the power of a piece of evidence to establish a particular fact by the requisite standard of proof (“more probable than not” in civil cases; “beyond a reasonable doubt” in criminal cones).

Or in Bayesian terms, we would say that a properly trained legal reasoner does not determine “relevance” (and hence admissibility) by asking whether an item of proof will on its own generate a posterior estimate either for or against the “truth” of that fact. Again, because the process of proof is cumulative, the only thing that matters is that a particular piece of evidence have a likelihood ratio different from 1 in relation to competing litigation hypotheses.

2. “I don’t believe it . . . .” This popular response, among both pre- and post-publication peer reviewers, doesn’t get the distinction between “evidence for” and the “truth of” an empirical claim.

In Bayesian terms, the reviewer who treats his or her “belief” in the study result as informative is unhelpfully substituting his or her posterior estimate for an assessment of the likelihood ratio associated with the data. Who cares what the reviewer “believes”? Disagreement about the relative strength of competing hypotheses is, after all, the occasion for data collection! If a judge or lawyer can “get” that a “brick is not a wall,” then surely a consumer of empirical research can, too: the latter should be asking whether an empirical study has “any tendency … to make a fact … of consequence” to empirical inquiry either “more or less probable” in the estimation of interested scholars (this is primarily a question of the validity of the methods used and the probative weight of the study finding).

That is, the reviewer should have his or her eyes glued to  the likelihood ratio, and not be distracted by any particular researcher’s posterior.

3.  “Extraordinary claims require extraordinary proof . . . .” No, they really don’t.

This maxim treats the strength with which a fact is held to be true as a basis for discounting the likelihood ratio associated with contrary evidence. The scholar who takes this position is saying, in effect, “Your result should see the light of day only if it is so strong that it flips scholars from a state of disbelief to one of belief, or vice versa.” 

But in empirical scholarship as in law, “A brick is not a wall.”  We can recognize the tendency of a (valid) study result to make some provisional apprehension of truth less probable than it would otherwise be while still believing—strongly, even—that the contrary hypothesis so supported is unlikely to be true.

* * *

Or to paraphrase a maxim Feynman is sometimes (mis)credited with saying, “Shut up & update!”

References

Federal Rules of Evidence (2018) & Advisory Committee Notes.

Lempert, R.O. Modeling relevance. Michigan Law Review, 75, 1021-1057 (1977).

 


PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (8)

Dan,

"If a judge or lawyer can “get” that a “brick is not a wall,” then surely a consumer of empirical research can, too..." - throwing judges and lawyers under the bus?

"...and not be distracted by any particular researcher’s posterior." - worded like a title IX warning?

"Extraordinary claims require extraordinary proof..." - I have always understood this in terms of the original Hume: "A wise man proportions his belief to the evidence", which I don't think you would disagree with at all. I don't see it as implying “Your result should see the light of day only if it is so strong that it flips scholars from a state of disbelief to one of belief, or vice versa.” I see that you are arguing for accumulation of evidence and against separate individual evidence judgements (dismissing small bricks as insignificant, so to speak), but I never understood the Sagan-esque quote to imply individual evidence judgements. I think you may end up confusing readers based on that interpretation.

June 6, 2018 | Unregistered CommenterJonathan

Hmm. I wondering if Dan's choice of a reviewer's comments to pick on for this particular form of illogic is a bit... pointed. Do you have a particular reviewer in mind, Dan? :-)

But I agree it's sadly all too common - and in both directions, too. A lot of people think a 95% significance is scientifically conclusive, when all it says is the likelihood ratio is big enough to take seriously - to move belief by a non-trivial jump in a particular direction, without necessarily drawing any conclusion or changing any minds. And vice versa, of course, thinking journal papers should only publish claims (or evidence for claims) believed to be true.

And welcome back, Dan! I was beginning to wonder if you was OK! Hope you had a good time, wherever you've been.

June 6, 2018 | Unregistered CommenterNiV

@Jonathan -- the "extraordinary claim ... extraordinary evidence" claim is ambiguous. If all it means is that evidence has to be very strong to offset long prior odds, then it is unobjectionable--indeed, banal. But if, as often the case, the"extraordinary claim ... extraordinary evidence" is being used to discount study as unentitled to be taken seriously, then the mistake I'm objecting to is being made.

glad you enjoyed "distracted by any researcher's posterior"

Oh-- & didn't mean to throw lawyers or judges under any bus; indeed, they are driving it here. It is one place where I think law has exemplary attitude on logic of proof

June 6, 2018 | Registered CommenterDan Kahan

@NiV-- thanks. was underwater w/ various deadline-sensitive projects ... (including grading 5,000 pages worth of exams...)

June 6, 2018 | Registered CommenterDan Kahan

This might be relevant:

http://philsci-archive.pitt.edu/14659/

The relevance would be: every time one binarizes belief (as when deciding to accept or reject evidence, instead of adhering to non-binary likelihood ratios), one suffers from the impossibility result in the above paper. So, delay binarization as long as possible (until action is necessary).

June 6, 2018 | Unregistered CommenterJonathan

3. “Extraordinary claims require extraordinary proof . . . .”
No, they really don’t.

Isn't "extraordinary claims require extraordinary evidence" the more commonly heard expression? Not "extraordinary claims require extraordinary proof"

I think those two versions are quite different, and I think that the latter expression makes much more sense.

For example, any evidence that the moon is made of green cheese would, by definition, be extraordinary evidence.

June 7, 2018 | Unregistered CommenterJoshua

Or in Bayesian terms, we would say that a properly trained legal reasoner does not determine “relevance” (and hence admissibility) by asking whether an item of proof will on its own generate a posterior estimate either for or against the “truth” of that fact.

So, in Bayesian terms, what does a properly train legal reasoner do?

Again, because the process of proof is cumulative, the only thing that matters is that a particular piece of evidence have a likelihood ratio different from 1 in relation to competing litigation hypotheses.

What is the relationship between "a particular piece of evidence" in the second sentence and "an item of proof will on its own" in the first?

Also if "the process of truth is cumulative" then what matters most is something cumulative -- not one particular piece of evidence.

Did you mean to write "he only thing that matters about a particular piece of evidence is whether it ..."??

- your frustrated wanna be editor

June 11, 2018 | Unregistered CommenterCortlandt

@Cortlandt & @Joshua-- am using "item of proof" & "piece of evidence" interchangeably.

June 11, 2018 | Registered CommenterDan Kahan

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>