Shut up & update! . . . a snippet
Dan Kahan Posted on Wednesday, June 6, 2018 at 1:23AM 
Also something I've been working on . . . .
1. “Evidence” vs.”truth”—the law’s position. The distinction between “evidence for” a proposition and the “truth of” it is inscribed in the legal mind through professional training and experience.
Rule 401 of the Federal Rules of Evidence defines “relevance” as “any tendency” of an item of proof “to make a fact … of consequence” to the litigation either “more or less probable” in the estimation of the factfinder. In Bayesian terms, this position is equivalent to saying that an item of proof is “relevant” (and hence presumptively admissible; see Fed. R. Evid. 402) if, in relation to competing factual allegations, the likelihood ratio associated with that evidence is either less than or greater than 1 (Lempert 1977).
Folksy idioms—e.g., “a brick is not a wall” [Rule 401, advisory committee notes])—are used to teach prospective lawyers that this “liberal” standard of admissibility does not depend on the power of a piece of evidence to establish a particular fact by the requisite standard of proof (“more probable than not” in civil cases; “beyond a reasonable doubt” in criminal cones).
Or in Bayesian terms, we would say that a properly trained legal reasoner does not determine “relevance” (and hence admissibility) by asking whether an item of proof will on its own generate a posterior estimate either for or against the “truth” of that fact. Again, because the process of proof is cumulative, the only thing that matters is that a particular piece of evidence have a likelihood ratio different from 1 in relation to competing litigation hypotheses.
2. “I don’t believe it . . . .” This popular response, among both pre- and post-publication peer reviewers, doesn’t get the distinction between “evidence for” and the “truth of” an empirical claim.
In Bayesian terms, the reviewer who treats his or her “belief” in the study result as informative is unhelpfully substituting his or her posterior estimate for an assessment of the likelihood ratio associated with the data. Who cares what the reviewer “believes”? Disagreement about the relative strength of competing hypotheses is, after all, the occasion for data collection! If a judge or lawyer can “get” that a “brick is not a wall,” then surely a consumer of empirical research can, too: the latter should be asking whether an empirical study has “any tendency … to make a fact … of consequence” to empirical inquiry either “more or less probable” in the estimation of interested scholars (this is primarily a question of the validity of the methods used and the probative weight of the study finding).
That is, the reviewer should have his or her eyes glued to the likelihood ratio, and not be distracted by any particular researcher’s posterior.
3. “Extraordinary claims require extraordinary proof . . . .” No, they really don’t.
This maxim treats the strength with which a fact is held to be true as a basis for discounting the likelihood ratio associated with contrary evidence. The scholar who takes this position is saying, in effect, “Your result should see the light of day only if it is so strong that it flips scholars from a state of disbelief to one of belief, or vice versa.”
But in empirical scholarship as in law, “A brick is not a wall.” We can recognize the tendency of a (valid) study result to make some provisional apprehension of truth less probable than it would otherwise be while still believing—strongly, even—that the contrary hypothesis so supported is unlikely to be true.
* * *
Or to paraphrase a maxim Feynman is sometimes (mis)credited with saying, “Shut up & update!”
References
Federal Rules of Evidence (2018) & Advisory Committee Notes.
Lempert, R.O. Modeling relevance. Michigan Law Review, 75, 1021-1057 (1977).
Reader Comments (8)
Dan,
"If a judge or lawyer can “get” that a “brick is not a wall,” then surely a consumer of empirical research can, too..." - throwing judges and lawyers under the bus?
"...and not be distracted by any particular researcher’s posterior." - worded like a title IX warning?
"Extraordinary claims require extraordinary proof..." - I have always understood this in terms of the original Hume: "A wise man proportions his belief to the evidence", which I don't think you would disagree with at all. I don't see it as implying “Your result should see the light of day only if it is so strong that it flips scholars from a state of disbelief to one of belief, or vice versa.” I see that you are arguing for accumulation of evidence and against separate individual evidence judgements (dismissing small bricks as insignificant, so to speak), but I never understood the Sagan-esque quote to imply individual evidence judgements. I think you may end up confusing readers based on that interpretation.
Hmm. I wondering if Dan's choice of a reviewer's comments to pick on for this particular form of illogic is a bit... pointed. Do you have a particular reviewer in mind, Dan? :-)
But I agree it's sadly all too common - and in both directions, too. A lot of people think a 95% significance is scientifically conclusive, when all it says is the likelihood ratio is big enough to take seriously - to move belief by a non-trivial jump in a particular direction, without necessarily drawing any conclusion or changing any minds. And vice versa, of course, thinking journal papers should only publish claims (or evidence for claims) believed to be true.
And welcome back, Dan! I was beginning to wonder if you was OK! Hope you had a good time, wherever you've been.
@Jonathan -- the "extraordinary claim ... extraordinary evidence" claim is ambiguous. If all it means is that evidence has to be very strong to offset long prior odds, then it is unobjectionable--indeed, banal. But if, as often the case, the"extraordinary claim ... extraordinary evidence" is being used to discount study as unentitled to be taken seriously, then the mistake I'm objecting to is being made.
glad you enjoyed "distracted by any researcher's posterior"
Oh-- & didn't mean to throw lawyers or judges under any bus; indeed, they are driving it here. It is one place where I think law has exemplary attitude on logic of proof
@NiV-- thanks. was underwater w/ various deadline-sensitive projects ... (including grading 5,000 pages worth of exams...)
This might be relevant:
http://philsci-archive.pitt.edu/14659/
The relevance would be: every time one binarizes belief (as when deciding to accept or reject evidence, instead of adhering to non-binary likelihood ratios), one suffers from the impossibility result in the above paper. So, delay binarization as long as possible (until action is necessary).
Isn't "extraordinary claims require extraordinary evidence" the more commonly heard expression? Not "extraordinary claims require extraordinary proof"
I think those two versions are quite different, and I think that the latter expression makes much more sense.
For example, any evidence that the moon is made of green cheese would, by definition, be extraordinary evidence.
So, in Bayesian terms, what does a properly train legal reasoner do?
What is the relationship between "a particular piece of evidence" in the second sentence and "an item of proof will on its own" in the first?
Also if "the process of truth is cumulative" then what matters most is something cumulative -- not one particular piece of evidence.
Did you mean to write "he only thing that matters about a particular piece of evidence is whether it ..."??
- your frustrated wanna be editor
@Cortlandt & @Joshua-- am using "item of proof" & "piece of evidence" interchangeably.