I’ve been asked to be part of an NAS working group that will develop a proposal on how science should figure in the training of lawyers. I’m going to put together a memo that outlines my own initial views and distribute it shortly before the first meeting (in mid January). Below is a condensed account of the points and themes that my memo will stress. But my ideas are provisional & formative; indeed, I share them to invite your reactions, which I expect to stimulate and educate my own thinking.
I welcome feedback not only on the substance but also on what to include in an annotated bibliography, the germ of which appears after the narrative section. The bibliography is not meant as a syllabus for a course; some of the items would no doubt be assigned in the sort of “forensic science literacy” course I am describing, but mainly I am trying to compile sources that help make the spirit & philosophy of such an offering more vivid for memo readers.
Feel free to respond via email to me (email@example.com).
A. General Points
1. What the aim should be—and what it shouldn’t
The 2009 NAS Forensic Science Report did more than identify various forms of proof that lack scientific validity. It also demonstrated that the U.S. legal system is suffused with a basic incomprehension of the fundamentals of sound science. The prospect that this deficit would continue to make the law receptive to specious forms of scientific evidence and unreceptive to valid ones motivated the Report’s core recommendation that the Nation’s universities be made instruments for bringing the “culture of science to law.”
Spelling out what law schools should be expected to contribute to this project is, in my view, the proper focus of the working group’s attention. Lawyers don’t need to be trained to do science but they can and should be taught to recognize what constitutes sound forensic science and what doesn’t. A model course should instruct students in the general concepts and procedures that one must understand in order to perform this recognition task reliably, including principles of validity; elements of probability; and methods of inquiry (more on these below). The goal should be to create an intellectual foundation broad and stable enough to support understanding of any particular type of legally relevant scientific material.
The aim of the working group should not be to try to compile a list of important current or future types of forensic science (e.g., fingerprints or neuroscience) or specific areas of study relating to the forensic process (e.g., reliability of witness identification or the pervasiveness of cognitive biases). These are matters that one would certainly imagine as the focus of either a more comprehensive or more advanced course in law and science, and certainly the greater the number of offerings law schools provide on law and science, the better. But the most critical objective is to identify the core offering (or core curricular content) that every programmust include.
By confining its focus to what is in fact essential, the proposal will underscore the theme that U.S. law schools must treat imparting forensic science literacy as an essential part of their curricula. Lawyers and judges who possess basic forensic science literacy can be expected to handle competently whatever particular forms of scientific proof they must deal with; ones who lack this capacity cannot be expected to handle any well.
2. Principles of validity
Here I have in mind the concepts essential to systematic evaluation of the soundness of any general form of scientific inquiry or any particular application of it. These include validity proper: do the methods and design employed genuinely support the inferences that the researcher seeks to draw (internal validity), and from those can one draw reasonable inferences about the real-world phenomena that are being modeled by the study (external validity)? Are the measures employed reliable: do they generate consistent results, and do results agree across trials and researchers? The topic of causal inference is also usefully considered together with these issues, as is the concept of hypothesis testing.
The goal is to make students acquainted with the sorts of criteria that those who reliably distinguish sound from unsound science use for that purpose. I doubt that forensic science literacy as a reliable capacity to recognize sound and unsound forms of science as applied to law can be reduced to any sort of checklist of do’s & don’ts, rights & wrongs. But the elaborated development of a set of criteria for “valid” forensic science is likely a sensible way, pedagogically speaking, to conjure the sort of atmosphere in which such a capacity can be acquired and refined.
Such instruction can easily be illustrated with legal examples because these are exactly the sorts of considerations an incomprehension of which is reflected in the practice of forensic science that the 2009 Report criticizes.
3. Elements of probability
Concepts of probability animate the methods and testing strategies of science (and ultimately the philosophy of competing conceptions of scientific understanding, although that’s a depth the forensic- science-literate lawyer needn’t reach unless he or she is drawn there by curiosity). But, again, forensic- science-literate lawyers don’t need to be trained to do sound science, only to recognize it. For this purpose, it is sufficient for them to be attain, first, a conceptual grasp of the basic elements of probability (e.g., normal distributions and standard deviation; nonnormal distributions, such as “survival” curves; measurement error, sampling error, and estimation; p-values and confidence intervals; Bayes’s Theorem and Bayesian inference) and, second, enough fluency with statistics to be able to read and comprehend the terms in which empirical results are ordinarily reported. They should also be made familiar with those characteristic shortcomings of unsound science that consist in an absence of genuine comprehension, as opposed to mechanical application, of statistical procedures. Once more, the law is filled with practical illustrations.
4. Methods of inquiry
The idea here would be to make students familiar with the conventional sorts of methods that will inform the sorts of empirical work they are likely to encounter as lawyers. These include, at a high level of generality, observational vs. experimental approaches; but at a more particular level, it would be useful, too, to supply students with the materials necessary to enable informed and critical reflection on specific methods that bear on important, domain-specific matters of inquiry (e.g., clinical trials and “blinded” experimental methods, “laboratory” vs. “field” experimentation; multivariate regression vs. “matching” for observational studies). Such instruction can usefully be guided by the objective of making prospective lawyers familiar with the characteristic limitations of studies that employ one or another method—ones associated not just in the misapplication or inappropriate uses of one or another method but also ones with the inherent imperfection of all testing strategies.
Of course lawyers should also be taught that precisely because all methods are imperfect, it is a mistake—a popular misconception that reflects science illiteracy— to equate scientific validity with the conclusive or final resolution of an issue, or even with proof that in itself satisfies any particular legal standard such as “beyond a reasonable doubt.” No more is or can be expected of forensic proof than that it supply a decisionmaker with more evidence for believing (or disbelieving) a proposition than she otherwise would have had (and of course forms that supply anything less than that should not be tolerated).
B. Annotated bibliography
Useful sources. Possible course materials but mainly sources that illustrate or reflect the points above
1. Principles of validity
National Research Council (U.S.). Committee on Identifying the Needs of the Forensic Science Community., National Research Council (U.S.). Committee on Science Technology and Law Policy and Global Affairs. and National Research Council (U.S.). Committee on Applied and Theoretical Statistics. Strengthening Forensic Science in the United States: A Path Forward, (National Academies Press, Washington, D.C., 2009) —relevant for all really
Daubert v. Merrell Dow Pharmaceuticals, 509 U.S. 579 (1993) (suggesting that principles of validity should be normative for evaluation of admissibility of expert proof)
Kumho Tire Co. v. Carmichael, 526 U.S. 137 (1999) (just kidding!)
United States v. Llera Plaza, 179 F. Supp. 2d 492 (E.D. Pa., Jan. 7, 2002) (holding on basis of brilliant application of the principles of validity that fingerprints are not and hence fingerprint experts should not be permitted to give conclusions on “matching” prints)
United States v. Llera Plaza, 188 F. Supp. 2d 549, 576 (E.D. Pa., March 3, 2002) (oops, nevermind!)
Curious what people would recommend here. Is there something for understanding of basic concepts of scientific validity that is as accessible and compact as say Abelson’s Statistics as Principled Argument, below?
2. Elements of probability
Finkelstein, M.O. and Fairley, W.B. A Bayesian Approach to Identification Evidence. Harvard Law Review 83, 489-517 (1970).
Finkelstein, M.O. Basic concepts of probability and statistics in the law, (Springer, New York, 2009).
Matrixx Initiatives, Inc. v. Siracusano, 131 S. Ct. 1309 (2011) (recognizing that significance testing for scientific studies is not a criterion of practical significance for causal inferences relating to law)
Abelson, R.P. Statistics as principled argument, (L. Erlbaum Associates, Hillsdale, N.J., 1995).
Gigerenzer, G. Calculated Risks: How to Know When Numbers Deceive You (Simon and Schuster, New York, 2002).
Motulsky, H. Intuitive biostatistics : a nonmathematical guide to statistical thinking, (Oxford University Press, New York, 2010).
3. Methods of inquiry
Fisher, F.M. Multiple Regression in Legal Proceedings. Colum. L. Rev. 80, 702-736 (1980).
Again, eager for suggestions here. There are lots of good “handbooks” for social science methods; but is there something that is more general, yet accessible and compact (again, compare Abelson)