The "judicial behavior" measurement problem: What does it *mean* to say that "ideology" explains judicial decisions?
This is another excerpt from Tthe latest CCP paper, "Ideology" or "Situation Sense"? An Experimental Investigation of Motivated Reasoning and Professional Judgment, Univ. Pa. L. Rev. (in Press). It presents what I consider to be the major methodological defect in observational--or correlational--studies that purport to find that "ideological" motivations explain variation in judicial decisions: the failure to specify a cogent theory of what counts as an "ideological" as oppoosed to a legal or jurisprudential motivation, and a resulting failure to specify what sorts of evidence would support an inference of "ideological" motivations.
A. Observational studies
Associated with the disciplines of political science and economics, studies that use observational methods make up the largest share of the literature on the impact of ideological motivations on judicial decisionmaking. Such studies use correlational analyses—in the form of multivariate regression models—that treat the “ideology” of individual judges as an “independent variable” the impact of which on case outcomes is assessed after partialing out or “controlling for” additional influences represented by other “independent variables.”
There are different methods for measuring judges’ “ideologies,” including (in the case of federal judges) the party of the appointing President and (in the case of Supreme Court Justices) the covariance of votes among judges who can be understood to be aligned along some unobserved or latent ideological continuum. Such studies tend to find that “ideology” so measured explains a “statistically significant” increment of variance in judicial determinations. Studies looking at the decisions of federal courts of appeals, which assign cases to three-judge panels for determination, also find that the impact of ideology so measured can be either accentuated or muted depending on the ideological composition of judges on the particular panel.
Critics of these studies identify methodological problems that they believe constrain the strength of the inferences that can be drawn from them. The most obvious of these is the sampling bias introduced by parties’ self-conscious selection of cases for litigation. . . .
Another, more subtle, but equally serious problem for observational studies of judicial ideology is the classification of “case outcomes.” In order to measure the impact of a judge’s “ideology” on decisionmaking, it is necessary to determine which outcomes are consistent with that judge’s ideology and which ones are not. Scholars doing observational studies generally classify outcomes as “liberal” or “conservative” based on the type of case and the prevailing party: for example, decisions favoring the government in “criminal” cases are deemed “conservative” and those the defendant “liberal”; in labor law cases, outcomes are “conservative” if they favor “management,” and “liberal” if they favor unions, and so forth.
The crudeness of this scheme not only injects noise into empirical analyses of case outcomes but also biases it toward overstated estimates of the impact of “ideology” on judicial decisionmaking. It is a well known feature of the Anglo-American system of law that it frequently demands that judges resort to normative reasoning. There is no way for highly general concepts such as “fraud,” “unreasonable seizure,” “unlawful restraint of trade,” “fair use,” “materiality,” “freedom of speech,” and the like to be made operative in particular cases without specifying what states of affairs those legal provisions should be trying to promote. Under “common law” style of reasoning dominant in Anglo-American law, the sorts of moral judgments that judges exercise to supply content to these types of concepts is not unconstrained; shared understandings of the general aim of the enacting legislature or other law promulgator, the appropriate deference to be afforded to previous elaborations of the content of the legal concept in question, and conformity to broader normative precepts that structure the law (“notice and opportunity to be heard,” “due process,” “like cases treated alike” etc.) limit the available interpretive options. But in ruling out many solutions, the sources of valid normative inspiration that judges can draw on often do not rule only one in.
In this environment, it is perfectly commonplace for judges who have competing “jurisprudential” orientations to disagree on what normative theory should animate a particular legal provision. It is not a surprise, either, that in those instances the competing orientations that guide judges will be correlated with alternative political philosophies or orientations on the part of the judges in question. Justice Douglas had a populist “economic decentralization” conception of “restraint of trade” for purposes of the Sherman Act; Professor and then Judge Robert Bork subscribed to an economic, “consumer welfare” alternative. These positions undoubtedly cohered with their respective political “ideologies,” too, and likely did as well with the “ideologies” of judges who championed one versus the other understanding of how U.S. antitrust law should be structured. But those who understand how the law works—and the contribution that judges, using normative theories play, in imparting content to it—would not characterize this debate as reflecting extralegal “ideological” considerations as opposed to the perfectly ordinary, acceptable exercise of jurisprudential judgments. Multivariate regression models are not necessary to ferret out the contribution that value-laden theories make to how judges decide these cases; judges openly admit that they are using such theories. Regardless of which President appointed these judges to the federal bench, no lawyer understands judges engaged in this sort of reasoning to be invoking “personal political preferences.”
An entirely different matter would have been presented, however, had Justice Douglas or Judge Bork proposed deciding an antitrust, labor law, free speech, criminal law or any other sort of case based on the religious affiliation of the litigants or on the contribution a particular outcome would have made to the electoral prospects of a candidate for President. The Sherman Act, the Wagner Act, the First Amendment, and even myriad criminal law statutes all demand the use of the form of guided normative theorizing we are describing. But the bare desire to use legal outcomes in particular cases (or in large classes of them) to disadvantage those who subscribe to a disfavored view of the best life or to advance the cause of a particular political party is plainly outside the range of considerations that can validly be appealed to in the exercise of normative reasoning intrinsic to law. Whether in the form of regression coefficient correlations, law-enforcement wiretaps, or anonymously leaked emails, evidence that judges of particular ideologies were being influenced by such considerations would be a ground for intense concern.
There is a distinction, in sum, between resort to normative considerations that are internal to law and ones external to it. The former are licit, the latter illicit, from the perspective that lawyers and judges in the U.S. system of justice share of what counts as valid legal reasoning.
The “prevailing party” outcome-classification scheme used in observational studies of judicial ideology is blind to the distinction. As a result, such studies will count in their estimates of the influence of “ideology” perfectly mundane associations between the jurisprudential philosophies of judges deciding cases on the basis of normative considerations internal to law and the party of the Presidents who appointed them or the voting records of those judges and judges who feel likewise about the normative theories that inform labor law, free speech cases, criminal cases and the like.
The correlations that these researchers report could also be capturing judges’ reliance on illicit political considerations, external to the law. But (critics point out) there is no way to know whether this is the case, or to what extent, given the indiscriminate coding of outcome variables that these studies employ.
Some candid adherents to the “ideology thesis” have acknowledged this point. But they have not supplied a response to what critics would identify as the significance of this concession. When observational-study proponents declare that they are finding that “ideology” accounts for judges’ decisions, they say they are measuring the extent to which those judges are not deciding cases on the basis of “law.” That is what gives this entire body of literature its currency—its “shock value.” But to the extent that the observational-study scholars are finding that judges who have different judicial philosophies will sometimes validly interpret the law to support different conclusions, then they are telling us something that already is clear— something, in fact, that the very judges whose behavior is being "explained" plainly say when they justify their decisions—and that gives no one any reason to be concerned about the quality of judicial decisionmaking.