follow CCP

Recent blog entries
popular papers

Science Curiosity and Political Information Processing

What Is the "Science of Science Communication"?

Climate-Science Communication and the Measurement Problem

Ideology, Motivated Cognition, and Cognitive Reflection: An Experimental Study

'Ideology' or 'Situation Sense'? An Experimental Investigation of Motivated Reasoning and Professional Judgment

A Risky Science Communication Environment for Vaccines

Motivated Numeracy and Enlightened Self-Government

Making Climate Science Communication Evidence-based—All the Way Down 

Neutral Principles, Motivated Cognition, and Some Problems for Constitutional Law 

Cultural Cognition of Scientific Consensus
 

The Tragedy of the Risk-Perception Commons: Science Literacy and Climate Change

"They Saw a Protest": Cognitive Illiberalism and the Speech-Conduct Distinction 

Geoengineering and the Science Communication Environment: a Cross-Cultural Experiment

Fixing the Communications Failure

Why We Are Poles Apart on Climate Change

The Cognitively Illiberal State 

Who Fears the HPV Vaccine, Who Doesn't, and Why? An Experimental Study

Cultural Cognition of the Risks and Benefits of Nanotechnology

Whose Eyes Are You Going to Believe? An Empirical Examination of Scott v. Harris

Cultural Cognition and Public Policy

Culture, Cognition, and Consent: Who Perceives What, and Why, in "Acquaintance Rape" Cases

Culture and Identity-Protective Cognition: Explaining the White Male Effect

Fear of Democracy: A Cultural Evaluation of Sunstein on Risk

Cultural Cognition as a Conception of the Cultural Theory of Risk

« Another country heard from: more data on impact of 'expert judgment' in insulating judges from popular information-processing biases | Main | Fun webinar event on politicization of science-- c'mon, sign up! »
Thursday
May142015

The "judicial behavior" measurement problem: What does it *mean* to say that "ideology" explains judicial decisions? 

This is another excerpt from Tthe latest CCP paper, "Ideology" or "Situation Sense"? An Experimental Investigation of Motivated Reasoning and Professional Judgment, Univ. Pa. L. Rev. (in Press). It presents what I consider to be the major methodological defect in observational--or correlational--studies that purport to find that "ideological" motivations explain variation in judicial decisions: the failure to specify a cogent theory of what counts as an "ideological" as oppoosed to a legal or jurisprudential motivation, and a resulting failure to specify what sorts of evidence would support an inference of "ideological" motivations.

A. Observational studies

Associated with the disciplines of political science and economics, studies that use observational methods make up the largest share of the literature on the impact of ideological motivations on judicial decisionmaking. Such studies use correlational analyses—in the form of multivariate regression models—that treat the “ideology” of individual judges as an “independent variable” the impact of which on case outcomes is assessed after partialing out or “controlling for” additional influences represented by other “independent variables.”  

There are different methods for measuring judges’ “ideologies,” including (in the case of federal judges) the party of the appointing President  and (in the case of Supreme Court Justices) the covariance of votes among judges who can be understood to be aligned along some unobserved or latent ideological continuum.  Such studies tend to find that “ideology” so measured explains a “statistically significant” increment of variance in judicial determinations. Studies looking at the decisions of federal courts of appeals, which assign cases to three-judge panels for determination, also find that the impact of ideology so measured can be either accentuated or muted depending on the ideological composition of judges on the particular panel.

Critics of these studies identify methodological problems that they believe constrain the strength of the inferences that can be drawn from them.  The most obvious of these is the sampling bias introduced by parties’ self-conscious selection of cases for litigation. . . . 

Another, more subtle, but equally serious problem for observational studies of judicial ideology is the classification of “case outcomes.” In order to measure the impact of a judge’s “ideology” on decisionmaking, it is necessary to determine which outcomes are consistent with that judge’s ideology and which ones are not. Scholars doing observational studies generally classify outcomes as “liberal” or “conservative” based on the type of case and the prevailing party: for example, decisions favoring the government in “criminal” cases are deemed “conservative” and those the defendant “liberal”; in labor law cases, outcomes are “conservative” if they favor “management,” and “liberal” if they favor unions, and so forth.  

The crudeness of this scheme not only injects noise into empirical analyses of case outcomes but also biases it toward overstated estimates of the impact of “ideology” on judicial decisionmaking.  It is a well known feature of the Anglo-American system of law that it frequently demands that judges resort to normative reasoning.  There is no way for highly general concepts such as “fraud,” “unreasonable seizure,” “unlawful restraint of trade,” “fair use,” “materiality,” “freedom of speech,” and the like to be made operative in particular cases without specifying what states of affairs those legal provisions should be trying to promote.  Under “common law” style of reasoning dominant in Anglo-American law,  the sorts of moral judgments that judges exercise to supply content to these types of concepts is not unconstrained; shared understandings of the general aim of the enacting legislature or other law promulgator, the appropriate deference to be afforded to previous elaborations of the content of the legal concept in question, and conformity to broader normative precepts that structure the law (“notice and opportunity to be heard,” “due process,” “like cases treated alike” etc.) limit the available interpretive options. But in ruling out many solutions, the sources of valid normative inspiration that judges can draw on often do not rule only one in.  

In this environment, it is perfectly commonplace for judges who have competing “jurisprudential” orientations to disagree on what normative theory should animate a particular legal provision. It is not a surprise, either, that in those instances the competing orientations that guide judges will be correlated with alternative political philosophies or orientations on the part of the judges in question.  Justice Douglas had a populist “economic decentralization” conception of “restraint of trade” for purposes of the Sherman Act; Professor and then Judge Robert Bork subscribed to an economic, “consumer welfare” alternative.  These positions undoubtedly cohered with their respective political “ideologies,” too, and likely did as well with the “ideologies” of judges who championed one versus the other understanding of how U.S. antitrust law should be structured. But those who understand how the law works—and the contribution that judges, using normative theories play, in imparting content to it—would not characterize this debate as reflecting extralegal “ideological” considerations as opposed to the perfectly ordinary, acceptable exercise of jurisprudential judgments.  Multivariate regression models are not necessary to ferret out the contribution that value-laden theories make to how judges decide these cases; judges openly admit that they are using such theories. Regardless of which President appointed these judges to the federal bench, no lawyer understands judges engaged in this sort of reasoning to be invoking “personal political preferences.”

An entirely different matter would have been presented, however, had Justice Douglas or Judge Bork proposed deciding an antitrust, labor law, free speech, criminal law or any other sort of case based on the religious affiliation of the litigants or on the contribution a particular outcome would have made to the electoral prospects of a candidate for President. The Sherman Act, the Wagner Act, the First Amendment, and even myriad criminal law statutes  all demand the use of the form of guided normative theorizing we are describing. But the bare desire to use legal outcomes in particular cases (or in large classes of them) to disadvantage those who subscribe to a disfavored view of the best life or to advance the cause of a particular political party is plainly outside the range of considerations that can validly be appealed to in the exercise of normative reasoning intrinsic to law. Whether in the form of regression coefficient correlations, law-enforcement wiretaps, or anonymously leaked emails, evidence that judges of particular ideologies were being influenced by such considerations would be a ground for intense concern.

There is a distinction, in sum, between resort to normative considerations that are internal to law and ones external to it. The former are licit, the latter illicit, from the perspective that lawyers and judges in the U.S. system of justice share of what counts as valid legal reasoning.

The “prevailing party” outcome-classification scheme used in observational studies of judicial ideology is blind to the distinction. As a result, such studies will count in their estimates of the influence of “ideology” perfectly mundane associations between the jurisprudential philosophies of judges deciding cases on the basis of normative considerations internal to law and the party of the Presidents who appointed them or the voting records of those judges and judges who feel likewise about the normative theories that inform labor law, free speech cases, criminal cases and the like.  

The correlations that these researchers report could also be capturing judges’ reliance on illicit political considerations, external to the law. But (critics point out) there is no way to know whether this is the case, or to what extent, given the indiscriminate coding of outcome variables that these studies employ.

Some candid adherents to the “ideology thesis”  have acknowledged this point.  But they have not supplied a response to what critics would identify as the significance of this concession. When observational-study proponents declare that they are finding that “ideology” accounts for judges’ decisions, they say they are measuring the extent to which those judges are not deciding cases on the basis of “law.” That is what gives this entire body of literature its currency—its “shock value.” But to the extent that the observational-study scholars are finding that judges who have different judicial philosophies will sometimes validly interpret the law to support different conclusions, then they are telling us something that already is clear— something, in fact, that the very judges whose behavior is being "explained" plainly say when they justify their decisions—and that gives no one any reason to be concerned about the quality of judicial decisionmaking.

PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (6)

Dan,

It's taken me a couple of readings to get to where I think I follow your thinking.

A) You distinguish between “jurisprudential” orientations and extralegal “ideological” considerations.
1. Both are influenced by ideology.
2. Both are 'normative'.
3. In cases where judges are required to interpret law and it's application to a particular situation (meaning especially high court decisions) the interpretation will be influenced by some normative philosophy, orientation, theory, etc.
4. The application of some degree of ideology is thus necessary for judging?
5. It would seem that you follow a minimalist rule where judges attempt to minimize the degree of normative judgement applied; one principle being to avoid extralegal considerations.
------------------------------------------------------

B) It's a mistake to judge the normative ideology of a case by looking at the the winners and losers and what ideological agenda was advanced. That analysis is not a reliable guide to the use or avoidance of extralegal considerations. Rather we should ask about the nature of the normative ideology that guided the decision.

--------------------------------------------------------

C) I assume that you do accept the idea that the judicial ideology of a judge will have some impact on how they decide cases. An example of this was the point of the contrast you made between Justice Douglas vs. Robert Bork:

Justice Douglas had a populist “economic decentralization” conception of “restraint of trade” for purposes of the Sherman Act; Professor and then Judge Robert Bork subscribed to an economic, “consumer welfare” alternative.

May 14, 2015 | Unregistered CommenterCortlandt Wilson

@Cortlandt:

The distinction is between reliance on values intrinsic to and extrinsic to *doing law*.

"interpretation" is not it, no. courts interpret the law every time they apply it. But if they rely on "ideological considerations" *extrinsic" to law, then they are not *doing law*; they are violating their duties as judges.

But when they rely on a legally valid *normative theory* to complete an otherwise underspecified legal provision, they *are* doing law. It is not remarkable to observe that those normative theories have points of contact with political ideologies; what sort of normative reasoning wouldn't & still be accepted by anyone as a sensible way to "fill in" an incomplete norm?

There's no way to construe "freedom of speech," "equal protection," "unreasonable restraint of trade," etc. w/o a *normative theory* of what those provisions are trying to accomplish. Judges will formulate competing theories that will likely correlate w/ their "ideologies." But no one who *gets* how law works thinks that judges are relying on "ideology" rather than *law* in such situations.

If they decide to construe the first amencment in a way that benefits a political party they like, or in a way that suppresses the ideas of those they disagree with -- that's a form of "ideological" decisoinmaking that is not law.

The observational study scholars say they are showing that judges rely on "iedology" & not "law" -- but what they treat as "ideological decisionmaking" confounds these very different things.

May 14, 2015 | Registered CommenterDan Kahan

" Judges will formulate competing theories that will likely correlate w/ their "ideologies." But no one who *gets* how law works thinks that judges are relying on "ideology" rather than *law* in such situations."

I'm guessing the distinction you're making is that they're not making an ideologically-correlated decision simply because they are a member of the club, but because they have constructed legal/social theories that simultaneously support one particular interpretation of an ambiguous law and justify a choice of social beliefs that happen to match an ideology.

But this assumes that ideologies are arbitrary labels for groups, that their beliefs are not founded on detailed theories and justifications that explain *why* a person chose to belong to a particular group.

People generally join an ideology *because* they think the underlying theories are correct, not vice versa. The left do not express sympathy with the poor simply because that's what the left do, but because they genuinely *feel* that sympathy, and have theories of the causes of poverty that justify that sympathy. Where such theories touch on legal issues, these will of course be differing legal theories.

So I'm not convinced you can get out of it that way!

If all you're saying is that judges have ideologically-related reasons for their ideologically-correlated decisions, then those who are ideologically opposed to those reasons will still have cause for concern. But was that what the study really showed? Didn't you show there was no correlation?

May 16, 2015 | Unregistered CommenterNiV

@NiV:

"Ideology" can influence (or be part of a path of influences) in judicial decisionmaking in two ways:

a. By being correlated with considerations that are legitimatelly considered by judges when they construe the law; and

b by being correlated with considerations are not legitimately considered by judges when the construe law.

The observational studies assert that they are showing "b" but uses measures that confound "a" & "b."

Our study clearly distinguished "b" & showed that judges were impervious when members of public -- people w/o legal training -- would not be.

I said if the observationalistis want to concede that they are measuring "a" they can. But then people will way "so what? What you are telling is is obvious from what judges say they are doing & besides that's just the way the law works. You are saying nothing more than 'judges who disagree about the law disagree about the law...'" They will also say, "the judges in those cases are not acting inconsistent with their duties-- so I'm not conerned in manner I would be if you were saying 'b' was going on-- wh8ich was what you actually were trying to convince everybody you were finding."

If you want to be concerned that judges, doing their duty & construing law, will reach results you disagree w/ through a process that reflects "a," that's fine. I'm sure that is very much on the mind of Presidents when they appoint members of the S Ct -- & Senators when they vote to confirm them or not.

Consiider what Obama said about voting against Roberts. Notice that Obama wasn't saying he thought Roberts would be putting "ideology" above the law or not deciding "neutrally" etc.

That's fine. Or least whether that's a stance a Senator should take is an interesting thing to discuss.

But it's a mistake for scholars to say that they are showing jduges are voting on the basis of "ideology" & not "law" when that isn't happening.

Political commentators & media explain every case they disagree with as reflecting the disposition of judges to put "ideology" ahead of "law" -- they then scream bloody murder about how "lawless" the "liberal" or "conservative" Justices or lower court juges are.

That sort of reaction makes no sense if we are tlaking aboutg "a." The commentators think that "b" is going on. They are encouraged to think that, too, by the sort of vandalism that Scalia enages in when he writes his hyperbolic, blow hard dissents.

Scholars should be trying to corrtect public misimpressions, by educating people about how the law works, and by presenting valid evidence that really refelcts whether jduges are being neutral in the way the "rule of law" and "liberal constitutionalism" that would be violated by "b."


The research I'm critiquing doesn't do that. It gleefully participates in the carnival of misunderstanding.

May 17, 2015 | Registered CommenterDan Kahan

"You are saying nothing more than 'judges who disagree about the law disagree about the law..."

And are you not saying that judges using legitimate considerations is legitimate?

"If you want to be concerned that judges, doing their duty & construing law, will reach results you disagree w/ through a process that reflects "a," that's fine."

That depends on whether you think arguments covered by b should be legitimate. It depends on whether you think those considerations are correct.

I'm not sure about what would constitute a 'legitimate consideration' in a legal case, so let's pick an example from an area I know better: climate science.

Suppose we ask our supposedly impartial scientist to rule on a particular question of climate science. So under (a) we might put considerations like "would give ammunition to the deniers" or "would delay political action" or "said by an inter-government authority endorsed by billions of scientists". Under (b) we might put such considerations as an insistence on replicability, validation, open publication of data and methods, and so on. One can legitimately disagree about the extent to which such standards must be insisted upon. But we know that sceptics are fanatics for scientists publishing data (so they can find something wrong with it) and therefore a judge who was strongly on one side of the debate about such criteria, a stickler for (inappropriately?) high standards of evidence say, would be biased towards the sceptics! You can easily imagine sceptics campaigning to get such judges appointed, and climate activists fighting it tooth-and-nail. Likewise vice-versa, I'm sure.

So it's not unreasonable that political figures charged with appointing judges would be interested in making sure the legitimate considerations used by judges align with their own, and that their opponents would be concerned about the political impact of the choices. They would consider it a political bias, even though it is not overtly about the politics.

People often hold their opinions on climate science because of their opinions about things like the need to publish scientific evidence, and where the burden of proof lies. People generally join an ideology *because* they think the underlying theories are correct, not vice versa. And to the extent that such debates are legitimate, it would be concerning to have them settled simply by stuffing the panel of judges with people who hold only one of the positions, instead of by winning the argument.

Do you see what I mean?

May 19, 2015 | Unregistered CommenterNiV

There is a distinction, in sum, between resort to normative considerations that are internal to law and ones external to it. The former are licit, the latter illicit, from the perspective that lawyers and judges in the U.S. system of justice share of what counts as valid legal reasoning.

Dan I think the expression of your idea here is foundering on some tricky semantics especially regarding the terms ideology (used both with and without quotes) and normative considerations.

In my thinking "ideology" and "normative considerations are, if not synonyms, then closely related in that ideology influences or even largely drives the normative considerations. Thus I would say that:
There is a distinction between a kind of ideology of law and/or an ideology that leads to a normative consideration that is internal to law and ones external to it. I am a legal layman. Am I missing some principle of law in thinking and speaking in this manner?

It seems to me that the situation is much more straightforward and easier to grasp when we accept that a limited degree of ideology comes into play in what you describe as the "normative considerations that are internal to law". As I see it there is an irreducible element of ideology that is required to judge cases where bright line law and/or legal tests cannot completely determine the ruling. Your writing goes back and forth between explicitly accepting this notion as seen in the quote below and something else.

"Ideology" can influence (or be part of a path of influences) in judicial decision making in two ways:

a. By being correlated with considerations that are legitimately considered by judges when they construe the law

The key distinction that emerges is that the role of ideology is limited, internal to the law, and even consistent with a shared perspective of what counts as valid legal reasoning.

It is a bit shocking to my naive / idealistic idea of the law to accept that a certain amount of ideology is inherent in, and indeed necessary for, judging but that seems to be the case. Not all decisions can be based on "bright line" law and legal tests therefor judges must interpret the law for the case in hand -- which is to say that judges must make judgments -- and those interpretations/judgments are influenced by their ideology / their normative considerations.

Another Topic:
The framing of norms internal to law and of a shared perspective suggest that a kind of "inter-rater reliability" should be possible. Yes? Thus, for example a Justice Douglas could review a decision by a Judge Bork and say "I might have decided that case somewhat differently (based on different legal norms and theories) but I can accept that Judge Bork's decision was reasonable and within the law".

May 19, 2015 | Unregistered CommenterCortlandt Wilson

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>