follow CCP

Recent blog entries
popular papers

Science Curiosity and Political Information Processing

What Is the "Science of Science Communication"?

Climate-Science Communication and the Measurement Problem

Ideology, Motivated Cognition, and Cognitive Reflection: An Experimental Study

'Ideology' or 'Situation Sense'? An Experimental Investigation of Motivated Reasoning and Professional Judgment

A Risky Science Communication Environment for Vaccines

Motivated Numeracy and Enlightened Self-Government

Making Climate Science Communication Evidence-based—All the Way Down 

Neutral Principles, Motivated Cognition, and Some Problems for Constitutional Law 

Cultural Cognition of Scientific Consensus

The Tragedy of the Risk-Perception Commons: Science Literacy and Climate Change

"They Saw a Protest": Cognitive Illiberalism and the Speech-Conduct Distinction 

Geoengineering and the Science Communication Environment: a Cross-Cultural Experiment

Fixing the Communications Failure

Why We Are Poles Apart on Climate Change

The Cognitively Illiberal State 

Who Fears the HPV Vaccine, Who Doesn't, and Why? An Experimental Study

Cultural Cognition of the Risks and Benefits of Nanotechnology

Whose Eyes Are You Going to Believe? An Empirical Examination of Scott v. Harris

Cultural Cognition and Public Policy

Culture, Cognition, and Consent: Who Perceives What, and Why, in "Acquaintance Rape" Cases

Culture and Identity-Protective Cognition: Explaining the White Male Effect

Fear of Democracy: A Cultural Evaluation of Sunstein on Risk

Cultural Cognition as a Conception of the Cultural Theory of Risk

« Have Republicans changed views on evolution? Or have creationists changed party? Pew's (half-released) numbers don't add up ... | Main | Can someone explain my noise, please? »

"Clueless bumblers": Explaining the "noise" in a polluted science communication environment...

So the question is: what explains the resistance of some individuals to the sort of conformity effects that are the signature of cultural cognition & like forms of motivated reasoning?  

To ground the question, I posed it as a challenge to come up w/ some testable hypothesis that would explain visible "outliers" in a couple of data sets, one that correlated environmental risk perceptions and cultural outlooks and another that correlated right-left political outlooks and "policy preferences" (positions on a set of familiar, highly contested political issues like climate change, gun control, affirmative action, etc.) 

Quite reasonably, the first conjecture -- advanced with palpable ambivalence by @Jen -- was that the "outliers" are people with an independent cast of mind, ones who resist "going with the crowd" and instead form positions on the basis of knowledge of, and reflection on, the evidence.

Well, of course I have measures of "cognitive reflection" and "political knowledge."

The  "cognitive reflection test" (CRT) is considered by many psychologists and behavioral economists to be the "gold standard" for measuring the disposition to use effortful, conscious forms of information processing ("System 2") as opposed to intuitive, heuristic-driven ("system 1") ones.  

If the "outliers" are people disposed to critically interrogate intuitively congenial assessments in light of available information, then we might expect them to have higher CRT scores.

Indeed, consistent with this expectation, several papers (like this one, & also this, & this, & this too)  have now been published that use the negative correlation between CRT and religiosity to support the inference that those who are highly religious are less disposed to engage in the sort of critical reasoning associated with making valid use of empirical evidence. (These studies all seem pretty sound to me; but the reported effects always strike me as quite small & also much less interesting than those associated with the interaction of religiosity & critical reasoning dispositions.)

The standard "political knowledge" test consists of a battery of very elementary civics/current-events questions (e.g., "How long is the term of office for a United States Senator? Is it two years, four years, five years, or six years?"; "Which party currently has the most members in the U.S. Senate?  Is it the Democrats, the Republicans, or neither one?").  

One might think that such questions would have no particular value -- either that "everyone" would know the answers or that in any case they are too simplistic to tap into the mix of motivations and knowledge that one might equate with a "sophisticated" understandings of matters political.  

But in fact, "political knowledge" has shown itself to be a highly discerning measure of the coherence of individuals' policy positions with one another and with their self-reported political outlooks and party attachments.  Use of the measure has played a very very significant role in informing the orthodox political science view that most members of the public are indeed intensely non-political and non-partisan, and hence motivating the project to understand how mass political preferences manage to display the sorts of regularities and order (such as "polarization" on various questions) that are so conspicuous in everyday life.

One answer to this question is that politically unsophisticated types "go with the crowd"-- by using various types of "cues" to orient themselves appropriately in relation to others who they experience some sort of affinity.  

As a result, we might think that the "outliers" -- the individuals who resist forming the "off the rack" clusters of views that are in effect badges of membership in one or another cultural or like affinity group -- would likely be high in political knowledge, and thus less dependent on "group views" to guide them in forming perceptions of risk or positions on largely utilitarian policy questions like whether "concealed carry laws increase crime-- or decrease it."

But as plausible as these conjectures are, they are wrong.  Or in any case, if we use CRT and political knowledge to test the "independence of mind" hypothesis, the data featured in the last post do not support that account of why the outliers are outliers.  On the contrary, those measures strongly support a conjecture that is diametrically opposed to it -- viz., that the outliers are "clueless bumblers" who lack the knowledge & collection of reasoning dispositions necessary to rationally pursue an important element of their own well-being....


This is another scatter plot based on the data reported in the last post to illustrate the correlation between environmental risk perceptions and cultural worldviews.  But now I've color-coded the observations -- the individual study participants-- in a manner that reflects their scores on a "long form" version (10 items rather than 3) of the CRT.

I am a statistical model of a polluted science communication environmentAs can be seen from the color of the observations inside the "outlier circles" (which are position in the same place as last time), the "outliers" are definitely not high in cognitive reflection.  On the contrary, they consist disproportionately of low-scoring respondents.  

High-scoring ones -- those in the 90th percentile and above -- are more likely to be "conformers."  Indeed, this can be seen from the regression lines that I've superimposed on the scatter plot. The effect isn't super strong, but they show that CRT magnifies the polarizing influence of cultural predispositions on environmental risk perceptions (an impact the "statistical significance" of which is reflected in the regression analysis that you can inspect by clicking on the image to the right).

Next, consider this:

Using the data that I reported last time to illustrate the connection between right-left political outlooks and "policy preferences," I've now color-coded the respondents based on their political knowledge scores.  

me too!Again, the "outliers" are not more politically sophisticated but rather considerably less so than the conformers.  The impact of political knowledge in amplifying the fit between political outlooks (measured by a scale that aggregates study particiants' responses to standard liberal-conservative ideology and partisan self-identification measures) & policy preferences is pretty darn pronounced (and measured in this regression).

These results shouldn't be a surprise-- and indeed, @Jen's trepidation in assenting to these ways of testing the "independence of mind" hypothesis reflected her premonition that they would likely be highly unsupportive of it.

On political knowledge, all I've done here is reproduce the conventional political-science wisdom that I referred to earlier.  "Political knowledge" amplifies the coherence of ordinary individuals' policy preferences and their fit with their self-professed political leanings.  So necessarily, those higher in political knowlege will display greater conformity in this regard, and those lower less.

But why exactly? This is an issue on which there is interesting debate among political scientists.

The traditional view (I guess it's that, although the scholars who started down this road were clearly departing from a traditional, and psychologically crude understanding of mass political opinion) is that those higher in "political knowledge" are "better informed" and thus able more reliably to connect their policy views to their values.

But another approach sees political knowledge as merely an indicator of partisanship.  People who are disposed to form highly coherent -- extremely coherent -- policy preferences to gratify their disposition to experience and express a partisan identity are more likely to learn about current events, etc.  

But they aren't necessarily making "better"use of information.  Indeed, they could well be making worse use of it, if the coherence that their policy positions reflect derives from some species of biased assessment of evidence.

This is now a position gaining in strength.  It is reflected in the very interesting & wonderful book The Rationalizing Voter by Taber & Lodge.

But the impact of cognitive reflection in mangifying this form of coherence is not what one would expect under T&L's "rationalizing voter" view.

Without reflecting on the possibility of any alternative, T&L embed politically motivated reasoning in the conventional "system 1/system 2" dual process theory of cognition.  For them, the tendency of partisans to fit evidence to their political predispositions reflects their over-reliance on heuristic-driven and bias-prone "system 1." "Political knowledge" magnifies motivated reasoning because, on their view, it is a measure of partisanship, and thus of the strength of the motivation that is biasing information processing.

If this were correct, however, then we should expect partisans who score higher in CRT to show less conformity or coherence in their views.  Those who score high in CRT are more disposed to use effortful, conscious "System 2" reasoning, which reduces their vulnerability to the cognitive biases that plague system 1 thinking.  If, as T&L posit, politically motivated reasoning is a system-1 form of bias, then its effects ought to abate in those who score highest in CRT.

Or in other words, on T&L's view, our "outliers" should be high in CRT. But they aren't. On the contrary, the outliers have the lowest CRT scores!

But this shouldn't come as a surprise either, at least to the 14 billion readers of this blog.

The reason CRT amplifies cultural cognition is that cultural cognition & like forms of motivated reasoning are not a bias at all. They are elements of information processing that predictably and rationally advance individuals' interests.

What an individual believes about the impact of carbon emissions on global warming, the safety of nuclear power, etc. has zero impact on the risk that person or anyone he or she cares about faces.  That's because the influence that that individual (pretty much any individual) has as consumer, voter, public conversant, etc. is too inconsequential to have any measurable impact on the activities that generate those risks or the adoption of policies intended to mitigate them.

But if an ordinary person makes a mistake about a "fact" that has come to be viewed as a symbol of his or her membership in & loyalty to an important affinity group, then that person's life could be miserable indeed. That person can expect to be viewed with distrust by those he or she depends on, and thus ostracized and denied all manner of benefit, material and emotional.

Perfectly rational for a person in that situation (the situation is not rational--it is collectively irrational; it is not "normal"-- it is "pathological"; it is tragic) to use his or her knowledge and reasoning abilities to give appropriate effect to evidence that promotes formation and persistence in beliefs that express her identity. 

And if he or she is more adept at cognitive reflection or some other element of critical reasoning, then we should expect that person to do an even better job of such fitting.  

This, of course, is the "expressive rationality thesis" that informed the CCP studies on the relationship between cultural cognition and science comprehension.  

The studies consist of observational ones demonstrating that cultural polarization increases as people become more "science literate" & experimental ones showing that the reason is that they are using their critical reasoning dispositions--including cognitive reflection and numeracy--in an opportunistic way that more reliably fits their beliefs to the ones that predominate in their group than to the best available evidence. 

My surmise is that the "political knowledge" battery does measure (even if crudely) elements of knowledge (or at least the disposition to attain it) that individuals need to have in order to form identity-congruent beliefs on disputed issues of risk and like facts.  Political knowledge magnifies coherence in policy preferences, on this view, not because it generates a biasing form of motivation -- the T&L position -- but because rational people can be expected to use their greater knowledge to promote their well-being.

So what about the outliers?

On this account, they are sad, clueless bumblers.  They lack the knowledge and reasoning dispositions to reliably form beliefs that advance their expressive interests.

They aren't reflective and independent thinkers; they are "out to lunch."

And I bet their lives are filled with misery and solitude....

Mine is, too, when I reach this sort of conclusion.

So give me some more hypotheses.

Give me some alternative measures for "independence of mind" and alternative strategies for using them to test whether there might still be some as-yet unidentified element of critical reasoning that resists cultural cognition, or at least its complicity in the effacement of reason associated with a polluted science communication environment.

And better still, use your reason to formulate and test and implement strategies for removing the pathological conditions that divert to such a mean & meaningless end the faculties that make it possible for us to know. 



PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (24)

One possibility for an "independence of mind" test could be a modification of the Asch conformity experiments. It's tough to implement in an online survey -- you'd probably have to pretend that the person was participating at the same time as other people AND show the other participants' answers first AND provide a personal identifier to give the person a social stake in the situation -- but it might be possible. (~30 seconds of googling "asch conformity online" didn't reveal any precedent, just one of your articles submitted for a gender-bias-in-publishing study.) Since it doesn't require much social acumen to figure out the social norms in an Asch experiment (everybody's doing it!), it might test for non-conformist tendency/independent thinking independent of ability to discern group beliefs. If Asch non-conformity predicts outlier status better than CRT score, it might suggest that the polarization is due to something besides using better reasoning abilities to better discern social meanings and interpret evidence accordingly.

December 30, 2013 | Unregistered CommenterMW

This isn't pleasant, but there are other ways to look at this/questions to ask.

1. What's the statistical definition of "outlier" here? Pictorial depictions are not reliable, and the circles claiming to identify outliers in these graphs are not convincing to me. "Visibility" doesn't an outlier make. No 95% confidence intervals are show. If they were, I'm betting the shape of distributions outside them would not fall into those circles.

2. The analysis (assuming those ARE outliers) indicates that the original hypothesis is not supported. Many words are used here to squeeze the results into a modified version of the original hypothesis. That's a no-no. The results indicate that one or both of two things needs addressing. First is the likelihood that the variables and tests to define them are faulty. I'm not convinced that the definitions of knowledgeability, etc. are well-taken. You can't measure accurately with flawed measuring devices. Second, quit dwelling on these measures and accept (and act on) what the analysis tells us: these variables don't explain the outcome, and it's equally likely that something else does.

Time to get going on "something else".

December 30, 2013 | Unregistered CommenterJack C. Schultz



You tell me how you'd like to define things in order to enable analyses that will make it possible to draw valid & interesting inferences -- & I'll do it.

No, the observations described as "outliers" were not "outliers" under a statistical test (like Mahalanobis measure).

They were just observations whose risk perceptions & policy prefernces were noticably out of line w/ those with similar cultural outlooks or political outlooks.

We are interested in figuring out why some peopole don't "conform" as much as others in the formation of risk perceptions & political prefernces.

Obviously, we need to specify what counts as "not conforming." I used a rough visual heuristic -- both in framing the problem & in graphically illustrating the results of a statistical test of a hypothesis about conformity & reflection (also political knowledge)-- namely that conformity *increases* as people become more reflective (or politically sophisticated).

If you favor specifying in some other way "how far" someone has to be from what we'd expect to be considered a "nonconformer" in an interesting test of a conjecture, be my guest.

[On the 0.95 CIs -- I think you are misunderstanding the hypotheses. The whole point of the post is that the correlations between cognitive reflection & pol knowledege, on 1 hand, & risk perceptions & pol prefs, on other, had signs that were the opposite of what one would expect under the hypothesis that more reflective & knowledgeable people would be less conforming. No need to draw any CIs to figure out that that's a result inconsistent with that hypothesis. But if you want to figure out what the 0.95 CIs are, just look at the regressions -- there's more than enough info for you to figure that out, & also to see that in fact the hypothesis that more reflective/knowlegeable people form "more conforming" risk perceptions & policy prefs is supported, p <0.05, by the data]

December 30, 2013 | Registered CommenterDan Kahan


Don't you suspect that "indepencence of mind" is very domain/task specific?

I'm pretty confident that peopole high in CRT *are* "independent" thinkers in contexts in which cognitive biases of sort associated with "system 1" pop up.

But politically motivated reasoning is -- I think it is clear at this point -- not a consequence of the sort of unreflective thinking associatd with system 1.

It's hard to figure out what the mechanism was in Asch (misunderstanding? confusion?). But I suspect that whatever it is, resistance to it isn't going to do much outside of domains in which the mechanism that generates the Asch result applies. And I doubt that Asch is about identity-proective cognition1

But you are right: someone likely has tested this.

People have done lots & lots & lots of testing for dispositions like open-mindedness. The work is very informative & cool. But it's sort of astonishing how *little* progress it has really made in pinning down such a dispositon

December 30, 2013 | Registered CommenterDan Kahan


Yes, I suppose an Asch experiment wouldn't really test "independence of mind" or "open-mindedness" or any unqualified virtue like that. It does seem to me that it would test a sort of resistance to social pressure. Sure, that could be domain/group specific -- a person might be sensitive to social pressure within their social circles or identity-defining groups while resistant to it in a group of strangers -- but it doesn't seem crazy that someone willing to defy a room of confederates would also be willing to disagree with his friends on climate change. An Asch experiment wouldn't differentiate between a completely socially tone-deaf person and someone who is brave enough to defy social norms and say what he knows to be true. But do you not think it would indicate some sort of (non)conformity disposition?

December 30, 2013 | Unregistered CommenterMW


I think the Asch experiment is a little bit like "They Saw a Game": a horribly flawed design that is now treated as furnishing canonical proof of somethign that clearly is true.

There are too many explanations for the result aside from "conformity" (the most likely explanation is that the subjects figured they were misunderstanding the problem once they saw the confederates agreeing on the "wrong" answer) ... Google that-- I'm sure this has been noted by many indepedent-minded people (ones not conforming to the orthodox treatment of it as a valid experiment!)

December 30, 2013 | Registered CommenterDan Kahan


Do you have, or could you somehow get in the future, data about respondents' big five personality info, and (based on comments from the previous post) Myers Briggs stuff? I mentioned this before, and I hate to try to bring personality into a conversation about cultural values and motivated reasoning... But I'd love to see if there's any correlation. Maybe for future surveys? Throw in whatever the most widely accepted battery for the Big Five traits, just for the fun of it? (Bonus questions?)

I do think CRT just isn't measuring what we'd like to measure here... And to your point, it makes sense because cultural cognition isn't a bias- so what kind of measure could tell us how independent/reflective someone is in generalas opposed to more specific CRT-measured reasoning? Would knowing more about personality traits be at all informative even though personality doesn't tell us how a person necessarily arrives at their values/worldview? I wonder.

I can see some of these outliers (and I'm still just playing along that these are genuine outliers as we've been discussing them, I'll leave that argument to the statisticians) being bumbling idiots but not all. And aside from some of them having those unique circumstances that somehow push against and overrule an in-group's influence on worldview/risk perception, I still bet there something else going on.

I'd also love to see if there's any correlation or negative relationship with the outliers and religiosity, not CRT. Is it pretty much the same? If not, the difference is of interest.

December 30, 2013 | Unregistered CommenterJen

Sorry for going off-topic once again, but I can't follow all this conversation about statistics, so in a sense I think y'all deserve it.

I've said it before and no doubt I'll say it again: I think that what you can tell from a static view of these data is quite circumscribed. What is more instructive, IMO, is to watch longitudinal trends over time:

What explanation can there be for the trends that chart shows? Does they show something about innate characteristics of those who identify with different political groups? Do more Republicans think, in 2013 as compared to 2009, that humans have existed in the same form since the beginning, because of some change in the evidence available? Because they have become more or less scientifically literate? Because they have become more religious? Because they have studied the evidence more intensely? Because their "world view" have changed? Because their "values" have changed? Because they are more, or less, "reflective?"

I would suggest that looking for any of those explanations would be barking up the wrong tree.

December 30, 2013 | Unregistered CommenterJoshua


You tell me how you'd like to define things in order to enable analyses that will make it possible to draw valid & interesting inferences -- & I'll do it.

This has been mentioned before by myself and others - the heirarchical/egalitarian axis fails to distinguish between opportunity heirarch/egalitarians and outcome heirarch/egalitarians. For example, an opportunity-egalitarian and a outcome-heirarch white racist will both be opposed to, say, affirmative action, for different reasons, yet they both seem to be lumped into the heirarchical group.

December 31, 2013 | Unregistered CommenterFrankL


I predict you will quit your job & become a social psychologist in 2014...

What you want, essentially, is a measure for a latent disposition (construct) different from the one measured by the CC hierarchy scale. To create it, we'd need to find valid indicators for it that could be aggregated by one or another scaling technique-- like factor analysis, which I know you find quite intriguing.

As you know, I am constrained to doing analyses w/ the variables in the data set. But maybe there is something in the data that you think would do a better job capturing the disposition you have in mind.

Take a look at the long form version of the CC hiearchy scale -- or even the individualism one, for that matter. Are there some items that you think would be better than the ones that I'm using?

Also, there are various risk-perception & in some data sets & policy-preference" items. One could defensibly combine them w/ worldview items to form a scale, if one had good reason for believing that particular risk perceptions or policy prefences are valid indicators of the disposition one has in mind (although then it will be the case that the disposition measure won't be able to help explain variance in those perceptions or preferences; it will presuppose that)

Indeed, one could, in theory, combine the attitudinal measures w/ all sorts of other charactderistics that are candidate indicators -- including religiosity, gender, race, region of the country, income, education ... But in practice, it is difficult to find psychometrically satifying scaling strategies that combine diverse types of data .... If my prediction about you is right, you'll spend all of 2015 & 2016 on that & conclude that you had better climb out of the rabbit hole if you ever want to communiate w/ others again...

December 31, 2013 | Registered CommenterDan Kahan


Definitely at some point we included "big 5" measures in data sets containing the culture measures. I can remember doing it yrs ago, when we were developing the scales. I can dig up the data, probably.

No Meyers-Briggs ..

I admit that I am pretty skeptical of self-report measures of cognitive style in general, and I am aware that the Meyers-Briggs framework-- which has lots of cognitive style items -- has been criticized for lacking basic psychometric properties like reliability.

But certainly, a fair test of "independent mind" hypothesis for "nonconformity" here would require using valid reasoning/personality measures other than CRT!

My *guess* is that one coudl find something w/ enough work, but that it would pick out only a tiny class of people. The problem isn't in people or in their reasoning faculties; it's in their reasoning environment -- or at least that is how it now appears to me.

December 31, 2013 | Registered CommenterDan Kahan


I'll look @ the link, but if the claim is that there has been a change in proportion of population saying that they "don't believe" in evolution between 2009 & 2013, then the person making the claim is probably cherry picking survey results.

If the correlation w/ "republican" has changed, that is much more likely a consequence of shifts in party identification -- those definitely jump around a bit

December 31, 2013 | Registered CommenterDan Kahan


see this.

No idea why there was a differnce in pew results on repubs for 2009 & 2013. But remember that Obama was elected & both houses of congress went democrat in late 2008, when looked like we might well be heading into an economic depression. Definitely not a "representative" time to estimate % Republican in population

December 31, 2013 | Registered CommenterDan Kahan


Haven't yet seen that particular criticism, although there are plenty of others: lack of ecological validity, (some) problems with replication, and yes, some questions about why subjects conformed (but not, as far as I've seen, whether they believed they'd misunderstood the question). It's definitely a blunt instrument that doesn't get at a very specific disposition. But if there is a resistance to social pressure disposition, I'd think that experiment would test it. If you were to find a correlation between Asch-type conformity and climate change belief outlier status, you think that would be essentially meaningless? (I don't anticipate you'd find much of one because I suspect other mechanisms dominate, but we're looking for tests, right?)


I haven't followed your comments on other threads, so I apologize if I've missed some background, but I'm skeptical that you're describing actual cultural scales or groups or even dispositions. Are there really opportunity hierarchs? It seems to me that communitarians and individualists would disagree about what opportunity equality means. (Some may see affirmative action as necessary to opportunity equality, while others see it as antithetical.) And do most people necessarily fall anywhere on the outcome H-E scale? It seems to me that you may be more properly describing a focus on opportunity or outcome, and opportunity/outcome may be pretty closely related to individualism/communitarianism. Maybe I'm putting too much on my intuition, from your comment, that you see yourself as an opportunity egalitarian and outcome neutral. Please let me know if I'm missing something!

December 31, 2013 | Unregistered CommenterMW

Dan -

Indeed. In fact, I'm not sure if there is a representative time...

But party identification is not a hard-and-fast demographic characteristic like race, age or gender. Instead, it can change in reaction to news and political events from the party conventions to the Sept. 11 attacks.


December 31, 2013 | Unregistered CommenterJoshua


Sure. But then it's very misleading to talk about "Republicans changing their views" etc. People w/ views change parties & those phenomena have to be looked at together to figure out what's going (if that's what one wants to do; such details might get in the way of heaping ridicule on people one doesn't like, in which case one can be expected to bulldoze through)

but in any case, the Pew numbers don't add up. Repubs became more creationist in 2013 but Is & Ds didn't change in any meaningful way. Since there are even more repubs now than in 2009, the overall pcet of creationist, logically, has to increase. Yet they found same % creationists in 2009 & 2013.

Something's off & somebody will eventually point this out (unless everyone keeps bulldozing)

December 31, 2013 | Registered CommenterDan Kahan

I'm skeptical that you're describing actual cultural scales or groups or even dispositions. Are there really opportunity hierarchs?

I think I am describing actual scales but not very well. It is an error to lump racists and libertarians together based on their opposition to affirmative action. The opportunity vs. outcome distinction was an attempt to make that distinction, but I'm hard pressed to think of an opportunity heirarchist who is not an outcome heirarchist. However, it's easy to see a distinction between opportunity- and outcome-egalitarians.

It seems to me that communitarians and individualists would disagree about what opportunity equality means. (Some may see affirmative action as necessary to opportunity equality, while others see it as antithetical.)

Well, that's the whole problem. Communitarians have blinders on in certain direction, individualists in another. Each pieces the world together according to what they see, and are astonished at how blind the "other" is. It astonishes me that liberals aren't a little bit queasy about letting a simple majority of voters decide which race gets what, for example.

Maybe I'm putting too much on my intuition, from your comment, that you see yourself as an opportunity egalitarian and outcome neutral. Please let me know if I'm missing something!

Well, practically speaking, I am quite opportunity-egalitarian. If we have an unfortunate situation, I want to first work VERY hard to remove unequal barriers to opportunity before venturing into the dangerous territory of outcome-egalitarianism. I say dangerous because I don't want a society where a simple majority of voters is able to authorize the government to discriminate or force discrimination on the basis of skin color, race, sex, sexual orientation, religion, marriage status, or wealth, even if I agree with the outcome. Today the majority may be attacking certain minorities and protecting others, including myself. Tomorrow the list may be different. I'm very resistant, but not fanatically opposed, to outcome-based solutions.

January 7, 2014 | Unregistered CommenterFrankL

I (and others) complain about the question set used to distinguish heirarchists and egalitarians (see, but the complaints are rather vague. Here is a shot at being more clear -

The general problem with this list is that it is "framed" by a liberal worldview (see One is basically given the choice of agreeing with a liberal world view concerning various groups, or be declared a heirarch (i.e. white racist, male sexist, anti-poor, etc.). As per the liberal worldview, there are no questions designed to detect black racists, feminist heirarchs, or prejudice against the rich. Also, no attempt to distinguish between equal opportunity and quota-enforced equal outcomes. Also, some statements are based on assumed facts which introduces the possibility of id-protective cognition on the part of the questioner. The list should strive to be a test of values, not of belief in facts, and "factual" statements should be suspect at least.

As a result, non-liberals who cannot bring themselves to sign on to a liberal worldview may be incorrectly classified as heirarchs.

For each of the points below, I try to give an analysis and then some "Clarifying" questions, questions that try to elicit distinctions among non-liberals, that clearly identify the true heirarchs. I have tried to make the statements as simple as possible, trying to get a gut reaction, not requiring a lot of reflection. Some of them are "conservative framing", with the same problems as a liberal framing.

Dan, I know you want something you can "get your hands on, analytically, and I realize these added questions do not give you that. I will try to organized them into some possibly orthogonal dimensions, maybe you can see how to do this too.

The usual disclaimer - trying to be objective, no doubt not fully succeeding.

------------------ Dan Kahan's Egalitarian/Heirarchist questions ------------------

People in our society often disagree about issues of equality and discrimination. How strongly you agree or disagree with each of these statements? [strongly disagree, moderately disagree, slightly disagree, slightly agree, moderately agree, strongly agree]

HEQUAL. We have gone too far in pushing equal rights in this country.

Analysis: - If you read it carefully, any "yes" answer must be against equal rights, a true heirarchist. How can the question be framed in a way that elicits true heirarchy and not liable to raise the hackles of a non- or low-heirarchist who is not a liberal? Not too fact-based, not group specific - good.

Clarifying statements:
• Some people in this country don't deserve equal rights. (Let's clearly identify the heirarchists)
• Everyone should have equal rights, but sometimes it goes too far beyond that. (opportunity vs outcome)

HREVDIS1. Nowadays it seems like there is just as much discrimination against whites as there is against blacks.

Analysis: White-specific. The kind of white person who agrees with this is likely to be racist. If a black person said it, the interpretation would be entirely different. As a sterotypical conservative statement, it may aggravate a liberal heirarch into giving a group-loyal response. It should not be directed to a specific race.

Clarifying statements:
• People of different races should stay separate, there is little hope that they can get along. (non-specific racist)
• I don't think of myself as a racist, but I do get along better with people of my own race. (non-specific racist light)
• I don't mind helping out my friends and neighbors and I don't care what color they are. (non racist)
• Discrimination against people based on their skin color is wrong. (non racist)

EWEALTH. Our society would be better off if the distribution of wealth was more equal.

Analysis: bad question - outcome specific. Being against redistribution does not make you a person who wants the rich to lock in their position. Need to distinguish libertarians from wealth-heirarchs.

Clarifying statement:
• Being rich in this society is an indication of low character (identify anti-rich heirarch)
• Poor people tend to be parasitic on our society (identify anti-poor heirarch)
• I'm ok with some people being rich as long as they come by it honestly. (opportunity/outcome)
• Its a dog-eat-dog world and I'm ready to fight for my piece of the action. (pro-rich heirarch)
• People who work hard and manage their money deserve to be more well off than those who do not. (opportunity/outcome)
• I am in favor of higher tax rates on the rich.
• We should not have laws that favor the rich over the poor. (opportunity/outcome)
• Rich people get rich for the wrong reasons. (maybe too fact-based)

ERADEQ. We need to dramatically reduce inequalities between the rich and the poor, whites and people of color, and men and women.

Analysis: Apparently outcome specific.

Clarifying statement:
• We need to make sure that everyone has an opportunity to succeed, regardless of race or sex. (opportunity/outcome)
• Some people in this country deserve to be less than equal. (Let's clearly identify the heirarchists)

EDISCRIM. Discrimination against minorities is still a very serious problem in our society.

Analysis: fact-based, not group specific

Clarifying statement:
• Discrimination against minorities is still a very serious problem in our society, but reverse discrimination is not the answer. (opportunity/outcome)

HREVDIS2. It seems like blacks, women, homosexuals and other groups don't want equal rights, they want special rights just for them.

Analysis: not too fact-based, but this pushes a conservative into group-protective answers.

Clarifying statement
• Minorities should have equal rights, but not special rights just for them. (better distinguish egalitarians from minority-heirarchs)
• Some people in this country don't deserve equal rights. (identify heirarchists)

HCHEATS. It seems like the criminals and welfare cheats get all the breaks, while the average citizen picks up the tab.

Analysis: fact-based. There is a certain level of cheating in welfare. There's a tradeoff - are the benefits worth the drawbacks? This statement says its not. So I guess its not too fact-based. People who say the tradeoff is worth it value the effects of welfare and criminal rehabilitation rather than punishment. Those who don't, don't. Is this then, a heirarchy question? regarding criminals, probably yes. Regarding welfare - it can be, but it may be an economic decision. Clarifying statements should try to draw this distinction.

Clarifying statement:
• Welfare should help people in need, but it should not be a way of life.
• Welfare is wrong, private charities and religious organizations can provide for the poor.
• Criminals and welfare cheats should be caught and punished.
• Criminals and the poor are victims of society and the money-grubbers need to pay reparations. (conservative framing)

EDIVERS. It's old-fashioned and wrong to think that one culture's set of values is better than any other culture's way of seeing the world.

Analysis - good - not fact-based. What about cultures of violence? Phrase it in a way a conservative might be ok with it.

Clarifying statement:
• Groups of people who threaten our country need to be opposed, but we need to be careful not to lump good people in with the bad.

HWMNRTS. The women's rights movement has gone too far.

Clarifying statement:
• Women deserve equal rights, but not special rights.

ESEXIST. We live in a sexist society that that is fundamentally set up to discriminate against women.

Analysis: An fact-based statement, or, we may agree on the amount of discrimination, disagree about the value of the tradeoff.

Clarifying statement:
• We should have a society that offers equal opportunity to men and women. (less fact-based)
• We live in a society that increasingly denigrates men and male values. (conservative framing)

HTRADFAM. A lot of problems in our society today come from the decline in the traditional family, where the man works and the woman stays home.

Analysis: "yes" probably identifies a masculinist heirarch. Lets define a feminist heirarch as well. Also someone who is tolerant of different family structures.

Clarifying statement:
• We should be tolerant of stay-at-home moms and career women alike.
• A lot of problems in our society today come from the decline in the traditional family, where the man provides for his family and the woman raises the children.

HFEMININ. Society as a whole has become too soft and feminine.

Analysis: fact-based. group specific. Really bad question. You accept that society needs to be feminized or be declared a sexist. If this identifies sexists (which it does not), then "Society as a whole is too hard and masculine" would also define sexists.

Clarifying statement:
• Our society should be flexible enough to adopt masculine values or feminine values, whatever is appropriate at the time.
• "Society as a whole is too hard and masculine" (conservative framing)

EROUGH. Parents should encourage young boys to be more sensitive and less rough and tough.

Analysis: In other words, boys should be more like girls? This is guaranteed to put anyone who is not anti-male into the sexist group. One of the worst questions imaginable. Agree that sensitivity and "rough and tough"-ness are mutually exclusive, and then agree that young boys should be feminized or else you are a sexist. There are some liberals I know that would be uncomfortable with this question as is one lesbian feminist that I don't know (see ).

Clarifying statement:
• The world would be a better place if we let the women run it (identify feminist heirarchs)
• The world would be a better place if we let the men run it (identify masculinist heirarchs)
• Men and women are different and we should accept that. It doesn't mean that one is superior or inferior.
• Society would be better off if young children were not taught traditional sex roles.
• Parents should encourage young boys to be physically active with less emphasis on computer games. (conservative framing)

January 7, 2014 | Unregistered CommenterFrankL


1. So the easiest, shortest response would be, "hey, the HE scale works."

The items aren't arguments. They are indicators of a latent disposition. Accordingly, we don't care if the propositions they assert are sound or unsound or ambiguous or sufficiently attentive to distincitons that someone thinking reflectively about them might prefer to make in answering one way or other. We only care about whether they are capturing the latent disposition we care about.

Are they? Well, their psychometric properties -- reliability, apprpriate loadings of items, etc. -- show that the HE items are measuring *something* in individual respondents.

Also, the thing the scale is expected to do: predicts the sorts of perceptions of risk, and the forms of responding to information, that the theory contemplates.

also I don't think "racism" is a fair interpretation of responses to any of the items, although I do think some of the Hierarchy-valenced responses connote a kind of anger and resentment toward certain sorts of egalitarian objectives.

2. Longer answer.

You've got some good points, I agree.

At least some of the items, I think, make it very ulikely that we'd observe "hierarchical" African-Americans if they are actually out there. There was nothing deliberate about this, but I think those items reflect that being able to distinguish among H & E whites was more important to making the scale work --likely b/c there aren't, for historical reasons, that many AA hierarchs.

Actually, the difference between african american & white egalitarians is something that the scales are definitely *not* able to handle on their own. But I don't think that the problem there is something relating to the way race figures in the HE questions; it is something to do w/ what it means to be white & egalitarian vs. African American & egaltarian that is too subtle for our measuring instrument to handle w/o help (and even w/ help I don't think we are getting it!)

More importantly, you are describing what strikes you as a cultural outlook that you believe the HE & IC scales aren't likely to be picking up. I can see why you say that.

The right thing to do, of coruse, would be to try to form better scales. To start, they would have have items that "on their face" seem to reflect the outlook you hvae in mind. That's something you've taken a shot at.

Then it would have to be shown that the items, when administered to real people, "scale" in a passable way. That is, they'd have to demonstrate the sort of inter-correlation that indicators or common measures of a common latent variable have to possess.

They'd also have to be "externally validated": that is, one would need to show that the scale the items form has the sort of relationship to other things -- other attitudes & policy prefs, e.g; likely other demographic characteristics of one sort or another -- that one would expect.

Finally, the scale, validated internally & externally in this way, would have to prove it can *do* something that otherwise we are unable to do. It would have to give us some additional explanatory, predictive & prescriptive power.

I'm not saying this can't be done!

On the contrary, I'm sure scales that are valid & more useful than ours are possible & I myself am constantly tinkering with alternatives (including, most importantly, ones that aren't limited to Likert item attitudinal measures & that aren't constratined to have a simple linear relationship to the unobserved, latent dispositions).

but I am saying, and am sure, that devising scales w/ these sorts of properties is not easy. and also that the admitted possibility that one could is not a reason, in the absence of doing it, to disregard the insights one is able to get from using functional set existing scales, such as ours.

The deficiciencies in our scales -- and necessarily in all functional scales of this sort -- is that they don't teach us as much as we'd like. But they still teach us more than we'd know if we didn't have them!

3. Trying things out

As I said in previous comment, I can't "test" as items for an alternative hierarchy scale items that aren't in our data sets. But there are lots of things in our data that might be reasonably good proxies for items of the sort you'd like to develop. Exploratory efforts to form "better" scales w/ those items would help you to develop ones w/ new, and more appropriate items specifically designed for the task you have in mind.

Would you retain *any* of the HE items? either from the short or long form? which?

Are there *policy* preferences you'd expect to cohere with the relevant dispositions? We have lots of those -- capital punishment, gun control, affirmative action, etc.

You wouldn't, if you used these items as "indicators," be able to use the resulting scale to "explain" variance *in those preferences* (that would be circular).

But the resulting scale might still have value in explaining variance in other sorts of attitudies or beliefs, etc.

January 8, 2014 | Registered CommenterDan Kahan


Are they? Well, their psychometric properties -- reliability, apprpriate loadings of items, etc. -- show that the HE items are measuring *something* in individual respondents.

I am sure they are, but I think that it is not only heirarchy, but also a degree of adherence (or not) to a liberal worldview.

Also, the thing the scale is expected to do: predicts the sorts of perceptions of risk, and the forms of responding to information, that the theory contemplates.

Yes, but if the question sets are shaded towards the one latent disposition of "are you a liberal or a conservative?", then the correlations with other liberal/conservative positions, like risk perception, will naturally occur.

... likely b/c there aren't, for historical reasons, that many AA hierarchs.

OMG - This is exactly what I am talking about. The question set ignores a distinction based on the questioner's (liberal?) preconceptions. I grew up in a mixed neighborhood, and trust me, black people are statistically just like white people when it comes to race-heirarchy. By black race-heirarchy I don't mean black people who believe in black supremacy or that a black person invented the bicycle, or anything, that's too reflective and intellectual. I mean the degree of comfort and prejudice when dealing with the other race. Sure, white people dominate the show, and black people have more reason to be uncomfortable and prejudicial when dealing with the white "controllers" (bank officers, employers), but, in my experience, the percentage that lose that attitude when dealing with their economic equals of any race is about the same as white people. Much as liberals would wish it weren't so, I think that black people are much like white people when it comes to racism. It's a human trait, not a white trait.

But hey, I'm an individualist, that whole idea works for me, it considers black people as individuals rather than a coherently single-minded group. Am I pointing out id-protective cognition on your part or am I engaging in it myself? All I am saying is that the question set must at least search for that distinction, if only to reject it.

... they'd have to demonstrate the sort of inter-correlation that indicators or common measures of a common latent variable have to possess.

Yes, certainly.

The deficiciencies in our scales -- and necessarily in all functional scales of this sort -- is that they don't teach us as much as we'd like. But they still teach us more than we'd know if we didn't have them!

Yes, but if the question set is framed by your expectations, it will confirm/deny only your expectations. Efforts should be made to include what you might call useless questions posed by people who are not in your cultural group, and the same goes for me. (Not that I am carrying out any parallel study. I'm focused on the SCOTUS, where the questions are chosen by 4 or more of the 9 justices who then answer those questions. That tends to be more inclusive of what any one justice might deem a "useless question")

Would you retain *any* of the HE items? either from the short or long form? which?

Well, its very easy for me to criticize things but to improve things is not so easy. Retaining or rejecting from the question set is too simplistic a fix. If we want to take the confirmatory approach, then we have to be very clear about exactly what latent variable we are defining, and a *lot* of thought has to go into the questions. How about this for starters:

How to find a good HE question set:

• Define the characteristic with a minimum of assumptions: What is a heirarch? - Can I say that heirarchs are e.g., racists, sexists, sexual-orientation-phobes, anyone who believes that their group is in some sense "superior" to others? Assume nothing (or is symmetric) about the race of the racist, (inclusive of white and black racists) the sex of the sexist (inclusive of feminist and masculinist heirarchs) or the sexual orientation of the sexual-orientation-phobes (homophobe, heterophobe), whether you or I believe such things are important or not. HREVDIS1: "Nowadays it seems like there is just as much discrimination against whites as there is against blacks" - bad. A black racist will disagree and be defined as an egalitarian. Better: "different races cannot coexist" (racist) and "I am most comfortable dealing with people of my own race" (milder racism) and "Discrimination against people based on their skin color is wrong" (egalitarian).

• Use stereo vision: I think you and I come from two very different political-cultural groups. There are other blog-followers who come from other groups. Accept their biased way of looking at things as complementary. Everyone's question set will to a varying extent be infected with a "are you with me or against me" bias and will fail to draw what may be important distinctions in their opponents. ("They all look alike to me"). If everyone were to negotiate and agree on a definition of heirarch and egalitarian, and that a question or statement was "fair", it probably would have lower bias. If one side proposes a statement that is unacceptable to the other side, let the other side propose a counter question and include both.

• Avoid framed questions: The question set should try to avoid eliciting id-protective responses, unless that is the point of the question. That means avoid statements that are liberal or conservative talking points. The HE question set is not very good in this regard. The Meyers-Briggs personality test is very good on this point, whether you agree with its value or not. HREVDIS1: "Nowadays it seems like there is just as much discrimination against whites as there is against blacks" - bad.

• Avoid loaded questions: Statements that implicitly assume something may be rejected either because the responder rejects the implication, or because they agree with the implication and reject the conclusion. EROUGH: "Parents should encourage young boys to be more sensitive and less rough and tough". - bad.

• Keep it simple: Questions should elicit "gut-level" responses, not carefully thought out intellectual responses that might be different depending upon whether the responder is rather reflective and analytic. Unless that is what is being tested.

• Keep it clear: Statements should not have ambiguous or group-dependent meanings. HEQUAL: "We have gone too far in pushing equal rights in this country". What are "equal rights"? Affirmative action is not. Yes it is. No it's not.

• Avoid fact-based statements: It confuses the validity of the fact with a persons predisposition. Also, a statement may be rejected because the person rejects the fact or because they accept the fact but reject its importance. EDISCRIM: "Discrimination against minorities is still a very serious problem in our society". - bad

Are there *policy* preferences you'd expect to cohere with the relevant dispositions? We have lots of those -- capital punishment, gun control, affirmative action, etc.

You wouldn't, if you used these items as "indicators," be able to use the resulting scale to "explain" variance *in those preferences* (that would be circular).

But the resulting scale might still have value in explaining variance in other sorts of attitudies or beliefs, etc.

There are other dimensions that come to my mind, but I have no idea how orthogonal they are or how they would fare with validation. In my mind there is, of course, a central versus distributed control dimension. (federal/state, federal/individual, regulated/free market, authoritarian/libertarian). Gun control advocates would probably be on the central control end, gun freedom advocates on the distributed control end. The opportunity/outcome egalitarian distinction doesn't clearly fall into this paradigm, both require central control, outcome egalitarianism requiring much more detailed control. The opportunity/outcome dimension basically accepts inequalities of outcome which may result from equality of opportunity, but is not to be confused with the idea that those outcomes establish a heirarchy which must then be enforced or entrenched. Capital punishment - hmm, I have to think about that. Abortion too.

January 9, 2014 | Unregistered CommenterFrankL


I'd rather cheer you on than argue w/ you in your points. Give it a go.

I keep thinking there must be some combination of policy/worldivew/riskperception items in CCP data set that you'd regard as potentially capturing your "hierarchies"

You might also try General Social Survey & World Values Survey--tons of data to poke around in

January 11, 2014 | Registered CommenterDan Kahan

I'd rather cheer you on than argue w/ you in your points. Give it a go.

Hmm - I don't have access to 1900 people who are a cross section of the population. Very valuable. Also, if you have any general criticisms of my list of desiderata, I will accept them without comment. I generally write stuff like I know exactly what I am talking about and then go thru the annoying process of inserting provisos and making it less confrontational. I was having fun writing the list and when I was done, had an attack of laziness.

I keep thinking there must be some combination of policy/worldivew/riskperception items in CCP data set that you'd regard as potentially capturing your "hierarchies"

Ok, I will try that approach.

January 12, 2014 | Unregistered CommenterFrankL


On "cheer you on"-- seriously!

On the 1900 -- I'm trying to help!

So again, if you want to take a stab at constructing prototype FLH (FrankLHierarchies) scales w/ items in my data, I'll happily report to you what the psychometric properties are.

& the GSS & WVS data sets-- more than 1900: close to 14 billion!

true, you'll be constrained to pick items that you didn't design. But that's the way to develop scales anyway: work off of existing data sets in exploratory/probing way to get your intuitions & the way real people (there's so squirrly!) actually answer questions into equilibrium (focus groups would also be useful if you are as parochial as me & 99.9% of the world population); then try modifying items in manner that you think appropriate & pretest them on samples of 200 or so (that's plenty for purposes of assessing scale properties)

Likely you have a statistics package for computing alpha, running factor analyses, etc? or -- even more likely, from what I can tell about you -- you have written your own programs?!

January 12, 2014 | Unregistered Commenterdmk38

Hi Dan - I am a picture person, and it limits my ability to understand technical things. To me, mathematical formulas and verbal descriptions are just a way of encoding knowledge, they are not knowledge per se. They allow me to form a picture, which I then understand. To me, mathematical formulas convey much more information than verbal descriptions, but they are "words" nevertheless. As a physicist, I understand the moment of inertia of a set of mass points, so I immediately understand the PCA technique, because the correlation matrix is equivalent to the moment of inertia.

I have a vague picture of how you are analyzing things. Rather than letting the data define a set of "important" orthogonal vectors as in PCA (and then maybe making sense of those vectors), I see you choosing vectors which make sense to you and then checking how "important" they are, and checking them for orthogonality. Maybe I am wrong in that characterization. The thing is, I am lost in space. I cannot use my honed ability to visualize things using PCA, and so I feel that to start coming up with FLH (FrankL heirarchy) vectors, I would just be stumbling around in the dark.

Can you do a PCA analysis of your data, not because I am going to jump to conclusions, but rather because it will allow me to picture the data more clearly in a way that I understand?

Also, I expect your data is proprietary, been there, done that, I understand completely. But if it is not, I would be happy to play around with it. Is it proprietary?

January 13, 2014 | Unregistered CommenterFrankL

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>