Do mass political opinions cohere? And do psychologists "generalize without evidence" more often than political scientists? - Cultural Cognition of Health

Stats Legend Andrew Gelman (whose blog everyone who enjoys being surprised and who values high-quality analytical thinking should read daily) has an interesting post on Steven Pinker.

Pinker asks “[w]hy, if you know a person’s position on gay marriage, can you predict that he or she will want to increase the military budget and decrease the tax rate,” a question he answers by observing that “[p]olitical philosophers have long known that the ideologies are rooted in different conceptions of human nature — a conflict of visions so fundamental as to align opinions on dozens of issues that would seem to have nothing in common.”

Gelman responds by (1) doing some quick GSS correlations, on the basis of which he concludes that “attitudes on such diverse issues are not so highly correlated”; and then (2) attributing Pinker’s error to Pinker’s being a psychologist rather than a political scientist and thus prone to “present[ing] ideas that are thought-provoking but . . . too general to quite work,” in contrast to political scientists who “take such ideas and try to adapt them more closely to particular circumstances.”

1 . Pinker is clearly right to note that mass political opinions on seemingly diverse issues cohere, and Andrew, I think, is way too quick to challenge this.

I could cite to billions of interesting papers, but I’ll just show you what I mean instead. A recent CCP data collection involving a nationally representative on-line sample of 1750 subjects included a module that asked the subjects to indicate on a six-point scale “how strongly . . . you support or oppose” a collection of policies:

Positions clustered on these “diverse” items big time. The average inter-item correlation was 0.66. The Cronbach’s alpha—a scale reliability measure based on item covariance and the number of items—was 0.91.

This is a degree of coherence that would  make any social scientist – psychologist or political scientist – beam. The highest possible alpha is 1.0, and anything above 0.70 is usually regarded as signifying a high degree of reliability. Low reliability, measured in this way, is it’s own punishment, since it constrains the power of any sort of explanatory or predictive model involving the scale. With a score of 0.91 you can be confident that the power of your model won’t be dissipated by the noise associated with the imprecision of the observable “indicators” you are using to measure the latent variable.

The latent variable being picked up by these policy items is obviously something akin to right-left political preferences, so let’s call the resulting measure “Liberal_policy.” (Additional items cohered better with each other than with these, forming a second “libertarian policy prefernce” scale; but let’s keep things simple.)

Being able to form a scale like this with a general population sample is pretty good evidence in itself (and better than just picking two items out of GSS and seeing if they correlate) that people’s opinions on such matters cohere.

But just to make the case even stronger, let’s consider how much of the variance in liberal policy preferences can be explained by ideology.

In the same data set, there was a five-point measure for self-described “liberal-conservative ideology” and a  seven-point one for identification with the two major political parties. Those two items were also highly correlated ( r = 0.70), so I combined them into a scale (α = 0.82) coded to represent a right-wing ideological disposition, which I labeled “Conserv_repub.”

Regressing Liberal_policy on Conserv_repub, I discovered that the percentage of variance explained ( R 2 ) was 0.60. That’s high , as any competent psychologist or political scientist would tell you, and as I’m sure Andrew would agree!

Now Andrew noted that the degree of coherence in political preferences tends to be conditional on other characteristics, such as wealth, education, and political interest. Typically, political scientists use a “political knowledge” measure to assess how coherence in ideological positions vary.

I had a measure of that (a 9-item civics-test sort of thing) in the data set too. So I added it and a cross-product interaction term to my regression model. It bumped up the R 2 – variance explained – by 4%, an increment that was statistically significant.

Seems small, but how practically important is that? A commenter on Andrew’s blog noted that I tend to criticize fixating on R 2 as an effect-size measure ; my point, which is one that good social scientists—political scientists and psychologists! Andrew too!–have been making for decades is that R 2 is not a good measure of the practical significance of an effect size, a matter that has to be determined by use of judgment with relation to the phenomenon at issue.

Well, to help us figure that out, I ran a Monte Carlo simulation to generate the predicted probability that a typical “Liberal Democrat” (-1 SD on Conserv_Repub) and a typical “Conservative Republican” (+1 SD) would support “stricter gun control laws” (seems topical; this is pre-Newtown, so it would be interesting to collect some data now to follow up), conditional on being “low” (-1 SD) or “high” (+1 SD) in political knowledge.

Seems (a) like variance in political knowledge (whatever its contribution to R 2 ) can matter a lot – the probability that a high–political-knowledge Republican will oppose gun control is a lot lower than that for a low–political-knowledge one—but (b) there is still plenty of disagreement even among low–political-knowledge subjects.

I’d say, then, that Andrew is being a bit too harsh on Pinker’s premise about political preference coherence.

2. Pinker is clearly wrong—not just in his answer but in his style of reasoning—to connect this sort of coherence to “different conceptions of human nature” among people of opposing ideologies

Pinker, however, is indeed doing something very objectionable: he is engaged in rank story-telling .