In awe of the Industrial Strength Risk Perception Measure. . . . - Cultural Cognition of Health

This is a little postscript on yesterday’s post on the CCP/APPC “Political Polarization Literacy” test.

A smart friend asked me whether responses to the items in the “policy preferences” battery from yesterday might be different if the adjective “government” were not modifying “policies” in the introduction to the battery.

I think, frankly, that 99% of the people doing public opinion research would find this question to be a real snoozer but in fact it it’s one that ought to keep them up all night (assuming they are the sort who don’t stay up all night as a matter of course; if they are, then up all day) w/ anxiety.

It goes to the issue of what items like these are really measuring—and how one could know what they are measuring.  If one doesn’t have a well-founded understanding of what responses to survey items are measuring— if anything —then t he whole exercise is a recipe for mass confusion or even calculated misdirection . I’m not a history buff but I’m pretty sure the dark ages were ushered in by inattention to the basic dimension of survey item validity; or maybe we still are in the dark ages in public opinion research as a result of this (Bishop 2005)?

In effect, my colleague/friend/collaborator/fellow-perplexed-conversant was wondering if there was something about the word “government” that was coloring responses to all the items, or maybe a good many of them, in way that could confound the inferences we could draw from particular ones of them . . . .

I could think of a number of fairly reasonable interpretive sorts of arguments to try to address this question, all of which , it seems to me, suggest that that’s not likely so.

But the best thing to do is to try to find some other way of measuring what I think the policy items are measuring, one that doesn’t contain the word “government,” and see if there is agreement between responses to the two sets of items. If so, that supplies more reason to think, yeah, the policy items are measuring what I thought; either that or there is just a really weird correspondence between the responses to the items—and that’s less a likely possibility in my view.

What do I think the “policy” items are measuring?  I think the policy items are measuring, in a noisy fashion (any single item is noisy)  pro- or con- latent or unobserved attitudes toward particular issues that themselves are expressions of another latent attitude, measured (nosily but less so  because there are two “indicators” or indirect measures of it) by the aggregation of the “partisan self-identification” and “liberal-conservative” ideology items that “Left_right” comprise.

That’s what I think risk perception measures are too—observable indicators of a latent pro- or con- affective attitude, one that often is itself associated with some more remote measure of identity of the sort that could be measured variously with either cultural worldview items, religiosity, partisan political identity, and the like (see generally Peters & Slovic 1996; Peters Burraston & Mertz 2004; Kahan 2009).

The best single indicator I can think of for latent affective attitudes is . . . the Industrial Strength Risk Perception Measure !

As the 14 billion readers of this blog know, ISRPMs consist in 0-7 or 0-10 rankings of the “risk” posed by a putative risk source. I’m convinced it works best when each increment in the Likert scale has a descriptive label, which favors 0-7 (hard to come up w/ 10 meaningful labels).

As I’ve written about before , the ISRPM has a nice track record.  Basically, so long as the putative risks source is something people have a genuine attitude about (e.g., climate change, but not GM foods ), it will correlate pretty strongly with pretty much anything more specific you ask (is climate change happening? are humans causing it? are wesa gonna die?) relating to that risk.  So that makes the ISRPM a really economical way to collect data, which can then be appropriately probed for sources of variance that can help explain who believes what & why about the putative risk source.

It also makes it a nice potential validator of particular items that one might think are measuring the same latent attitude.  If those items are measuring what you think, they ought to display the same covariance patterns  that corresponding ISRPMs do in relation to whatever latent identity one posits explains variance in the relevant ISRPM.

Now the nice thing here is that the ISRPM measure, as I use it, doesn’t explicitly refer to “government.” The intro goes like this …

and then you have individual “risk sources,” which, when I do a study at least, I always randomize the order of & put on separate “screens” or “pages” so as to minimize comparative effects:

Obviously, certain items on an ISRPM  battery will nevertheless imply government regulation of some sort.

But that’s true for the “policy item” batteries, the validity of which was being interrogated (appropriately!) by my curious friend.

So , my thinking went, if the ISRPM items had the same covariance pattern as the policy items in respect to “Left_right,” the latent identity attitude formed by aggregation of a 7-point political identity item and a 5-point liberal conservative measure, that would be a pretty good reason to think (a) the two are measuring the same “latent” attitude and (b) what they are measuring is not an artifact of the word “government” in the policy items—even if attitudes about government might be lurking in the background (I don’t think that in itself poses a validity problem; attitudes toward government might be integral to the sorts of relationships between identity and “risk perceptions” and related “policy attitudes” variance in which we are trying to explain).

I found 5 “pairs” of policy-preference items an corresponding ISRPMs.

The policy-preferences weren’t all on yesterday’s list. But that’s because only some of those had paired ISRPMs.  Moreover, some ISRPMs had had corresponding policy items not on yesterday’s list.  But I just picked the paired ones on the theory that covariances among “paired items” would give us information about the performance of items on the policy list generally, and in particular whether the word “government” matters.