follow CCP

Recent blog entries
« 3 kinds of validity: Internal, external & operational | Main | Are judges biased? Or is everyone else? Some conjectures »
Tuesday
Jul012014

Climate science literacy, critical reasoning, and independent thinking ...

Who you are, not what you know...My paper “Climate Science Communication and the Measurement Problem” features a “climate science literacy” (CSL) test. 

I’ve posted bits & pieces of the paper & described some of the data it contains.  But I really haven’t discussed in the blog what I regard as most important thing about the CSL results. 

This has to do with the relationship between the CSL scores, critical reasoning, and independent or non-conformist thinking.  I’ll say something—I doubt the last thing—about that now!

1. The point of the exercise: disentangling knowledge from identity. I’ll start with the basic point of the CSL—or really the basic point of the study that featured it and the Measurement Problem paper.

Obviously (to whom? the 14 billion regular readers of this blog!), I am not persuaded that conflict over culturally disputed risks in general and climate change in particular originates in public misunderstandings of the science or the weight of scientific opinion on those issues. 

That gets things completely backwards, in fact: It is precisely because there is cultural conflict that there is so much public confusion about what the best available evidence is on the small (it is small) class of issues that display this weird, pathological profile

Given the stake they have in protecting their status in these groups, people can be expected to attend to evidence—including evidence about the “weight of scientific opinion” (“scientific consensus”)—in a manner that reliably connects their beliefs to the position that prevails in their identity-defining groups.

But there are two ways (at least) to understand the effect of this sort of identity-protective reasoning.  In one, the motivated assimilation of information to the positions that predominate in their affinity groups generates widespread confusion over what “position” is supported by the best available scientific evidence.

Call this the “unitary conception” of the science communication problem.

Under the alternative “dualist conception,” “positions” on societal risk issues become bifurcated.  They are known to be both badges of group membership and matters open to scientific investigation.

Applying their reason, individuals will form accurate comprehensions of both positions.  

Which they will act on or express, however, depends on what sort of “knowledge transaction” they are in.  If individuals are in a transaction where their success depends on forming and acting on the position that accurately expresses who they are, then that “position” is the one that will govern the manner in which they process and use information.

If, in contrast, they are in a “knowledge transaction” where their success depends on forming and acting on the positions that are supported by the best available evidence, then that is the “position” that will orient their reasoning.

For most people, most of the time, getting the “identity-expressive position” right will matter most. Whereas people have a tremendous stake in their standing in cultural affinity groups, their personal behavior has no meaningful impact on the danger that climate change or other societal risks pose to them or others they care about.

But still, every one of them does have an entirely separate understanding of the “best-available-evidence” position.  We don’t see that—we see only cultural polarization on an issue like climate change—because politics confronts them with “identity-expressive” knowledge transactions only.

So too do valid methods of public opinion study (observational and experimental) geared to modeling the dynamics of cultural conflict over climate science.

Politics and valid studies both assess citizens' climate-science knowledge with questions that measure who they are, whose side they are on.

But if we could form a reliable and valid measure that disentangles what people know from who they are, we would then see that these are entirely different things, entirely independent objects of their reasoning.

Or so says the "dualist" view of the science communication probolem.

The aim of the “climate science literacy” or CSL measure that I constructed was to see if it was possible to achieve exactly this kind of disentanglement of knowledge and identity on climate change.

I refer to the CSL measure, in the paper and in this blog, as a “proto-” climate-science literacy instrument.  That’s because it's only a step toward developing a fully satisfactory instrument for measuring what people know about climate science. 

Indeed, the idea that there could be an instrument of that sort is absurd. There would have to be a variety, geared to assessing the sort of knowledge that individuals in various settings and roles (“high school student,” “business decisionmaker,” “policymaker,” “citizen” etc.) have to have.

But if the “dualist” conception of the science communication problem is correct, then in any such setting, a CSL, to be valid, would have to be designed to measure what people know and not who they are.

Seeing whether that could be done was the mission of my CSL measure. In that respect, there is nothing “proto-” about it.   

2. The strategy

The strategy I followed to construct a CSL of this sort is discussed, of course, in the paper.  But that strategy consisted of basically two things.

The first was an effort to create a set of items that would avoid equating “climate science literacy” with an affective orientation toward climate change. 

For the most part, that’s what perceptions of societal risks are: feelings with a particular valence and intensity.  As such, these affective orientations are more likely to shape understandings of information than be shaped by them.

The affective orientation toward climate change expresses who people are as members of opposing cultural groups engaged in a persistent and ugly form of status competition.  If we ask “climate science literacy” questions the answers to which clearly correspond to the ones people use to express their group identities, their answers will tell us thatwho they are—and not necessarily what they know.

To avoid this confound, I tried to select a set of items the correct responses to which were balanced with respect to the affective attitudes of “concern” and “skepticism.”  Scoring high on the test, then, would be possible only for those whose answers were not “entangled” in the sort of affective reaction that defines who they are, culturally speaking.

Second, I used a semantic device that has proven successful in disentangling identity and knowledge in measuring people’s positions on evolution.

As I’ve discussed in this blog (and as I illustrate with data in the paper), the true-false question “humans evolved from another species of animal” doesn’t measure understanding of evolution or science comprehension generally.  Rather it measures a form of identity indicated by religiosity.

But if one simply prefaces the statement “According to the theory of evolution,” the question elicits responses that don’t vary based on respondents’ religiosity. Because it doesn’t force them to renounce who they are, the reworded question makes it possible for religious respondents to indicate what they know about the position of science.  (The question is then revealed, too, to be far too easy to tell us anything interesting about how well the person answering it comprehends science.)

I thus used this same device in constructing the CSL items. I either prefaced true-false ones with the phrase “Climate scientists believe . . .” or used some other form of wording that clearly separated “knowledge” from “belief.”

3. The “results”

The results strongly supported the “dualistic” position—i.e., that what people know about climate change is unrelated to their “belief in” human-caused climate change.  Their position on that measures who they are in the same manner as items involving their political outlooks generally

In this way, it becomes possible to see that the cultural polarization that attends climate change is also not a consequence of the effect that cultural cognition has on people’s comprehension of climate science.

It is a consequence of the question that the “climate change” poses to ordinary citizens.

Democratic politics is one of the “knowledge transactions” that measures who one is, whose side one is on, not what one knows about the weight of the best scientific evidence.

People on both sides of the issue, it turns out, don’t know very much at all about climate science.

But if democratic politics were asking them “what they know,” the answer would be a bipartisan chorus of, “We are in deep shit.”

So climate communicators should be working on changing the meaning of the question—on creating conditions that, like the reworded evolution question and related classroom instructional techniques in that setting, make it possible for citizens to express what they know without renouncing who they are.

If you want to see how that's done, book yourself a flight down to SE Florida.  Right now.

4. The “holy shit!” part: the vindication of reason as a source of independent thinking

Now, finally, I get to what for me is the most gratifying part: the vindication of critical reasoning.

The CSL measured featured in the paper is positively correlated with science comprehension in both “liberal Democrats” and “conservative Republicans”!

Why is this so amazing?

As the 14 billion regular readers of this blog know, a signature of the pathology that has infected public discourse on climate change is the impact of science comprehension in magnifying polarization.

 The individuals whose science comprehension and critical reasoning dispositions are most acute are themost polarized.

What you *know*-- not who you are!

Experiments show that individuals high in the dispositions measured by science literacy batteries, the Cognitive Reflection Test, the Numeracy scale and the like use their reasoning proficiency to selective conform their assessment of evidence to the position that predominates in their group.

Polarization over climate change is not a sign that people in our society lack science comprehension.

It is proof how hostile the putrid spectacle of cultural status competition is to the value our society should be getting form the science intelligence it manages to impart in its citizens.

As the 14 billion regular readers know, too, this doesn’t amuse me.  On the contrary, it fills me with despair.

I was heartened in a simple “methods” sense that the CSL had the indicated relationship with science comprehension.  That the two rise in tandem helps to validate the CSL as a measure of what people know, and to corroborate the conclusion that “what do you believe about climate change?,” on which polarization increases as people become more science comprehending, measures nothing other than who they are, what side they are on.

But on an emotional level, I was much more than simply heartened.

I was elated to see the vitality of reason and critical thinking as a source of independent thinking and open-mindedness—to be assured that in fact this aspect of our intelligence hadn’t been annihilated by the sickness of cultural status competition, if it ever existed in the first place.

Remember, the CSL was deliberately designed to disentangle knowledge from identity. 

One of the central devices used to achieve this effect was to balance the items so that respondents’ affective orientation toward climate change—concern or skepticism—would be uncorrelated with their CSL scores.

Thus, to do well on the CSL, individuals had to answer the questions independently of their affective orientations, and hence with the source of them: their cultural identities.

The people who did that the most successfully were those who scored the highest in science comprehension, a disposition that features critical reasoning skills like cognitive reflection and  numeracy, as well as substantive science knowledge.

More later on this, but look: here are your Ludwicks!

This is what happens when one measures what people know.

But this is how it can be, too, in our political life.

If we can just make democratic politics into the sort of “knowledge-assessment transaction” that doesn’t  force people to choose between expressing what they know and expressing who they are.

PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (10)

"If we can just make democratic politics into the sort of “knowledge-assessment transaction” that doesn't force people to choose between expressing what they know and expressing who they are."

Oh that's all? ...millions of years of social evolution might make this a little more challenging than just convincing people that it's nice to be like those electrons who get to be both waves and particles... haha!

But I'll start telling people today!

In seriousness, though, the most important takeaway (to me) of a lot of the recent posts seems to be:

"So climate communicators should be working on changing the meaning of the question—on creating conditions that, like the reworded evolution question and related classroom instructional techniques in that setting, make it possible for citizens to express what they know without renouncing who they are."

Anyway, I am dying for more details on the Ludwicks, Dan. Tell me more. Can we find them in isolation or only in measured in relief against a group of non-Ludwicks? Can non-Ludwicks become Ludwicks?

July 1, 2014 | Unregistered CommenterJen

Good work - you are getting to the real heart of the issues. Things are never as simple as the other guys just do not understand the facts. Where all this seems to be leading is the gap in risk perception at high science comprehension is a political construction as people on both sides are being played for the political benefit. Which side is more wrong? Only time will tell but please do not suppress the analysis based on your political bias.

July 1, 2014 | Unregistered CommenterBill

@Bill: I agree. In particular that there is an "agency" problem here -- too many individuals & groups to whom others rely to warn them of cultural encroachment have opted to gratify their appetite for recognition by stimulating constant anxiety. They are the tapeworms of cognitive illiberalism.

July 2, 2014 | Registered CommenterDan Kahan

@Jen:

I know you were only kidding. B/c I know you know that the idea that this is normal b/c of evolution or otherwise ignores the denominator. A quick trip to Fla is the way to correct the biased sampling that generates the impression that this is normal.

I will share more info on what one can learn by using score on CSL as measure of Ludwick reasoning style.

July 2, 2014 | Registered CommenterDan Kahan

Haha, no no no. I would never imply that this is normal. I wouldn't want to get sent back down to CCP remedial classes... I meant the tendency of people to even be susceptible to cultural cognition in the first place, or the fact that, in these rare cases where issues do become polarized in the first place- there is a lot at work. Rare as it may be for something like climate change to become so polarized and for the communication environment around it becoming so "polluted," it's still likely the result of a complex interaction of evolutionary, social, learned, and other motivations.

Just like a meteorological temperature inversion, this may not be the norm but even in its rarity, its a complex phenomenon that can be triggered by a number of different things.. its far far easier to observe and explain than to predict and prevent...

You know all that. I'm just laughing because your point about making "democratic politics into the sort of “knowledge-assessment transaction” that doesn't force people to choose between expressing what they know and expressing who they are" makes it sound so simple! Haha.

July 2, 2014 | Unregistered CommenterJen

When the climategate tar file was release, I downloaded it and extracted the files and examined them.

The most interesting thing was the purported log of someone trying to recreate the data for an earlier paper.

From reading this, I am highly doubtful about the "scientific" agreement.

A lot of the effort was directed toward correlating data sets between stations. But it turns out that much of the original data was lost and the only thing the worker had was derrivative data sets. The thing that occured to me was "Did anyone do a sanity check where instead of correlating the data, just let each station vote and see if what the uncorrelated outcome was?"

So in the end, the argument for climate change is a suspect appeal to authority.

July 23, 2014 | Unregistered Commenteranon

Writing papers for circulation between a small group of academics is one thing. Claiming that your research is significant to the point of taxing me or otherwise reducing my modest standard of living is another.

So you make a claim and point me to five papers that support your claim. Each paper has a link to github where I can download the original datasets and the programs that move the data into the charts and conclusions. I run the code and see the same pretty picture. Perhaps I query the web sites or web services of the sources of the original data. Maybe I glance over the code to see if it looks reasonable.

It is the same requirement that professors have always made of students, "Show your work."

After I do all this, I might believe you. What politics has to do with this is not clear unless you are trying to say that Democrats are more gullible and trusting than Republicans.

Because up to this point, the papers are the only artifacts and if you ask to see the data and the methods, you are called a moron or worse.

July 23, 2014 | Unregistered Commenteranon

@Anon:

If you'd like to try one that looks precisely at the question who-- "Democrts" or "Republicans" -- is more open-minded & why, then try

Kahan, D. M. (2013). Ideology, Motivated Reasoning, and Cognitive Reflection. Judgment and Decision Making, 8, 407-424.

Data

Codebook, explanatory notes, & analysis scripts

If you send me an email, I can direct you to or send you more data & also give advice about how to do the analyses if you are using a statistics package other than Stata.

--Dan

July 23, 2014 | Unregistered Commenterdmk38

Dan,

Thanks for the courtesy of your reply.

Just to be clear, I am in no way critical of your work. It is the evidence for climate change that I wonder about.

You just had an open comment section and I took advantage of that. Too bad that the debate is so emotionally charged.

Anon

July 23, 2014 | Unregistered Commenteranon

@Anon--

don't worry-- I didn't perceive your comments to be disparaging.

July 24, 2014 | Registered CommenterDan Kahan

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>