follow CCP

Recent blog entries
popular papers

Science Curiosity and Political Information Processing

What Is the "Science of Science Communication"?

Climate-Science Communication and the Measurement Problem

Ideology, Motivated Cognition, and Cognitive Reflection: An Experimental Study

'Ideology' or 'Situation Sense'? An Experimental Investigation of Motivated Reasoning and Professional Judgment

A Risky Science Communication Environment for Vaccines

Motivated Numeracy and Enlightened Self-Government

Ideology, Motivated Cognition, and Cognitive Reflection: An Experimental Study

Making Climate Science Communication Evidence-based—All the Way Down 

Neutral Principles, Motivated Cognition, and Some Problems for Constitutional Law 

Cultural Cognition of Scientific Consensus
 

The Tragedy of the Risk-Perception Commons: Science Literacy and Climate Change

"They Saw a Protest": Cognitive Illiberalism and the Speech-Conduct Distinction 

Geoengineering and the Science Communication Environment: a Cross-Cultural Experiment

Fixing the Communications Failure

Why We Are Poles Apart on Climate Change

The Cognitively Illiberal State 

Who Fears the HPV Vaccine, Who Doesn't, and Why? An Experimental Study

Cultural Cognition of the Risks and Benefits of Nanotechnology

Whose Eyes Are You Going to Believe? An Empirical Examination of Scott v. Harris

Cultural Cognition and Public Policy

Culture, Cognition, and Consent: Who Perceives What, and Why, in "Acquaintance Rape" Cases

Culture and Identity-Protective Cognition: Explaining the White Male Effect

Fear of Democracy: A Cultural Evaluation of Sunstein on Risk

Cultural Cognition as a Conception of the Cultural Theory of Risk

« "Clueless bumblers": Explaining the "noise" in a polluted science communication environment... | Main | Wastebook honorable mentions: more "federally funded" CCP blog posts! »
Tuesday
Dec242013

Can someone explain my noise, please?

Okay, here's a great puzzle.

This can't really be a MAPKIA! because I, at least, am not in a position to frame the question with the precision that the game requires, nor do I anticipate being in a position "tomorrow" or anytime soon to post "the answer."  So I'll treat "answers" as WSMD, JA! entries.

But basically, I want to know what people think explains the "noise" in data where "cultural cognition" or some like conception of motivated reasoning explains a very substantial amount of variance.

To put this in ordinary English (or something closer to that), why do some people with particular cultural or political orientations resist forming the signature risk perceptions associated with their orientations?

@Isabel said she'd like to meet some people like this and talk to them.

Well, I'll show you some people like that.  We can't literally talk to them, because like all CCP study participants, the identities of these ones are unknown to me.  But we can indirectly interrogate them by analyzing the responses they gave to other sorts of questions -- ones that elicited standard demographic data; ones that measured one or another element of "science comprehension" ("cognitive reflection," "numeracy," "science literacy" etc); ones that assess religiosity, etc. -- and by that means try to form a sense of who they are.

Or better, in that way test hypotheses about why some people don't form group-identity-convergent beliefs.  

Here is a scatter plot that arrays about 1000 "egalitarian communitarian" (green) and "hierarchical individualist" (black) outlooks (determined by their score in relation to the mean on the "hierarchy-egalitarian" and "individualist-communitarian" worldview scales) in relation to their environmental risk perceptions, which are measured with an aggregate Likert scale that combines responses to the "industrial strength" risk perception measure as applied to global warming, nuclear power, air pollution, fracking, and second-hand cigarette smoke (Cronbach's alpha = 0.89). 

You can see how strongly correlated the cultural outlooks are with risk perceptions.  

click me ... click me ... click me ...When I regress the environmental risk perception measure on the cultural outlook scales (using the entire N = 1928 sample), I get an "impressively large!" R^2 = 0.45  (to me, any R^2 that is higher than that for viagra use in explaining abatement of "male sexual dysfunction" is "impressively large!"). That means 45% of the variance is accounted for by cultural worldviews -- & necessary that 55% of the variance is still to be "explained."

But here's a more useful way to think of this.  Look at the folks in the dashed red "outlier" circles.  These guys/gals have formed perceptions of risk that are pretty out of keeping with that of the vast majority of those who share their outlooks.

What makes them tick?

Are these folks more "independent"-- or just confused?

Are they more reflective -- or less comprehending?

Are they old? Young? Male? Female? (I'll give you some help: those definitely aren't the answers, at least by themselves; maybe gender & age matter, but if so, then as indicators of some disposition or identity that can be pinned down only with a bunch more indicators.)

The idea here is to come up with a good hypothesis about what explains the outliers.

A "good" hypothesis should reflect a good theory of how people form perceptions of risk.  

But for our purposes, it should also be testable to some extent with data on hand.  Likely the data on hand won't permit "perfect" testing of the hypothesis; indeed, data never really admits of perfect testing!

But the hypotheses that it would be fun to engage here are ones that we can probe at least imperfectly by examining whether there are the sorts of correlations among items in the data set that one would expect to see if a particular hypothesis is correct and not if some alternative hypothesis is.

I've given you some sense of what other sorts of predictors are are in the dataset (& if you are one of the 14 billion regular followers of this blog, you'll be familiar with the sorts of things that usually are included).  

But just go ahead & articulate your hypothesis & specify what sort of testing strategy --i.e., what statistical model -- would give us more confidence than we otherwise would have had that the hypothesis is either correct or incorrect, & I'll work with you to see how close we can get.

I'll then perform analyses to test the "interesting" (as determined by the "expert panel" employed for judging CCP blog contests) hypotheses.

Here: I'll give you another version of the puzzle.

In this scatterplot, I've arrayed about 1600 individuals (from a nationally representative panel, just like the ones in the last scatterplot) by "political outlook" in relation to their scores on a "policy preferences" scale.

The measure for political outlooks is an aggregate Likert scale that combines subjects' responses to a five-point "liberal conservative" ideology measure and a seven-point "party identification" one (Cronbach's alpha = 0.73).  In the scatterplot, indivduals who are below the mean are colored blue, and those above red, consistent with the usual color scheme for "Democrat" vs. "Republican."

The measure for "policy preferences" has been featured previously in a blog that addressed "coherence" of mass political preferences.

It is one of two orthogonal factors extracted from responses to a bunch of items that measured support or opposition to various policies. The "policies" that loaded on this factor included gun control, affirmative action, raising taxes for wealthy people, and carbon-emission restrictions to reduce global warming. The factor was valenced toward "liberal" as opposed to "conservative" positions.

The other factor, btw, was a "libertarian" one that loaded on policies like legalizing marijuana and prostitution (sound familiar?).

So ... what "explains" the individuals in the dashed outlier circles here-- which identify people who have formed policy positions that are out of keeping with the ones that are typical for folks with their professed political outlooks?

click me!!! C'mon!!!!The R^2 on this one is an "impressively large!" 0.56.  

But hey, one person's noise is another person's opportunity to enlarge knowledge.

So go to it! 

PrintView Printer Friendly Version

EmailEmail Article to Friend

References (1)

References allow you to track sources for this article, as well as articles that were written in response to this article.

Reader Comments (37)

As usual I have difficulty telling how much of this post is supposed to be serious.
There is no 'noise' unless you incorrectly presume that the C C theory explains life the universe and everything.

December 24, 2013 | Unregistered CommenterPaul Matthews

@Paul:

Seriously?

December 24, 2013 | Registered CommenterDan Kahan

How much data you have about reflection and metacognition? This isn't really enough to qualify as a WSMD, JA! entry (maybe later today if I have some time to sit and type it out) but my gut (I know, dirty word) would lead me, based on some of the work/research/observations I made over the last year when I was exploring that "design strategy" for scicomm, to think something about getting people to be more self-reflective is at play.

Sure, maybe those folks who are outliers are really just outliers- grew up in a weird mix of cultural influences, or experienced some life event that makes them divert from their cultural in-group on a single topic here or there, but for the rest of the gang, I'd love to see more about whether they are a) naturally more reflective people (but I'm not sure whether I like the the usual three question cognitive reflection test as the only way to test that- what are other good ways to measure how reflective someone is?) or b) have been prompted, in some way, (by some form of science communication perhaps? or whatever other events and messages have the power to prompt it), to reflect on the issue at hand, even if said person isn't as naturally, inherently, reflective. I.e. maybe some people would pass that CRT with flying colors, but maybe others who don't score very high and aren't "reflective" by nature can still be prompted to reflect or practice some level of metacognition as they learn or think about a topic.

That's my guess. To me, the most likely factor that could lead to someone adopting an attitude or perception that goes against all the other attitudes and perceptions that make up their worldview has to involve some level of individual reflection that allows them break out of the identity-protective cognitive tendencies, if only for limited scope. I'd love to see data on this, but I never have seen anything very satisfying. As an teacher, I knew the factor that influenced whether I could change a kid's mind the most wasn't the info they learned but whether I could get them to practice some level of metacognition. I never thought about that outside the classroom but the more I learn about cc and science communication, the more I see parallels.

December 24, 2013 | Unregistered CommenterJen

@Jen:

I have cognitive reflection test. Do you think it measures (even if imperfectly; or even more imperfectly than one would like) the disposition you have in mind? (In case reading/not reading: SEN COBURN, GRETA VAN SUSTEREN & ALLEN WEST: THIS BLOG POST IS NOT FUNDED BY NSF!) (Sorry about that, but I'm sure you understand the need to anticipate such odd assumptions.)

Are there other things you'd expect to correlate w/ reflectiveness? How about politiacal knowledge/sophistication?

Education?

What about religiosity? Correlated? Positively? Negatively?

December 24, 2013 | Unregistered Commenterdmk38

Yep, I know you have the CRT data; I'd be curious to see how that maps out here but I also would honestly be surprised to see a correlation because I don't know if the CRT data (which consists of what, by the way? Is it just a measure of how many of those questions are answered correctly? Is there more to it?) is really a great indicator of the more general concept of reflection that I'm wondering about here. Or maybe it is and I'm just not giving its due credit. I do think maybe if my gut (ahem, I mean hypothesis) were right, I guess we'd expect to see outliers with higher than average reflection... Because it would require some level of reflection to go against the identity protective tendencies that otherwise influence perceptions. But, I also imagine there are plenty of highly reflective people who are part of a particular cultural worldview and reflect on their values and then still to embrace them- so I wouldn't say being more reflective would make someone more likely to be one of the outliers or be part of the noisy area.. But perhaps in general we'd see just higher than average reflection in the outliers. Eh. (Shrug. I'm not sure I even agree with that but it's the best I can do at present).

That said, like I mentioned above, I think maybe I'd like to research more about our current understanding if reflection- and learn more about how people (yourself included) can measure it. Are there ways to measure not just what we think is a person's general tendency to reflect, but also how a person has come to a particular perception? For example, I may be a pretty non reflective person and answer all the CRT answers wrong, and maybe I'm a stereotypical hierarchical individualist but I had some kind of event in my life, or certain conversations, or a person I trust (fellow H-I perhaps, who himself is more reflective?) that somehow prompted me to reflect on a specific topic- and maybe I came around to a different way of thinking- only on that topic. How can we measure that? I'm not sure the data is there now but as a side note I wonder how we could measure that in the future aside from asking people to explain when in their lives they came to see something a certain way, and looking at the difference between answers like "I always had this attitude" vs "I developed this attitude after X" (or etc- that was just a random example/suggestion). I also someone's wonder if the concept of "openness" (as a part of the ol' big five personality scaled) doesn't factor in here somewhere- but that's measuring and then comparing apples and oranges (is it?)

Anyway all that aside, I'm not sure what else, if anything, would be correlated with reflectiveness (especially if measured just by the CRT.) I would conjecture that political knowledge and education might have a weak but positive correlation with reflectiveness, if anything... Because folks who are more likely to be naturally reflective may be more willing to seek out ways to further their reflective tendencies with more information, different perspectives, etc... maybe thes folks are more inclined to inductive reasoning and thus will have a higher level of understanding, and appreciation for the complexity of things like politics or science consensus. At the same time I might expect a negative correlation with religiosity, but I also wouldn't be surprised if there was no correlation because I can see there being some rather reflective people who still embrace religion- the same reflectivness that allows them to break out of the usual identity protective cognition to hold a couple outlier views may also make it possible for them to embrace religious views and yet break with the crowd on certain aspects of it.

I still get hung up on the difference, if there is one, between being a generally more reflective person, and being a person who is prompted to some level of reflectiveness about a certain topic or issue or perception. It would be nice if we could help the world all move toward being more reflective as in the former, but I think the more practical approach seems to be in finding ways to do the latter.

Do you have any data about how, why, or when people developed their current views or attitudes? Do any of the studies that involve reading essays from various "experts" give us anything to grab into in terms of whether reading those essays prompts any kind of reflection? When a person reads an opinion written by someone sharing their own worldview, but whose opinion contradicts the usual in-group values, would we call that reflection? Is there a way to measure if that kind of interaction/effect has been especially relevant for some of the outliers in the red circles above? Or is the entire question moot because only people who are inherently more reflective likely to be affected by those instances?

Hmmmmm.

December 24, 2013 | Unregistered CommenterJen

"Sure, maybe those folks who are outliers are really just outliers- grew up in a weird mix of cultural influences, or experienced some life event that makes them divert from their cultural in-group on a single topic here or there, ..."

That would be my guess as what is operative. We all have various cultural or group identifications, and "outliers" are merely those who have a different hierarchy of indentifications,

"but for the rest of the gang, I'd love to see more about whether they are a) naturally more reflective people (but I'm not sure whether I like the the usual three question cognitive reflection test as the only way to test that- what are other good ways to measure how reflective someone is?) or b) have been prompted, in some way, (by some form of science communication perhaps? or whatever other events and messages have the power to prompt it), to reflect on the issue at hand, even if said person isn't as naturally, inherently, reflective. "

But why wouldn't being more reflective be, also, a result of mixture of cultural influences and unique life experiences? Would being more reflective just be some indicator of some intrinsic quality rather than external influences? And would there be an assumption that (at least for some people) being an outlier is a function of being more reflective?

Let's take Dan as an example. My guess is that he'd likely show up as something of an outlier on many of these measures of social and/or political orientation, and that he'd score as "reflective" on one of the assessment tools used to measure reflectiveness. But I'd guess that he'd likely have many colleagues that would gain a similar measure of reflectiveness but not show to be as much of an outlier on social/political orientation. Or let's take FrankL. My guess that w/r/t social/political orientation, he would not show up as a very significant outlier - but he certainly seems to be a reflective person.

December 24, 2013 | Unregistered CommenterJoshua

@Joshua:

There's a difference between "outlier" & "misfit."

December 24, 2013 | Registered CommenterDan Kahan

Jen -

I didn't see your second post before I wrote mine, and I think that there is some overlap...

"because folks who are more likely to be naturally reflective may be more willing to seek out ways to further their reflective tendencies with more information, different perspectives, etc... maybe thes folks are more inclined to inductive reasoning and thus will have a higher level of understanding, and appreciation for the complexity of things like politics or science consensus. "

I suggest a diametric, perhaps balancing phenomenon for an equal number of people - whereby some people who are more driven to confirm their identifications (and in that sense, are less "reflective.") are motivated to gain more information.

"For example, I may be a pretty non reflective person and answer all the CRT answers wrong, and maybe I'm a stereotypical hierarchical individualist but I had some kind of event in my life, or certain conversations, or a person I trust (fellow H-I perhaps, who himself is more reflective?) that somehow prompted me to reflect on a specific topic- and maybe I came around to a different way of thinking- only on that topic."

In the same sense that I am skeptical that the ways that we measure intelligence are biased (i.e. weighted towards some forms of intelligence like logico-mathmatical as compared to interpersonal or kinesthetic or artistic), so am I skeptical of attempts to measure some generic attribute of being "reflective." How do we know that being reflective transfers across domains? Aren't we are all variably reflective in different areas of our?

And then even beyond that, I think there is a parallel with the nature/nurture question for intelligence; is someone being "more reflective" likely to be a manifestation of some intrinsic quality or an outcome that results from external developmental influences? And like with nature/nuture, is it even possible to determine an answer? How can we really control for any possible environmental influences to determine that someone is "naturally reflective"? How could we possibly know whether someone is more reflective because that's just who they are or because of external influences or life experiences?

December 24, 2013 | Unregistered CommenterJoshua

@Jen:

I can certainly post some follow up data w/ CRT. I could do some other things too -- like religiosity or education. But I'm eager to avoid preempting informed conjecture by tossing a bunch of co-variate darts at the board. "Trying this, trying that" would not only risk "overfitting." It also would tend to obscure the important point that an "outlier" disposition is probably going to be something distinct from the characteristics or apptitudes or traits that any particular covariate measures. Probably it makes sense to try to construct a latent variable representation of the dispositoin w/ multiple indicators -- ones that might by themselves completely miss the mark but do something interesting when co-occuring w/ others in appropriate patterns ... But I know you aren't proposing throwing darts, and certainly seeing how CRT figures in will help to calibrate & orient folks

December 24, 2013 | Registered CommenterDan Kahan

Dan -

"There's a difference between "outlier" & "misfit.""

I don't follow (except if that's just a self-depricating joke).

December 24, 2013 | Unregistered CommenterJoshua

Haha!

Indeed. @Joshua, that's my overall concern anyway. Is reflectiveness innate? Is it a learned behavior? If so, when and from whom? Is it part and parcel with the rest of one's culturally-influenced existence? Would a correlation be redundant? Is it something separate, in the same way personality traits (like the "big five") are not the same as values or the same as perceptions?

I do the same thought experiment with myself and many I know (so and so is a very reflective person, etc etc) and that's what drives my gut. Honestly what it boils down is that reflectiveness (whatever it seems to be made of or where it develops from) seems to have the most potential (in my eyes) to counter or overwrite identity protective cognitive tendencies. Who knows.

December 24, 2013 | Unregistered CommenterJen

@Joshua

Let's take Dan as an example ..."

December 24, 2013 | Registered CommenterDan Kahan

Dan -

I got that you were commenting on my suggestion to use you as an example, but didn't know if you were being serious that there's a difference between "outlier" and "misfit" or were just making fun of yourself as a misfit.

I like to think that irrespective of whether being reflective is in some measure intrinsic, being reflective can, potentially, be a learned behavior - if a "teacher" has the skills necessary to create an instructive environment conducive to such learning taking place. I don't think that it is something that can be taught didactically - but something that people can learn through experience if they are guided through the right kinds of experiences.

But even if I am right about that, a follow-on question for me is, as I said, whether the attribute of being reflective is transferable across domains. I see much reason to believe that it isn't, particularly. I think I can see many examples where people's degree of reflectiveness is variable across domains. To the extent that person A might be more generally reflective than person B, I would think it would most often be a function of person A having more varied life experiences.

December 24, 2013 | Unregistered CommenterJoshua

@Joshua:

It's clear the CRT measures a kind of reflection -- the sort that consists in recognizing that some intuitively appealing response is not supported by evidence on hand. The sort of non-reflection it contemplates is the sort that is associated with cognitive biases of various sorts.

But would you expect that sort of reflection to make someone more readily form culturally discordant perceptions of risk (or policy preferences)?

We can avoid some of the difficulties you are raising by stipulating that the disposition contemplated by the "reflection hypothesis" is one that consists in some conscious interrpution of the mental processes that otherwise generate culturally motivated cognition. Then we wouldn't have to commit ourselves to saying tendentious things like " 'reflection' is just one thing" or "reflection is not domain specific" etc.

But we shouldn't stipulate that whatever it is that causes people to form culturally discordant risk perceptions is reflection of the sort I just described.

If we do that, we'll be ruling out (or intefering with our understanding of) important alternative hypotheses.

The most immediately obvious one is that the "outliers" are clueless bumblers.

December 24, 2013 | Registered CommenterDan Kahan

I like to think that irrespective of whether being reflective is in some measure intrinsic, being reflective can, potentially, be a learned behavior - if a "teacher" has the skills necessary to create an instructive environment conducive to such learning taking place. I don't think that it is something that can be taught didactically - but something that people can learn through experience if they are guided through the right kinds of experiences.

Agreed.

I think this is where we may end up down a rabbit hole, though, in trying to define different types of reflection here, (as much as I'd like to), for the points Dan made in his most recent reply. I do agree though, even just from my brief years (only 8 of them to be exact) spent as a teacher- I had students I would definitely describe as being generally very reflective people- able to instinctively forgo (or even just delay) the development of culturally motivated views in favor of a more empirically reasoned (or if not empirically, more personally/independently reasoned) view. These kids would be the ones I assume score high on the CRT. But I also had students who I would imagine would do the opposite- and yet through exactly the process you describe- the design of learning experiences (i.e. I couldn't tell them, or teach them, to be reflective- I designed experiences for them to have- {and now I try to explain how that relates to my new career path in user experience design and people still look confused, haha}), I was able to prompt more reflection, situation by situation. I can't say whether I was able to move the needle for any of those students in terms of their overall innate reflectiveness- would they score better on a CRT afterward? Or did I just give them the appropriate experience and cognitive tools to allow for increased reflection in a certain time and place on a particular topic (kairos?)

The CRT doesn't measure that situation or topic (or domain, to take your points), specific reflection,


It's clear the CRT measures a kind of reflection -- the sort that consists in recognizing that some intuitively appealing response is not supported by evidence on hand,

but it would definitely be nice to find a way to measure this more specific type of reflection.

It's sort of like the difference between someone's ability to weigh all options/info and come to an informed decision, vs. someone's ability to change their mind on something they already have an opinion. These are two types of mental processes, but I think we refer to both kinds of reflectiveness here.

Either way, I agree with Dan here:

We can avoid some of the difficulties you are raising by stipulating that the disposition contemplated by the "reflection hypothesis" is one that consists in some conscious interrpution of the mental processes that otherwise generate culturally motivated cognition. Then we wouldn't have to commit ourselves to saying tendentious things like " 'reflection' is just one thing" or "reflection is not domain specific" etc.

If we stick to this kind of hypothesis, I'd be very very curious to see what can shake out of the data Dan already has access to.

December 24, 2013 | Unregistered CommenterJen

@Dan - think it would be a good idea to expand the number of factors from two to three and maybe four. Take three to start with. Maybe those outliers are very understandable as being different along a third dimension, and only appear odd because they have been shoe-horned into two dimensions. I think it would be interesting to analyze the data using three factors, and then try to figure out what characterizes that third dimension, the way individualist-communitarian and heirarchical-egalitarian characterize the first two. Also, examine if the outliers in the 2-factor plot are "explained away" by the third factor.

BTW - Regarding factor analysis, I'm still trying to understand how the data is analyzed, how the factors are determined. Reading Rummel, I read where the usual method is principle axis method - just eigenanalysis of the covariance matrix. (or is it the correlation matrix?). Another method is where the diagonal of the covariance matrix is modified to be a "communality measure". Which one is used for these analyses (or is it something different altogether?) - thx

December 24, 2013 | Unregistered CommenterFrankL

@FrankL:

Identifying some additional dimension of individual variance -- maybe in style of cognition, as we've been speculating, but perhaps in some refined or alternative specification of the sort of group affinities that motivate formation of the relevant perceptions and policy stances -- is exactly what were after here. Do you have a candidate dimension, including a theory about how it works and a set of likely indicators? As I mentioned to Jen, I'm less keen on just poking about in an exploratory way, since I'm fairly confident that any number of meaningless correlations could be found that would "increase" R^2. I'd rather hypothesize & test.

On factor extraction -- there are quite a number of different strategies. The most important one is between "principal component" and any form of "common factor analysis." Mathematically, former includes both common & unique variance associated with all the indicators, latter only the common variance (the "communalities" are what is used in the matrix from which factors extracted). Among CFA there are different approaches using alternative forms of general linear model -- maximum likelihood, ordinary least squares, etc. -- for estimating the factor loadings & determining error. Principal axis, in my impression, is pretty much default choice (it does use communalities in matrix).

These sorts of things are in fact the focus of holy wars among hard core psychometricians. In my experience, the differences between all these methods -- principal component & common factor as well as various forms of latter -- makes little difference. Likely that's because I tend to work with highly reliable scales comprising highly comprable forms of indicators.

Indeed, I am suspicious when I see people in heated arguments over the extraction method. They often are trying to use math as a substitute for theory & inference as opposed to a device for structuring and disciplining them. It seems weird to me to try to resolve issues about measurement independently of the theoretical & practical aims that are motivating the measurements

One can use either covariance or correlation matrix. Latter standardizes, which I think is pretty sensible, since otherwise there can be artifactual weighting of items with higher variance. (Indeed, when I use aggregate Likert scales, as I do often, I always standarize the items first.)
Often this becomes an issue when one is trying to extgract factors from indicators that use diverse scales or measures.

Indeed, a major limitation of factor analysis is that it can't handle binary or non-ordinal categorical data, can't combine diverse or mixed forms of data in a sensible way, and can't be used to model non-linear relationships.


December 25, 2013 | Unregistered Commenterdmk38

Identifying some additional dimension of individual variance -- maybe in style of cognition, as we've been speculating, but perhaps in some refined or alternative specification of the sort of group affinities that motivate formation of the relevant perceptions and policy stances -- is exactly what were after here. Do you have a candidate dimension, including a theory about how it works and a set of likely indicators? As I mentioned to Jen, I'm less keen on just poking about in an exploratory way, since I'm fairly confident that any number of meaningless correlations could be found that would "increase" R^2. I'd rather hypothesize & test.

If a strong correlation is found, doesn't that imply "meaning", almost by definition?. I mean, if one exploratory factor reduces variance by 90%, my impulse is to ask "what IS that factor?" rather than hypothesizing some factor that makes perfect sense to me, and is therefore biased to my worldview, and celebrating a 75% reduction in variance by that posited factor. In other words, when I "hypothesize and test", my hypothesis will be biased, that hypothesis will make sense to me, it will comport with my world view. What seem to me to be "meaningless correlations" may just be trying to tell me my world view is flawed. I will in effect be paying the price of 90-75=15% variance in order to have the posited factor protect my worldview - i.e. not be "meaningless" to me.

Also, the "meaningfulness" of a particular factor is suggested by the eigenanalysis. If two eigenvectors give a 40 and 30 percent reduction and a third gives a 20 percent reduction, that third factor is trying to tell me something, and I see my job as trying very hard to make sense that third factor, rather than rejecting it as meaningless because I can't presently make sense of it. I want to take that 20% factor and look at the specific questions that create that loading, the questions which on one end of that third axis tend to be answered one way and another way on the other end. Then I ask, "what is the essential difference in the world views of the people who disagree with each other on this point, this factor? And refuse to throw my hands up and say it's meaningless.

I mean, I understand that a 90% factor may well be exactly explained by the 75% hypothesis, with the other 15% being objectively meaningless noise, but the potential for the introduction of bias with that hypothesis, well, I am relatively risk-averse, I guess. I don't trust my own or anyone else's ability to objectively hypothesize here, I would rather let the data teach me objectivity rather than suppose I have it. It's bad enough that the questions only dissect things according to the world view of the questioner(s), I don't want to introduce another opportunity to make the data conform to my world view.

December 25, 2013 | Unregistered CommenterFrankL

@FrankL

I'm worriedyou are forming a numerological interpretation of factor analysis! Nothing follows from covariances by themselves; they are meaningful only in relation to some inference they support, the validity of which necessarily turns on something other than covariances.

But in any case, I think you need to leave factor analysis aside for a moment. It's not actually what this problem is about (I don't think!).

We are trying to *explain* variance in risk perceptions &/or policy preferences-- not variance in the *indicators* of some latent variable being measured with factor analysis.

The "2 dimensions" reflected in the scatter plots were not "extracted" from a "common factor analysis" -- as "hierarchy-egaltiariansim" or "individualism-communitarianism," say, or "public safety" & "deviancy risk" were.

The outcome variables here -- perceptions/preferences -- were formed from one set of indicators and the explantaory variables -- cultural outlooks & pol. outlooks -- were indepenently formed from another.

The independence of the derivation of these two measures is essential, actually, for the type of question we are trying to answer.

We want to know, essentially, "who fears/prefers what & why?" The answer will be unsatisfying if the "who" is derived from the "what": it would be wholly unedifying -- completely circular-- to say "a high degree of risk concern is explained by one's membership in the group of people who have a high risk concern, and a low one by being in the 'low concern' group" etc. (Imagine how confused people would be if they professed to find that helpful!)

Well, here one construct (cultural or politcal outlook) does indeed turn out to have a very strong correlation w/ another distal or analytically separate one (risk perceptions/preferences).

The meaning we get from that is *not* from the correlation. Rather the correlation gets meaning from the theory that led us to expect (or not) one might be involved in a causal process that generated the other; the correlation gives us more reason to believe that theory is getting things right.

But there is "left over variance." What "explains" that?

Could be some sort of disposition of reflection or independence of mind. Could be cluelessness. Could be something else.

But it *can't* be anything that is derived analytically-- say by factor analysis -- from the indicators of the very thing (risk perceptions or policy prefernces) we are trying to explain.

So come up with a theory-informed hypothesis about the "who" that we observe in the "outlier" circles. Then we'll form indicators -- ones that are distinct from the ones used to mesure th3e risk perceptios or policy preferences we are trying to explain -- and we will see if R^2 goes up, and more importantly, whether we are predicting the relevant "who" -- the outliers -- and not merely explainign differences of degree among the "conformers."

We do have to have some theory, though, for I'm sure if I start plugging variables in in promiscuous but theoretically unmotivated way, I'll ring up a "really impressive!" but meaningless R^2 from the simple chance covariance of those things w/ my outcome variables.

December 27, 2013 | Registered CommenterDan Kahan

Hi Professor Kahan,

I'm a fan of the blog, and hope your holiday season is going spectacularly.

Aside from recently going back to school so I can work on interesting questions like the one posed, I've been working in high frequency algo trading for a little over four years. So I have a bit of experience with extremely noisy data, but admittedly probably not a rigorous enough background in statistical theory to give a suitable answer (most of my work as a quant involves market mechanics). Nevertheless, I will give it a shot. I see three things that might be interesting.

Contact Hypothesis

You obviously know the ramifications of this one more than I. But it's quite possible that these outliers routinely associate with people outside of their normal cultural outlooks. An example would be a person who works in an industry overrepresented by opposing cultural views. Their beliefs might be more flexible due to prolonged exposure to water-cooler chatter.

There is no intragroup relationship

First, the language of the post makes it sound as if these outliers are the reason the r^2 is 0.45 instead of 1. We know that's not necessarily true. I'm curious to know the r^2 after removing the areas you have circled. I suspect the fit isn't going to be remarkably better.

Next, after removing the outliers, examine each cultural group (left and right of 0) individually. I suspect the intragroup relationship is very weak.

If we can't explain variance intragroup then I think it follows that chasing group outliers will lead us to OverFitsville

which brings me to

Explaining these outliers has no out of sample predictive power

Coming from the land of large data sets and the "artistry" of sample selection, I have an unfortunate close familiarity with the deceptiveness of free parameters.

I propose that whatever variables we use to explain the outliers in this sample will have no future predictive power to determine a person's environmental risk perceptions.

December 27, 2013 | Unregistered CommenterZachary David

@Dan

I'm worriedyou are forming a numerological interpretation of factor analysis! Nothing follows from covariances by themselves; they are meaningful only in relation to some inference they support, the validity of which necessarily turns on something other than covariances.

I agree completely, covariances are just mathematical curiosities unless they are associated with "meaning". The question is how to we associate the two? Ok, maybe I was being too hardcore, but the fact remains that the process of associating meaning will introduce the bias of the person doing the associating. Hypothesizing and testing introduces more bias than searching for the "meaning" of a exploratory correlation, but there is probably true noise in the exploratory correlation, much less so in the H&T approach. So I understand and am concerned about the potential for "numerology" as you say. I am more comfortable with noise I can sense than with my bias that I cannot sense. I think the best thing is to try to sense meaning from the exploratory correlation, then hypothesize that meaning and test for correlation - blend the two. I mean, I want to use the data to make me think outside my world-view box, and the hypothesis-test approach does not do that as well.

The "2 dimensions" reflected in the scatter plots were not "extracted" from a "common factor analysis" -- as "hierarchy-egaltiariansim" or "individualism-communitarianism," say, or "public safety" & "deviancy risk" were.

The outcome variables here -- perceptions/preferences -- were formed from one set of indicators and the explantaory variables -- cultural outlooks & pol. outlooks -- were indepenently formed from another.

I need to understand this - as an example, what was the process that yielded the heirarchy-egalitarian (HE) and individualist-communitarian (IC) factors? I start out thinking that a correlation matrix was calculated, "communality" values substituted, etc. etc. until a factor matrix was found, and then rotated, and the HE and IC axes identified. I must be missing something, because then I don't understand how the risk/deviancy factors are found. Maybe the two are derived from a subset of the questions? Can you give me some details on the mathematical process, because I can't exactly understand the way in which meaning is associated with these dimensions.

So come up with a theory-informed hypothesis about the "who" that we observe in the "outlier" circles. Then we'll form indicators -- ones that are distinct from the ones used to mesure th3e risk perceptios or policy preferences we are trying to explain -- and we will see if R^2 goes up, and more importantly, whether we are predicting the relevant "who" -- the outliers -- and not merely explainign differences of degree among the "conformers."

Well, I looked at Zachary David's response, and it seems to me that he is assuming lack of meaning, and then proving it. Maybe my reading was too cursory, though.

Jen's idea that they are more reflective (and therefore more complex) sounds possible to me, but people who tend to go against their group would tend to be individualists. I use Mathematica for a lot of calculations, and it has an option to produce a "3-D plot" which is really a 2-D plot that you can rotate with a mouse. I think a plot of the risk/deviancy versus both the HE and IC axes might show the outliers to be more individualistic.

I also like the idea that the outliers are in a mixed environment and not under as much pressure to conform to a particular world-view. Their deviation may simply be the result of seeking the center of their mixed environment, or being forced into reflection due to the conflict in their mixed environment. I wouldn't know how to detect that.

Another aspect of risk perception that sticks in my mind is framed in terms of the theory of evolution (I think all studies of human behavior should aim for this). In a society where numbers matter, where the larger your group of genetically related individuals (tribe), the more reproductively fit are the members, males will tend to be less risk-averse. Males are much more expendable than potentially reproductive females. If you have ten women and one man, in a year you can have ten women, one (tired) man, and ten babies. If you have ten men and one woman, in a year you have ten (angry) men, one woman, one baby. When numbers matter, a tribe with congenitally risk-prone males will grow faster than one without, all other things being equal. Selection against this trait is proceeding apace in crowded (urban?) environments, but much less so in rural environments. So who are the outliers? Reflective people who understand that global warming does not pose the same kind of personal risk as a hunting accident, or joining the armed forces? Rural women? Urban males? Does the question set resolve whether a person was raised in a rural or urban environment?

Another thing that fascinates me is the Briggs-Meyers personality test (see http://www.personalitypathways.com/type_inventory.html ) Does anyone think that this 4-factor analysis of personality has anything to offer? I can see all sorts of familiar world views in the 16 possibilities. Is ISFJ a conservative? Is INFJ a liberal? Am I an INTP?

December 29, 2013 | Unregistered CommenterFrankL

Hey Frankl, I did a much longer response that you'll find just above the "reader comments" section.

"Response: Prediction v Inference: On Kahan’s “explain..
by Zachary David at Numbers and Words on December 28, 2013
I thought about your question some more. I simulate your data set and argue that the problem is misspecified."

It links to an external post on my blog where I simulate Kahan's data set in Matlab from independent normal distributions, apply the OLS and obtain the same r^2. Then I explain why there's a problem with the way the challenge is framed.

We have to re-design the way this is presented first.

It'd be best as a classification tree problem

December 29, 2013 | Unregistered CommenterZachary David

@Zachary David - The link you provided is what I read. I had to add the disclaimer that maybe my reading was too cursory, because I did not understand some of the points, and formed my tentative opinion based only on what I thought I understood. You stated that there was no apparent intragroup relationships in the two groups, and developed a model without intragroup relationships which yielded statistically similar results. I would conclude that the "real" data is consistent with a lack of intragroup structure, but that your model does not exclude such structure. For example, only a single axis in cultural space was used in displaying the "real" data. (The "left-right" axis of heirarchical-individualist vs egalitarian-communitarian"). The orthogonal axis (heirarchical-communitarian vs. heirarchical-communitarian) was omitted, which means that there is group structure, but it has been ignored, probably because that dimension did not yield much correlation with the risk perception. If you accept at face value that the question set draws every major distinction (I do not) and that the two factors (HE and IC) are the only two significant coherent factors to be drawn from the question set (I plead ignorance, since I don't completely understand how they were arrived at), then yes, your conclusion is valid.

I have a nagging suspicion that I am missing your point, hence my disclaimer, because I don't quite get the prediction/inference distinction, and I don't understand why some missing distinction, perhaps not even drawn by the question set itself, could not account for the outliers.

December 29, 2013 | Unregistered CommenterFrankL

FrankL, ok now I see what you're saying. When Kahan collapsed the 2-axis compass into 1, we say that he could have "lost information." I suspect he did it this way so that he could perform a simple OLS.

Now that we recognize that the OLS is not the correct statistical test, we could decouple the cultural outlook variables into simple classes without losing information. Then a machine learning classification tree will pick up on that fairly easily.

Let us denote the classes of a person in set notation.
examples:
Person1: {Hier , Ind , Risk- };
Person2: {Egal , Comm, Risk+};

If we were to feed in all the samples as such to a classification tree and prune to 2 levels, it would probably yield the exact same results as collapsing to the single-axis test we currently have.

Now the fun part: since we have no specified the problem as one of prediction, we have access to all sorts of fantastic machine learning algorithms. And since these ML algorithms can use different types of cross-validation to minimize overfitting there is No Hypothesis Necessary. You can put in many classes.

December 30, 2013 | Unregistered CommenterZachary David

@Zachary:

I need to reflect more on your excellent points & on the cool simulations you've done.

But you are definitely right that increasing the R^2 won't necessarily or even likely explain the "outliers." There's plenty of variation packed in all around the "conformers," and likely it will be easier to find things that explain it among them. The observations that are "way off" -- real outliers!-- are more likely to be either screwed up people or loafing study subjects (one nice thing about N = 2000 is that one doesn't have to try to fish out & remove "misbehaving" subjects-- there are enough of them to "cancel" out one another's noise -- & thus can avoid the obvious hazard of confusing observations that don't fit one's expectations with ones that are "misbehaving").

Also agree that anything we figure out this way would need to be validated by being applied to another sample. As I mentioned in some of the comments, I don't want to keep "throwing covariate" darts at the data until we've hit every observation, for as you point out, that will inevitably generate overfitting -- or correlations w/ noise & not signal!

But do you agree that the disciplined form of exploration benig proposed/undertaken needn't have that outcome?

I'm saying "no fishing expeditions -- give me a good theory & testing strategy & then I'll take a look."

What we come up w/ that way has a decent enough chance to be real to make it worth the time taking a look & worth the effort to replicate w/ another data set if we find something.

Indeed, one reason -- in addition to just fun -- for inviting others to conjecture about my data (e.g., here is that I figure getting others, esp those less "committed" to my own hypotheses, makes it more likely I'll flush out something reliable than if I undertake such a thing all by myself.

More later!

December 30, 2013 | Registered CommenterDan Kahan

@FrankL:

Will need to have more time to work through your points too.

But on the HE/IC: 2 orthogonal factors extracted (using max likelihood estimation -- but woudn't be different if used principal axis or OLS or even "principal component" instead of "common factor"!) from the 12-item "short form" battery. Essentials here.

The "environmental risk perception scale" was just an aggregate Likert one -- I normalized the scores on the indicated risk perception items (which were all "industrial grade", summed them, and then standardized the sum.

For the "policy preferences," I applied factor analysis to a bunch of 6-point support/oppose likert policy preference items. 2 came out -- one that seemed recognizably "liberal-conservative," which is the one I used for this analysis, and another that was recognizably libertarian. Actually, I've seen political scientists using this approach a lot for forming an "ideology" measure-- it has a lot more power than a single "liberal-conservative" item.

But as I said, the outcome measures -- the indicators used to form the risk perception scales & the political preferences scale -- were not included in the factor analysis used to dervie the cultural outlooks (or in the scale used to measure political outlooks, which, again, was another aggregate Likert one, based on liberal-conservative idoelogy & political party id).

If the outcome measure variables were included in the scales, then the "explanatory power" of the scales with respect to the outcome variables would be unimpressively, uninformatively circular! (But I've actually seen people do this...).

I'll have to see what you & @Zachary were saying about collapsing HE/IC & why OLS was not good statistical test.

But in fact, I just "collapsed" HE/IC so that I could array the observations on a single dimension for the scatter plot. The regression analysis treated the two cultural worldview factors as separate continuous measures.

However, I did only include in the scatterplot subjects who would be classified as "Hierarch individualsits" & "egalitarian communitarians" based on their scores in relation to the mean on the two scales. I left out "hierarch communitarians" & "egaltiarian individualists."

Do you think they might be less subject to motivated reasoning? Do you think cultural "individaulism" predicts independence of mind?

December 30, 2013 | Registered CommenterDan Kahan

@Dan

Have you applied an OLS to each group independently yet?

I agree that the way you have framed the question by asking for testable hypotheses is less likely to lead to overfitting. However, since the regression only has inter-group power, — i.e. it does not explain intragroup variance — adding an additional variable to capture the outliers is likely to lead to false conclusions (both type 1 and 2 errors) about whatever we test.

Do you have much experience with classification trees? Let's re-frame the question in that manner. Then we can talk about our classification error rates (confusion matrix). I think it will also be more intuitive to formulate a hypothesis that deals more simply with Risk+ or Risk-, rather than the magnitudes.

Let me know if this is something you'd be interested in. I have quite a bit of experience with that.

December 30, 2013 | Unregistered CommenterZachary David

@Zacahary:

B/c the cultural outlook scales are continuous -- & treated as continuous predictors in the model -- the regression will pick up "intragroup" variance. I just split the sample into discrete groups for graphic illustration purposes. (I still haven't had chance full to absorb your post; likely you used the regression parameters to drive your simulation?)

Indeed, this property of the measures -- that they are continuous & can capture heterogeneity of attachments to affinity groups -- is one of their greatest strengths. If they were just simple binary classifications, there'd be a lot less insight.

Insight that would be squandered if regression analyses were performed separately on the sample after dividing it up into 4 groups. That would reduce statistical power, but also create the risk of spurious findings of "significant" effects -- b/c of the happenstance of random lumps or clumps in how an effect ends up "spread out" in relation to where I split the sample along the continuous worldview scores. Same thing about chopping up the outcome variables!

But I am confident that the basic point of your comment still holds.

That is, I think that instead of saying "what varaibles can I add that will reduce my variance?," more could be learned by coming up w/ defensible metrics for "nonconformity" & trying to come up with predictors for that. Likely OLS regression wouldn't be the right way to do that.

I think the "overfitting" danger would still be there, though. That's not a consequence of the modeling strategy; its a consequence of the ad hoc, exploratory style of engagement with the data that we are undertaking.

If I'm right, then why not try to be "disciplined" in the exploration -- do only what we think makes sense on the basis of a theory we otherwise have readon to credit -- & then validate w/ another data set?

Disagree?

But for sure I'm interested in learning more about the testing strategy you have in mind. (Sounds akin to how one would think of the fit of a logit model, which as you know predicts likelihood of an observation taking on a discrete value -- although the predictors can still be continuous.)

Also, I can see why it seems problematic to look for nonconformers only in 1/2 the sample -- as I was necessarily proposing in how I framed the problem as applied to risk perceptions. I could "redo" the analysis for risk perceptions after genuinely collapsing the 2 cultural outlooks to a single HI-EC dimension, in which case "hierarch individaulsits" and "egalitarian individualists" would be concentrated around the "mean" on both the x- & y-axes of the scatterplot. Or I could just "redo" using right-left political outlooks, which is pretty much what one would have if one squashed the 2-dimensional cultural framework into a one-dimension ideology one.

December 30, 2013 | Registered CommenterDan Kahan

@Dan

I meant have you applied a regression separately to each group to see if there is a relationship between the magnitudes of the cultural outlook and environmental risk. If your r^2 of an OLS performed on each group individually is close to 0, then it is appropriate to turn this into a binary classifier.

My simulations showed that the way you have defined the problem guarantees you to get a strong r^2 even if there is actually no relationship in the magnitudes. This should be a primary concern.

Just because the variable is continuous does not mean its magnitude actually contains information related to the question we're trying to answer. Performing a separate OLS on each group independently can allow us to justify dichotomizing the variables. And in this case, the design of the metric gives us sufficient reason to split at 0.

Here's a way to justify classification:

Test the hypothesis: "the strength of a person's affinity to a cultural outlook group is indicative of the strength of their belief in environmental risk perceptions." Here you would take the absolute value of all of your data. If you fail to reject the null then we can dichotomize.

December 30, 2013 | Unregistered CommenterZachary David

Also I think you have to normalize the data for that magnitude to work properly.

December 30, 2013 | Unregistered CommenterZachary David

@Zachary:

Isn't your question equivalent to asking whether the effect of one or the other of the cultural predispositions on the outcome variable is linear or uniform across the range of the scale or instead varies & gets stronger as one approaches one end or the other, or some other region where you think people are "more" prone to be "conformists" in the sense we are talking about?

If so, why not just tell me what sort of polynomial model you think should fit the data as is -- w/o splitting the sample? A quadratic ought to work if what you are positing is that the impact of motivated reasoning (and hence the correlatoin between cultural outlook & risk perception) is greater as one becomes "more [whatever--hierarachical, egalitarian, conservative]."

Any splitting is going to be ad hoc: the "groups" are just a way to visualize regions of the space defined by the orthogonal factors; there's nothing that says that "ECs start here & here, HIs there & there" etc.

Also, if we split the data, we won't know if any "difference" observed in the influence of cultural affinity in the two subsamples is an artifact of where we happened to do the splitting.

If the effect of a cultural predispositoin is uneven across the space it occupies, then it should be the case that a nonlinear model "fits the data better" than a linear one. But we should also have a good reason -- based on theory & other sorts of observations we have confidence in -- for positing that the relationship "looks like" the nonlinear one we are fitting to the data, to avoid overfitting.

does this rsponse -- whether or not you agree w/ it -- show that I'm engaging your point? Or am I missing it entirely?

Also-- I'm even more likely not be grasping fully the point about R^2 & classifying as opposed to scaling, but for sure a model that treats "cultural type" as a classification based on subjects scores on the 2 worldview scales in relation to the scale means ("HI," "EC," "HC," & "EI") explains a *lot* less variance than a model that treats the worldview scales as 2 continuous predictors (ones that can interact, too).

But am still digesting the post & so maybe that's why I'm not keeping up!

December 30, 2013 | Unregistered Commenterdmk38

@dmk38

My suggestion to do two regressions was only to verify that my suspicions are correct in that there is no association between the magnitudes. Another way to put it is: if we take the absolute value of cultural outlooks and environmental risk perceptions and run a regression, we should see an r^2 close to 0. Perhaps I should have framed it this way earlier. (there are some other considerations when doing a combined magnitude test on the whole sample but I don't think we'll run into those effects)

I'm not sure we were given any information which would justify using a non-linear model. We would need to know more about the individual metrics themselves. I assume that the population has some predictable distribution along these lines.

I'm arguing that if there is no relationship in the magnitudes, we should move to a classification model.

December 30, 2013 | Unregistered CommenterZachary David

@dmk

Also-- I'm even more likely not be grasping fully the point about R^2 & classifying as opposed to scaling, but for sure a model that treats "cultural type" as a classification based on subjects scores on the 2 worldview scales in relation to the scale means ("HI," "EC," "HC," & "EI") explains a *lot* less variance than a model that treats the worldview scales as 2 continuous predictors (ones that can interact, too).

We can't compare classification to OLS in this manner. They're in two different domains.

You're still thinking along the lines of explanation of variance. Which is what the r^2 will give us.

In a classification model your measure is not how much variance you explain. Our metric is based on classification error (false positives or false negatives).

Apples to oranges.

Take the absolute value of the cultural outlook and environmental risk perception vectors, then perform a regression. The weak r^2 implies that the vast majority of the information content is in the direction, not the magnitude.

You're actually making the problem harder for yourself by keeping in unnecessary noise.

December 30, 2013 | Unregistered CommenterZachary David

@Zacharay:

Okay. Read the post carefully & thought about it!

There's still going to be explanatory power in the continuous variables if I split my data in 1/2's or quarters. There won't be nearly *as much* -- b/c I'll have drastically contracted my sample size & also the variance in the data! But I'm pretty confident-- willing to bet, oh, $10,000-- that it's not the case that " there is no intragroup relationship in the data - i.e. the magnitude of a person’s Cultural Outlook is not associated with the magnitude of their Environmental Risk Perception."

For sure I can see a relationship between x & y even in the "green" & "black" observations by themselves.

But I'll run the regressions so we can both see!

December 31, 2013 | Unregistered Commenterdmk38

@Zachary:

Here you go! As I said, definitely the case that the cultural worldview scales explain variance "within" as well as "between" groups (something that follows logically from R^2 being higher when the worldview s are treated as 2 continuous predictors rather than 4 "categories," as I mentioned)

I don't disagree, though, that a modeling strategy involving classification might be better suited to the sort of inference we are trying to make. I think you'd just have to show me by doing it.

Even if one didn't throw away the information associated with treating the cultural outlook variables as continuous, one could model "nonconformers" defined as observations having a risk-perception value that differs by some specified amount from the predicted value derived from a regression.

Likely one would want to treat as "nonconformers" *more* subjects than appear in circles in my Figures. My guess is that individuals needn't be as "far off" from the mainstream as the observations in the circles before membgers of their community would regard them as tending to run against the grain.

Indeed, some of the observations in the circles might be genuine statistical outliers -- & it is unlikely that anything we do will "explain" them.

I shouldn't have used the term "outliers" in setting up the problem; it causes confusion -- even offense! -- in those who have a French Academy mentality about how statistics relate to reasoning)

"Nonconformers" is better -- both descriptively & analytically, since it focuses attention more on the issue of "how far" one's views have to stray before one will be viewed as having veered from orthodoxy within a particular group.

Would you like to have the data to play with?

December 31, 2013 | Registered CommenterDan Kahan

I would love to look at the original data. I think that would allow me to see things from your perspective much better.

You may have missed the first footnote in my post, but here it is: "It’s important to note that there is always a significantly non-zero probability that anything I say is completely wrong. Despite the existence of any strong language or absolute statements, it would be incorrect to think that I am absolutely confident in this or any argument. Beware of people who are."

I'm here to explore.

January 1, 2014 | Unregistered CommenterZachary David

lol on @Paul

Please help me out--I am unsuccessfully searching for your comment or discussion about why dividing up a population to get a better fit (less noise) is a suspect way to proceed. Or, if less work, maybe you could just tell me what the objection is.

I'm focused on explaining the noise--not sure it's noise, based on the clusters. Inclined to think there's a fallacy of composition underlying the disconnect.

January 2, 2014 | Unregistered CommenterTerry

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>