follow CCP

Recent blog entries
popular papers

Science Curiosity and Political Information Processing

What Is the "Science of Science Communication"?

Climate-Science Communication and the Measurement Problem

Ideology, Motivated Cognition, and Cognitive Reflection: An Experimental Study

'Ideology' or 'Situation Sense'? An Experimental Investigation of Motivated Reasoning and Professional Judgment

A Risky Science Communication Environment for Vaccines

Motivated Numeracy and Enlightened Self-Government

Making Climate Science Communication Evidence-based—All the Way Down 

Neutral Principles, Motivated Cognition, and Some Problems for Constitutional Law 

Cultural Cognition of Scientific Consensus
 

The Tragedy of the Risk-Perception Commons: Science Literacy and Climate Change

"They Saw a Protest": Cognitive Illiberalism and the Speech-Conduct Distinction 

Geoengineering and the Science Communication Environment: a Cross-Cultural Experiment

Fixing the Communications Failure

Why We Are Poles Apart on Climate Change

The Cognitively Illiberal State 

Who Fears the HPV Vaccine, Who Doesn't, and Why? An Experimental Study

Cultural Cognition of the Risks and Benefits of Nanotechnology

Whose Eyes Are You Going to Believe? An Empirical Examination of Scott v. Harris

Cultural Cognition and Public Policy

Culture, Cognition, and Consent: Who Perceives What, and Why, in "Acquaintance Rape" Cases

Culture and Identity-Protective Cognition: Explaining the White Male Effect

Fear of Democracy: A Cultural Evaluation of Sunstein on Risk

Cultural Cognition as a Conception of the Cultural Theory of Risk

« It's that time again: Science of Science Communication course in Spring Term | Main | Science curiosity research program »
Wednesday
Nov212018

An adventure in science communication: frequentist vs. Bayes hypothesis testing

A smart person asked me to explain to her the basic difference between frequentist and Bayesian statistical methods for hypothesis testing.  Grabbing the nearest envelope, I jotted these two diagrams on the back of it:

 

Frequentist: reject the null, p < 0.05; Bayesian: H1 (2.0) is 5x more consistent with the observed effect (1.5) than is H0 (0.0).

Displayed on the left, a frequentist analysis assesses the probability of observing an effect as big as or bigger than the experimental one relative to a hypothesized “null effect.” The “null hypothesis” is represented by a simple point estimate of 0, and the observed effect by the mean in a normal (or other appropriate) distribution.

In contrast, a Bayesian analysis (on right) tests the relative consistency of the observed effect with two or more hypotheses.  Those hypotheses, not the observed effect, are conceptualized as ranges of values arrayed in relation to their probability in distributions that account for measurement error and any other sort of uncertainty a researher might have.  The relative probability of the observed effect with each hypothesis can then be determined by examining where that outcome would fall on the hypotheses’ respective probability distributions.

I left out why I like the latter better. I was after as neutral & accessible an explanation as possible.

Did I succeed? Can you do better?

PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (97)

Dan,

How about:

https://doi.org/10.1007/s13164-018-0421-4

November 21, 2018 | Unregistered CommenterJonathan

Fun question!

The way I do it is to start with the Bayesian approach, and then explain what gets left out to arrive at the Frequentist approach.

First, you need an intuitive, graphical way to describe conditional probabilities. I use Venn diagrams where the area of each region is proportional to the probability. Then the conditional probability P(A|B) is the fraction of area B that is covered by A.

Then it's fairly easy to show that P(H|E) = P(E|H)P(H)/P(E). This is just a simple statement about areas. We interpret E as the evidence seen, and H as the hypothesis we want to prove or disprove. Then P(H|E) is the probability of the hypothesis given the evidence was observed, P(E|H) is the probability of observing the evidence given the hypothesis, and P(H) and P(E) are the prior probabilities of the hypothesis being true and the evidence being seen.

The thing on the left is what we want, but how do we fill in the things on the right? The first one we can have a go at - we need a 'statistical model' for each hypothesis that tells us how likely different observations are if our conjecture about how the world works is true. Some people start with a common statistical model that describes all the possible hypotheses, others consider the model to be uncertain and part of the hypothesis. Both Bayesians and Frequentists treat this bit pretty much the same (although it is not without its own controversies).

The P(E) bit has an uncontroversial meaning, but it's usually unknown and very hard to estimate. However, there is a neat trick we can do to get rid of it. We write the equations for two different hypotheses, and divide one equation by the other. The P(E) then cancels out.

P(H0|E)/P(H1|E) = [P(E|H0)/P(E|H1)] [P(H0)/P(H1)]

The alternative hypothesis H1 can be either something specific, or simply "H0 is not true".

This is where the first major problem arises, because for a vague statement like "H0 is not true" most statistical models are pretty useless at answering the question "What is the probability of the evidence being observed if H0 is not true?" The truth could be ***anything***; any insane rule, any strange weirdness is allowed. If your hypotheses about some parameter are x=0 and x>0, the latter includes possibilities like x=10^(10^(10^100)) and bigger! You've got infinitely many unknowns and no way to pick between them. How can you even calculate that?! The answer is, you can't.

Bayesians usually try to pick a specific H1 that they've got a statistical model for rather than saying "not H0" (and then having to give up). And then they leave open the question of whether there might be a better alternative hypothesis out there. Frequentists just drop all the terms involving H1 as unknowable, and trust that this is not one of those rare awkward situations where this will come back to bite them.

Finally, we come to the last term: the P(H) or P(H0)/P(H1). This is the problem of the priors, and is the biggest hole in the Bayesian approach because there is no objective method for assigning a value to it. Bayesians acknowledge the gap, and either submit intuitively reasonable guesses and assumptions, pick something that seeks to minimise the distorting effect, or leave the question unanswered and only tell you how much the evidence should move you without saying where you should start or where you will end up.

By contrast, the Frequentists again dismiss it as unknowable and ignore the problem.

Thus, the Frequentists are using something like P(H0|E) = P(E|H0) (or at least that it is hand-wavingly proportional to it), and saying that if the observed outcome is very unlikely under the null hypothesis, then the null hypothesis is very unlikely given the evidence. They cover some of the possible ways this can go wrong by separately examining other quantities like the 'power' and 'bias' of a test. The Bayesians are including a bunch of extra variables, but then having to limit the questions they can ask to exclude some, and then arbitrarily guessing at and hand-waving about the remaining values because they've got no objective and rational way to assign them.

The Bayesians are being a bit more explicit about where and how they're fudging the mathematics, but they're both facing the same basic problem. It's Hume's problem of induction. It can't be solved with mathematics, and fundamentally our methods only work because the universe happens to be unreasonably cooperative and regular. "The eternal mystery of the world is its comprehensibility."

November 21, 2018 | Unregistered CommenterNiV

NiV,

Did you read the link I dropped above - and if so, how have you modified your belief in things like: "Frequentists just drop all the terms involving H1 as unknowable, and trust that this is not one of those rare awkward situations where this will come back to bite them" and "...Frequentists again dismiss it as unknowable and ignore the problem"?

November 21, 2018 | Unregistered CommenterJonathan

"Did you read the link I dropped above - and if so, how have you modified your belief ... ?"

I skimmed through it, but didn't parse it in detail.

I didn't notice anything to suggest I was wrong. Should I have?

November 21, 2018 | Unregistered CommenterNiV

NiV,

I think someone reading your explanation first, then reading that Colling & Szucs paper would lower their belief in your explanation.

Having read Colling & Szucs paper first, then reading your explanation, I don't appreciably modify my belief in direction of your explanation. This is because your explanation seems to be of the form: viewing frequentism as a modified form of Bayesian inference, one notices that frequentists make certain unsafe systematic assumptions. But, frequentism is not a modified form of Bayesian inference (as is spelled out in Colling & Szucs, and certainly elsewhere). Hence, your conditional is undercut by having a dependent clause that you don't support but that Colling & Szucs provide evidence against.

So, the question becomes, why are you explaining frequentism as if it is a modified form of Bayesianism with additional (and not always safe) assumptions?

November 21, 2018 | Unregistered CommenterJonathan

"So, the question becomes, why are you explaining frequentism as if it is a modified form of Bayesianism with additional (and not always safe) assumptions?"

Because to understand the differences between them you first have to put them into a common frame of reference, and it's easier to understand the Frequentist approach starting from the Bayesian framework.

It's like trying to explain the difference between Newtonian physics and Special Relativity. You start by explaining Special Relativity, and then say Newtonian physics is the approximate limit of behaviour at speeds much lower than the speed of light, or equivalently, the limit you'd get if you made the speed of light infinite.

Newtonian physics is not - historically or conceptually - a modified version of Special Relativity. But if you want to understand the differences between the two, then describing Newtonian physics as the low-speed limit of relativity is far easier than starting with Newton and then trying to introduce all the conceptual leaps and counter-intuitive changes that Einstein added.

The problem is that the underlying philosophy behind Frequentist null hypothesis testing is normally not discussed. It's one reason why people misunderstand it so often. Why do they consider p-values, the probability of an event given the null hypothesis, when what you actually want to know is the probability of the null hypothesis being true? They don't say. They just tell you the method, as if it gives you the answer to the exam quesrion. And then afterwards they kind of note in passing, like as a minor footnote of no special importance, really, don't worry about it, that p-values don't in fact tell you the probability of the hypothesis being true, and it's all a bit more complicated. And then they stop, and don't tell you what comes next.

However, if you look first at the Bayesian formula, and then note that all the bits they leave out are precisely the bits nobody knows how to calculate, it all suddenly makes sense! The reason they use p-values as a proxy for the truth of the hypothesis is that they can't generally formulate the alternative hypothesis or the priors in a way that makes mathematical sense.

There's more to it than that, of course. But if you want a 10-minute summary of the *essential* core difference for a smart person, it's a good place to start.

--
And of course I was hoping to start a discussion. I'm glad that bit worked!

November 21, 2018 | Unregistered CommenterNiV

Most scientists would like to know the probability (P(H)) for their hypothesis H, given their data, assuming nothing else. Unfortunately, this isn't possible.

Frequentists keep the "assuming nothing else", discard any hope for P(H) and opt for something else instead that is still useful. Bayesians try to keep P(H) and discard any hope for "assuming nothing else".

Frequentism gives you (if done correctly) something that will "objectively track truth". Bayesianism doesn't guarantee that, but instead gives answers that are (apparently) more intuitive and easier to manipulate and interpret.

However, I wonder if scientists had used Bayesianism extensively for nearly 100 years (as they have frequentism), if they'd be upset with Bayesianism's ability to lead them astray. Being apparently easy to understand isn't always a good thing. Consider the discussion in Colling and Szucs about the stopping rule - it's nuanced in Bayesianism even though it is often thought to not be. And there's always the problem of the priors.

November 21, 2018 | Unregistered CommenterJonathan

"Frequentists keep the "assuming nothing else", discard any hope for P(H) and opt for something else instead that is still useful. Bayesians try to keep P(H) and discard any hope for "assuming nothing else"."

Yes, I'd say that was a fair summary. :-)

"However, I wonder if scientists had used Bayesianism extensively for nearly 100 years (as they have frequentism), if they'd be upset with Bayesianism's ability to lead them astray."

Oh, scientists using Bayesian methods extensively complain about its shortcomings, too! I've been involved in debates on the relative merits of uniform priors versus non-informative priors - which even scientists used to the Bayesian approach get confused about. You might also be interested in Lindley's paradox, if you've not come across it before.

In practice, all statisticians I've ever come across use *both*, as they deem appropriate, and always have. Fisher (probably the leading Frequentist) praised Bayes and regarded his updating method as uncontroversially the right approach for estimation. It was primarily the interpretation of the result as the true, inferred probability of an unobserved parameter (Fisher called it 'inverse probability', but it's known as the posterior distribution now) rather than an estimate subject to error, particularly in the case of uniform priors, that he regarded as a mistake. The debate is long and subtle, and I don't think it's possible to do justice to it briefly. This can only be a starting point.

November 22, 2018 | Unregistered CommenterNiV

@NiV-- Put me in the "leave the question unanswered and only tell you how much the evidence should move you without saying where you should start or where you will end up" camp (see pp. 390-91).

November 22, 2018 | Registered CommenterDan Kahan

A problem with Bayesianism that I think is underappreciated is that a Bayesian result usually assumes that the new evidence is completely independent of anything that one used to generate one's prior.

November 22, 2018 | Unregistered CommenterJonathan

Indeed. Very true.

Another is that the statistical model by which one calculates the probability of each outcome under each hypothesis is also potentially open to question; a part of the hypothesis, and subject to updating as new evidence comes in. If your current model says that the evidence seen is highly unlikely under *any* of the available hypotheses, then maybe it's the model that's wrong. (It's where the models come from in the first place.)

November 22, 2018 | Unregistered CommenterNiV

If you're science curious, would you rather see frequentist or Bayesian results?

Understanding science curious as Dan has spelled it out, I'd guess that SC folks would rather see frequentist results. I'm basing this on the feeling that frequentism gives one of encouraging surprise of a certain type - a "maybe I'm completely wrong about this" kind of surprise that is offered by frequentist statistics objectively tracking truth independently of one's priors. Bayesian statistics never say you were completely wrong. They imply no evaluation on what you should have believed prior to seeing the experiment. To SC folks (high enough in OSI to understand this much about Bayesianism and frequentism), this aspect of Bayesianism could be a buzz-kill. Which might just make them less SC.

November 23, 2018 | Unregistered CommenterJonathan

Here's a test for the frequentist v. Bayesian methodologies:
-----------------------------------------------------------------------------------
"Recent Census Bureau projections indicate that racial/ethnic minorities will comprise over 50% of the U.S. population by 2042, effectively creating a so-called “majority–minority” nation. Across four experiments, we explore how presenting information about these changing racial demographics influences White Americans’ racial attitudes.
[..]... Whites exposed to the racial demographic shift information preferred interactions/settings with their own ethnic group over minority ethnic groups; expressed more negative attitudes toward Latinos, Blacks, and Asian Americans; and expressed more automatic pro-White/anti-minority bias..."
http://groups.psych.northwestern.edu/spcl/documents/Craig%20&%20Richeson%202014%20PSPB.pdf
----------------------------------------------------------------------------------------------------------------------------------------------------

That's why the entire American right opposed Johnson's Civil Rights legislation ending European preference in immigration. More diversity inevitably leads to civil strife.

But the really interesting part is that, spooked at the effect of its projections (really common sense requiring no demographic forecast) The Census Bureau has now stopped posting projections for the year minorities will cross 50%. That was the "prior" in 1965, and it has now translated into observable frequencies - that's where all these "divergences" and "polarization" come from.

Trying to use "science curiosity" as explanatory variable is like counting angels dancing on heads of pins - nobody knows, few care. Survival instinct is however an overriding consideration.

November 23, 2018 | Unregistered CommenterEcoute Sauvage

http://groups.psych.northwestern.edu/spcl/documents/Craig%20&%20Richeson%202014%20PSPB.pdf

Better link - sorry formatting isn't always automatic.

November 23, 2018 | Unregistered CommenterEcoute Sauvage

PS if all else fails, kindly follow link to article given in NYT
https://www.nytimes.com/2018/11/22/us/white-americans-minority-population.html

November 23, 2018 | Unregistered CommenterEcoute Sauvage

PPS Johnson's immigration legislation has received less attention than the other twin catastrophes of his presidency, hyperinflation leading to monetary collapse and the Vietnam war, but in the long run it's proved more destructive than either. And keep in mind said legislation concerned legal immigrants, not migrant invaders - on the latter, the Right firmly backs Col. Lind (Rtd, USMC): https://www.traditionalright.com/the-view-from-olympus-invasion/

November 23, 2018 | Unregistered CommenterEcoute Sauvage

Ecoute -

That's why the entire American right opposed Johnson's Civil Rights legislation ending European preference in immigration. More diversity inevitably leads to civil strife.

Really? How are you defining civil strife? How are you measuring diversity?

Is the US more or less diverse than 25, 50, 100, or 200 years ago? Is there more or less civil strife?

Which countries have the most civil strife - those with the most diversity or the least diversity? Seems to me that there is no consistent causal relationship between diversity and civil strife. Seems to me that the degree of civil strife in a society is a function of many factors. Why do you single out that one factor? Are you cherry-picking? Is there any connection between your singling out that one factor as causal for strife, and your determination that antisemitism had nothing to do with motivating a man who made comments about killing Jews to murder Jews as they worshiped in a synagogue?

What evidence do you use to make your claim?

November 23, 2018 | Unregistered CommenterJoshua

Dan,

I read the PRMP papers, so won't read again as they are already incorporated in my priors (and wouldn't want to increase their impact merely by rereading!).

That Laws of Cognition paper is new to me. How should I decide how much of what's in this paper is based on things I've already incorporated into my priors (having read 37 other papers by you)? I'll read it anyway, on off chance it contains new things...

Also, there's evidence that humans aren't, and probably can't be, Bayesian:

https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1006572

And, remember, ought implies can.

November 24, 2018 | Unregistered CommenterJonathan

Another issue I have with Bayesianism is how hard it is for people to conceptualize even the approximate odds of a very improbable event. I think of this as a bias due to our use of the common probability scale of [0..1], and how difficult it is for people to conceive of the difference between 0.00001 and 10^-50 (such an issue might be less of a problem with an information metric like surprisal). In Bayesian terms, this difference is huge - and counters the common belief that biased priors eventually wash out (not in the lifetime of the universe!). It's likely that, if this isn't appreciated, wide use of Bayesianism would lead to an increasing embrace of pseudoscientific theories.

November 24, 2018 | Unregistered CommenterJonathan

Here's a good read: Bayesian Reasoning for Intelligent People

http://tuvalu.santafe.edu/~simon/br.pdf

November 24, 2018 | Unregistered CommenterJonathan

@Jonatah-- the Laws of Cognition paper discusses mechanisms other than PMR that complicate formation of truth-convergent LRs

November 25, 2018 | Registered CommenterDan Kahan

Jonathan - Dan's article is orders of magnitude better than the lemon you linked (with due deference to Mr DeDeo). Dan shows a touching concern for ultimately getting to The Truth, a concern wholly absent from the other paper. Though I wish he hadn't preceded it by the term "converging on", which reminds me of a horrible time spent trying to understand path integrals, until one day I finally gathered up all my courage and asked "why do we try to corral all these infinities into convergence when we already know the particle will travel in a straight line?" and was promptly answered with "we don't understand these infinities, but we can make a stab at calculating them". That permanently ended my interest in string theory, but it was the honest answer.

If DeDeo wanted to re-name his paper "Bayesian Reasoning for Honest People" he would accept betting on his belief that Hillary Clinton would win in 2016 with 99% probability. If you claim to understand your Bayesian calculation you bet on it. For the record, DeDeo quotes approvingly Wang, but fails to mention Wang is an honest man who did bet on his own 99% prior by saying if Clinton didn't win, he, Wang, would eat an insect on live television. And he did.

November 25, 2018 | Unregistered CommenterEcoute Sauvage

Dan - unless I miss something in your article, I don't see how a search for truth CAN be reconciled with PMR. Don't you inevitably end up with "Cleopatra was black"-type inanities?

Here is NYU savant Kwame Appiah:

"...Dick Hernstein, (who was a psychologist at Harvard) ..was interested in the connection between race and intelligence, and he thought that once you thought that there was a connection between race and intelligence, you were invested in a view which has unfortunate social consequences, independently of whether it was true, and so there he was talking about the ways in which you could be motivated, in that case intellectually motivated, by an attachment to a thesis, to do things that had bad effects."
http://the-orb.org/2017/09/20/interview-with-kwame-anthony-appiah/

I love "independently of whether it was true"!

November 25, 2018 | Unregistered CommenterEcoute Sauvage

Here's a pro-Bayesian, pro-climate change consensus (trigger warning: 97% figure), anti-cultural cognition (and other untamed Bayesianism) paper that says that science communicators (notably Dan, cited) just don't get Bayesianism properly:

https://onlinelibrary.wiley.com/doi/full/10.1111/tops.12173

November 25, 2018 | Unregistered CommenterJonathan

Jonathan -

A simple statement about consensus hides multiple complex and overlapping evidential signifiers, including but not limited to the reliability of the communicator, the sources providing the consensus information, and their degree of independence.

Yes, that. Studying the effect of providing information on the "consensus" in a "lab-based" context is of very limited value, IMO.

November 25, 2018 | Unregistered CommenterJoshua

@Jonathan-- it's a good paper. I don't feel my understanding of Bayesianism, or my use of it to help delineate the presence & consequences of motivated reasoning, is challenged by it. Indeed, in other works, Hahn has recognized that endogeneity of prior & likelihood ratio prevents Bayesian updating from being truth-convergent.

November 26, 2018 | Registered CommenterDan Kahan

Jonathan -

I'm listening to The Ezra Klein Show with @TuneIn. #NowPlaying http://tun.in/tiQXTH

Haidt.

I'm finding him increasingly annoying. My most annoyed moment came at about 58 minutes in, where - without any apparent sense of irony - he blames triggering-happy, self-victimizing leftists (who supposedly always say "Punching down always good, punching up always bad") for triggering the rightwing to be hateful towards what the left has to say.

Reminds me of when people argue that "skeptics" wouldn't be so "skeptical" if those name-calling, capitalism-hating, pro-childstarving, pocket-lining, data manipulating, dogma-mongering "alarmists" would just stop being mean and calling them "deniers" and stop talking about a consensus (because they can't comprehend their appeal to authority and ad populum fallacy, doncha know).

November 26, 2018 | Unregistered CommenterJoshua

link drop:

https://phys.org/news/2018-11-science-knowledge-shifted-religious-political.html

Sociology researchers at the University of Nebraska–Lincoln published research in the Sociological Quarterly that examined how science knowledge and curiosity are associated with both religious beliefs and political ideologies. They found neither religiosity nor political affiliation was related to overall scientific curiosity, but differences were found when looking at scientific knowledge.

Unfortunately, can't find non-paywall version. So, in it's place, will continue my usual rant:

https://www.newscientist.com/article/mg24032052-900-time-to-break-academic-publishings-stranglehold-on-research/

November 26, 2018 | Unregistered CommenterJonathan

Joshua,

Am ambivalent about Haidt. I agree that he does seem to go overboard with his social commentary, but his science seems to be quite good. Moral foundations was one of the studies that got an OK from Many Labs 2. Also, I agree with him that there are problems with having a left dominated academia in the social sciences (regardless of the cause): https://psyarxiv.com/e4xw3/.

Been wondering ever since Haidt cleverly named the pro-right foundations "binding" and the pro-left foundations "individuating" whether he is mediating his own behavior and fame as a form of social corrective. Who knows, maybe he was partially responsible for that "traditional liberal" moniker that gets my goat.

November 26, 2018 | Unregistered CommenterJonathan

Dan,

About that second Hahn et al "How good is your evidence..." paper. This brings up another of my pet peeves with Bayesianism, related to one above.

Bayesians know about surprisal, but still seem to prefer to eye-ball prob-scale [0..1] effects, and then suffer bias from that examination. Hahn shows off some very nice charts which have a prob-scale [0..1] dimension, and then says stuff somewhat like: See, they get closer - isn't that so nice!. But the surprisal scale (-log prob), which has the advantage of having actual physical support (its entropy!, its actual information!) and other mathematical niceness, Bayesian inference doesn't push things closer together at all. This is because Bayes theorem in the log world is perfectly linear:

-log(P(H|E)) = log(P(E)) - log(P(E|H)) - log(P(H))

If we're some surprisal (information/entropy) distance apart, and we both modify our beliefs subject to evidence we both deem (hopefully accurately) independent of our priors like perfect Bayesians, we end up exactly as far apart in surprisal terms as before. Suddenly, inference from pretty pictures doesn't provide a warm fuzzy.

This is nearly the same as that issue I raised above with 0.00001 vs. 10^-50. But, turned around a bit. If I'm 0.99 sure of X and you're 0.9999 sure of X, we probably get along much better than if I'm 0.1 sure of X and you're 0.9 sure of X (assuming X is important enough to quarrel about). But, if we were true Bayesians all the way, we shouldn't! We should have about 10 times more animosity towards each other at 0.99 vs. 0.9999 than at 0.1 vs. 0.9. Hahn (and others) appear to me to be arguing that normatively we should be Bayesians, and then drop Bayesianism in favor of our prob-scale bias to determine our social/practical belief distance. Although, to be fair, some Bayesians argue instead that we should be perfect Vulcans and communicate entirely by infinite precision mathematical quantities.

My favorite interpretation is that since we're probably much friendlier at 0.99 vs. 0.9999 than at 0.1 vs. 0.9, then that means we're not very good at the conversion between beliefs and numeric values. Maybe this is related to your Likert-scales-found-wanting finding...

November 26, 2018 | Unregistered CommenterJonathan

Jonathan - I don't understand your assumption on difficulty in comparing low-probability events.

See here:
https://www.theguardian.com/society/2018/nov/26/rare-cancer-linked-breast-implant-used-by-millions-women-lymphoma

If I were considering having such implants and were told the cancer risk is (per UK estimate) 1 in 30,000 I would probably ignore it. But if, as per Australian estimate, the risk is closer to 1 in 1,000, I probably wouldn't go ahead. Nor do I think such a calculation is uncommon.

Related, another reason I didn't think much of the DeDeo paper: re-formulating the problem as a bet, which he refuses to do, clarifies matters enormously. How much would you pay to insure your home against space meteorites (assuming such insurance is available). That risk is obviously smaller than 1 in 30,000 as in the implants. Would you pay a quarter? All insurance pricing is based on some Bayesian calculation. Frequentist odds aren't as important.

November 27, 2018 | Unregistered CommenterEcoute Sauvage

Ecoute,

This Searle quote from the DeDeo paper comes close to one issue I have:

it seems to be a strict consequence of the axioms [of decision theory, the translating of
probabilities into actions] that if I value my life and I value twenty-five cents ... there must be some odds at which I would bet my life against a quarter. I thought about it, and I concluded there are no odds at which I would bet my life against a quarter, and if there were, I would not bet my child’s life against a quarter. ... I argued about this with several famous decision theorists, starting with Jimmy Savage in Ann Arbor and including Isaac Levi in New York, and usually, after about half an hour of discussion, they came to the conclusion, “You’re just plain irrational.” Well, I am not so sure.

I feel similarly to Searle here - I can't determine what odds I'd need in a life against a quarter bet, and it seems that no odds would satisfy me. I can try to explain this, but I'm not sure the explanation matches with the actual reason. Certainly, part of it has to do wit DeDeo's point about that being the last bet I'd lose (although, think of the marketing possibility - you'll never lose another bet again!). Another explanation might be that my uncertainty about extreme odds/probabilities/values clouds my ability to judge the fairness of such bets to be within my comfort zone. I would probably instead pay to stay out of situations where such a bet is forced. But, how much?

(BTW - this doesn't mean I think Searle is rational - he's been accused of things that, if true, would suggest otherwise. Or, perhaps he is super-rational in a Machiavellian sense.)

Also, as you probably suspect, "All insurance pricing is based on some Bayesian calculation" does not increase my ease with Bayesianism, but instead decreases my (already low) trust in insurance. But, as Hahn says, that might be a purely Bayesian calculation on my part! Dan's probably right here instead - if it were socialized (even partially) insurance of some type instead of entirely commercial (hence left instead of right leaning), I'd have much higher trust, provided I can manage to keep Republicans away from screwing it up. As of earlier this month, my trust in that last bit has increased.


Oh - this is an idea for a test to differentiate pure Bayesianism from cultural cognition: take some partisan issue such that subjects diverge further when exposed to scientific claims (such as climate change or nuclear power), and see if they claim as well after diverging that they already were completely familiar with that claim. If purely Bayesian, they must either not budge or say that the claim is somehow new to them. If instead Dan is correct that they are corrupted Bayesian, they will both diverge and say the claim is completely familiar (hence should already be accounted for in priors). Although there are alternative explanations to Dan's for that, at least we can rule out pure Bayesianism.

November 27, 2018 | Unregistered CommenterJonathan

@Jonathan-- I think we make such bets 100's or 1000's of times per day. E.g., did you drive to work today? I haven't read the paper-- maybe it makes/anticipates this point.

November 27, 2018 | Registered CommenterDan Kahan

Dan,

"I think we make such bets 100's or 1000's of times per day. E.g., did you drive to work today?"

I suspect that people don't frame activities such as driving to work as a bet, although it is. It's probably the lack of such framing that allows them to make the drive. And then they text while driving, again without framing that as a bet. I suspect that in these cases, people consider that the appearance of control that they have in the situation is such that they think the risk of an accident is ~0 - or, rather that they don't even think of the risk at all because of the appearance of control. That people take such risks does not imply that they calculate such risks, and certainly not that they do Bayesian calculations. It also doesn't imply that they would take those same risks if those behaviors are framed as bets in ways that nullified the appearance of control.

November 27, 2018 | Unregistered CommenterJonathan

Does "socialized" insurance mean subsidized insurance? If yes, it's not insurance at all, and neither Bayesian nor frequentist probabilities are relevant.

On climate change, today Krugman (who oddly used to be a good economist) comes up with a classic - discarding probabilities altogether and instead formulating the problem in Manichaean terms:
"..While Donald Trump is a prime example of the depravity of climate denial, this is an issue on which his whole party went over to the dark side years ago. Republicans don’t just have bad ideas; at this point, they are, necessarily, bad people."
https://www.nytimes.com/2018/11/26/opinion/climate-change-denial-republican.html

I hope I'm not in breach of cultural cognition rules if I call this ()&*^%$.

November 27, 2018 | Unregistered CommenterEcoute Sauvage

"...discarding probabilities altogether and instead formulating the problem in Manichaean terms"

Coincidentally, am reading about just that now:

The Binary Bias: A Systematic Distortion in the Integration of Information

"...at this point, they are, necessarily, bad people" Yes, certainly sounds both Manichaean and prejudicial. Hmmm... wherever have I seen such Manichaean prejudice about outgroup members before?

November 27, 2018 | Unregistered CommenterJonathan

Jonathan - from your Binary Bias article:
" Highly numerate
individuals have less-distorted probability functions
(Patalano, Saltiel, Machlin, & Barth, 2015) and may be
less susceptible to the binary bias."

But just to take the latest post here: there is absolutely no doubt Krugman is highly numerate - he got his doctorate at Tech before admissions / graduations were polluted by political correctness. Why, then, is he sounding like some mad prophet of the "Repent, sinners, the end of the world is nigh" variety? Some overwhelming Bayesian prior is obviously clouding his judgement.

And if you have managed to remember the answer to your question "...Hmmm... wherever have I seen such Manichaean prejudice about outgroup members before?" I'd like to see it.

November 28, 2018 | Unregistered CommenterEcoute Sauvage

There's a cottage industry of attempts at fitting Bayesianism to empirical human probability judgments. Here's a new one:

https://psyarxiv.com/af9vy/

I'm finding these fascinating, although sounding a bit like epicycles:

We introduced a two-stage model of Bayesian inference, in which people are assumed to first draw samples from their posterior distribution and then make the best estimate possible given those samples. An exact Bayesian model does not have this uncertainty about probabilities – while there is uncertainty about the true state of the world, there is never any uncertainty about the probability of any state or collection of states. Our approach thus better matches the phenomenology of making probability estimates, as any hemming and hawing what estimate to make clearly points to some uncertainty about the probabilities. To explain the classic demonstrations of incoherence in probability judgments, we then simply assume that people are aware (at some level) of this uncertainty, and adjust for it appropriately.

This paper addresses specifically the conservatism bias. Which hadn't occurred to me when thinking about how bad people are at near 0 or near 1 prob estimation. Perhaps we know we're bad, and stay away.

Or, for fans of just-so-evo stories, perhaps we reserve near 0 and near 1 probs categorically for specific semantic (non-numerical) purposes because these probs haven't been encountered in natural environments (organisms won't likely encounter enough data to classify something that close to 0 or 1 based on rational empiricism alone) often enough for our innate prob estimation to have needed to evolve to accommodate sufficiently exactly. Consider the Gran Sasso neutrino event of 2011 discussed in that DeDeo paper: "Most physicists have very strong beliefs in special relativity; they form these on the basis of both experiments, and also on the general coherence of the theory itself—what William Whewell first called “consilience”". That second coherence-based adjustment of belief is what I'm referring to, and it's that which prevented scientists from falling into the pseudoscientific trap that some journalists of the event fell into - because those journalists' appreciation of the coherence of relativity wasn't strong enough to move fast-than-light travel into that close-to-0 category that one can't escape with evidence alone. (Also the scoop premium in journalism and man-bites-dog stories thing, certainly. Any good just-so-evo story needs friends.)

November 28, 2018 | Unregistered CommenterJonathan

"Here's a pro-Bayesian, pro-climate change consensus (trigger warning: 97% figure), anti-cultural cognition (and other untamed Bayesianism) paper that says that science communicators (notably Dan, cited) just don't get Bayesianism properly"

Excellent paper! Thanks for the trigger warning, by the way, but actually I approve of their use of the example because they do the analysis of how people modeled as Bayesians ought to react to it properly.

"If we're some surprisal (information/entropy) distance apart, and we both modify our beliefs subject to evidence we both deem (hopefully accurately) independent of our priors like perfect Bayesians, we end up exactly as far apart in surprisal terms as before."

Per the usual Bayesian assumptions, yes. (And I agree with your point.)

But I'd note the caveat that this assumes all parties also use the same statistical model to calculate the likelihood function. People using different models will calculate different likelihood ratios and jump by different amounts. And the model itself may be subject to being updated by the evidence. In the real world they're not a given. We have to deduce those from observation, too.

And if you change your model of the world as a result of new evidence, that would require you to re-evaluate all the previous evidence you've seen, and it's effects on your priors!

Since your model of the world has to start off with unsupported priors too, there's no more guarantee that people will converge on models than that they converge on subject matter beliefs derived using them.

Textbooks start with the simplified 'lies to children' version, and then slowly fill in the gaps. Since everyone drops out of studying the subject at some point, virtually everyone is left with some level of the 'lies to children' story. Our understanding of Bayesian inference is no exception.

November 29, 2018 | Unregistered CommenterNiV

NiV,

Yes - excellent point (again) about models and their dependencies - which you've made before (and I agreed with before). However, I still don't think Dan is convinced of this - my impression is that he (still) thinks that belief divergence necessarily implies corrupted Bayesianism. I think it implies neither Bayesianism nor corruption, but certainly an uncorrupted model-accommodating Bayesian network could work this way as well.

I think that the independence test I proposed would give a clue here. If evidence is presented that might not be independent of subjects' priors - such as scientific consensus arguments - and belief movement (in either direction) based on this evidence is positively correlated with claims that the evidence is familiar to the subject, then I think this would be a problem for Bayesianism, even the corrupted varieties that Dan has proposed, since even these should filter out familiar evidence already incorporated into priors. Seems it would instead imply a more affective explanation - one isn't updating priors, calculating likelihoods, or any other calculation so much as reacting to a stimulus one perceives as a threat. One could go further and test if, when repeated, this reaction follows the typical gradual habituation curves that repeated exposure treatment has on other fears. If it doesn't stop immediately (suppose the subject was lying about familiarity initially - they would be the second time they hear the same argument), then I think Bayesianism is out as an explanation, even the model-accommodating network variety.

November 29, 2018 | Unregistered CommenterJonathan

Jonathan - I liked the Hahn et al article very much also (and I also think use of the 97% is justified in this case) but I don't see how your interpretation of worldview revisions applies. See poll data:

http://climatecommunication.yale.edu/visualizations-data/ycom-us-2018/?est=happening&type=value&geo=county

Essentially, this says people may or may not believe the climate warnings, but they're definitely sick and tired of hearing them. Two-thirds of all people NEVER want to discuss the topic again, and an overwhelming majority (77%) think that media should mention it at most once a month. I hear the incoming House majority plans to make a major issue of it - which naturally cheered me up no end.

November 29, 2018 | Unregistered CommenterEcoute Sauvage

An apparently unrelated datum on cultural cognition, today's victory of Carlsen over Caruana. I know many chess experts; all of them supported Norway's Carlsen, even though Caruana was playing for the US. That level of play is beyond me, but I also rooted for Carlsen - somehow Caruana struck me as an alien interloper. Here's a brief video:
https://www.youtube.com/watch?v=N3YHvvivozM

November 29, 2018 | Unregistered CommenterEcoute Sauvage

Ecoute,

"Two-thirds of all people NEVER want to discuss the topic again, and an overwhelming majority (77%) think that media should mention it at most once a month."

Do you really parse that data that way, or you just baiting? To me, it looks like 2/3 never discuss it, and 77% hear about it at most once a month in the media. I don't see anything normative in those questions. Too bad this map doesn't show reps vs. their constituents.

As for the incoming House majority - they apparently have Trump derangement syndrome to thank:

http://www.pewresearch.org/fact-tank/2018/11/29/in-midterm-voting-decisions-policies-took-a-back-seat-to-partisanship/

Perhaps they should send him a nice Christmas card and an ugly sweater (sign of affection).

November 29, 2018 | Unregistered CommenterJonathan

Dazzling. https://www.youtube.com/watch?v=7JwlOjMI420

November 29, 2018 | Unregistered CommenterEcoute Sauvage

Jonathan - as you can tell I was reading fast while watching the final game, so if I mis-read I apologize. Will re-read later. But I cannot be suspected of "baiting"!!!

November 29, 2018 | Unregistered CommenterEcoute Sauvage

Jonathan, thanks!

"If evidence is presented that might not be independent of subjects' priors - such as scientific consensus arguments - and belief movement (in either direction) based on this evidence is positively correlated with claims that the evidence is familiar to the subject, then I think this would be a problem for Bayesianism, even the corrupted varieties that Dan has proposed, since even these should filter out familiar evidence already incorporated into priors."

Possibly. You would have to be very careful about how you tested that, though.

For example, it could be argued (assuming one accepted the 'ad populam' heuristic) that for any claim there will be some unknown proportion of affirming and disagreeing arguments floating around the debate-o-sphere, and every time you come across one, it's a data point regarding the frequency of each. If you hear ten arguments and they all make the same claim, with none against, that strongly suggests that the majority of the arguments circulating support that claim. If you come across six for and four against, you would more likely conclude the issue is contested. So even hearing evidence that you've heard several times before in every detail could arguably be taken as additional confirmation. The evidence itself is repeated, but the multiple events in which it is transmitted to you are supposed independent.

I personally don't doubt that human reasoning is at best a heavy approximation to Bayesian updating consisting of a bunch of "good enough" heuristics - both evolved and learnt through experience. The choice is based on a trade-off between computational effort and accuracy, implemented on a non-numeric neural-net-type computer made of meat. It would be silly to think it was going to be perfectly Bayesian - it just has to be close enough for users to usually survive and have offspring. However, I do also think it's usually a hell of a lot better approximation than the simplistic textbook models many theorists propose! Humans very often do model multiple mechanisms of uncertainty and unreliability in information sources in considerable detail, including the potential for non-independence.

Not always, of course. Some people assume that if they're told 97% of scientists agree on something, that's extremely unlikely to happen coincidentally. Others consider the possibility, for example, that most scientists haven't looked at the data themselves, but get their beliefs from what other scientists say, same as the rest of us. (And obviously if they happen to know the actual figure direct from the source survey was 82%, maybe even less, yet all these 'scientists' keep citing the 97% figure like they believed that was what the evidence said, they do more than just "consider" it!) The amount of detail people put into their mental models varies considerably - and related to their motivation and preconceptions, of course - so the first thing you need to do in any experiment to understand how people think is to ask them detailed questions about what precise definitions, models, and mechanisms they're assuming. If you try to do some statistical experiment to tease out details of their mental mechanisms assuming everyone defines terms like "expert" or "global warming" or "recent" or "significant" or "average" or even "temperature" the same way, you're quite likely to get muddled, meaningless results.

November 29, 2018 | Unregistered CommenterNiV

Some people assume that if they're told 97% of scientists agree on something, that's extremely unlikely to happen coincidentally. Others consider the possibility, for example, that most scientists haven't looked at the data themselves, but get their beliefs from what other scientists say, same as the rest of us. (And obviously if they happen to know the actual figure direct from the source survey was 82%, maybe even less, yet all these 'scientists' keep citing the 97% figure like they believed that was what the evidence said, they do more than just "consider"

Perhaps for "some people."

But my guess is that most people accept a scientific consensus as a useful but not dispositive decision-making heuristic in most cases, except when they are "motivated" otherwise - in which case they get into endelss arguments to quantify the magnitude of the consensus (the outcomes of which are, of course, easily predictable in association with ideology), or about the value of consensus, etc..

November 30, 2018 | Unregistered CommenterJoshua

"But my guess is that most people accept a scientific consensus as a useful but not dispositive decision-making heuristic in most cases, except when they are "motivated" otherwise..."

Yes, Joshua. That's what I said. :-)

Some people are "motivated" to take it as dispositive.

November 30, 2018 | Unregistered CommenterNiV

I wish there was consensus on the consensus number. For instance, there's:

https://www.skepticalscience.com/global-warming-scientific-consensus-intermediate.htm

which cites several different studies, including ones like Doran 2009 and Anderegg 2010, which get to the famous 97% figure with climate scientists most active, not papers. At least, when a popsci article mentions the 97% number, it should cite the source (so, kudos to skepticalscience here). But, that's a pet peeve of mine (have lots of them, making up for all those cats I don't have) - popsci should always cite (and link) its (hopefully non-paywalled) sources so the curious public can take a look for themselves. The public might learn something, and thereby find meaning in learning, at hopefully higher rates than the meager "learning" figures in:

http://www.pewresearch.org/fact-tank/2018/11/30/members-of-both-parties-find-meaning-in-family-but-differ-when-it-comes-to-faith/

Guess we should be thankful learning broke the top 9.

November 30, 2018 | Unregistered CommenterJonathan

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>