## An adventure in science communication: frequentist vs. Bayes hypothesis testing

A smart person asked me to explain to her the basic difference between frequentist and Bayesian statistical methods for hypothesis testing. Grabbing the nearest envelope, I jotted these two diagrams on the back of it:

Frequentist: reject the null, p < 0.05; Bayesian: H1 (2.0) is 5x more consistent with the observed effect (1.5) than is H0 (0.0).

Displayed on the left, a frequentist analysis assesses *the probability of observing an effect as big as or bigger than the experimental one relative to a hypothesized “null effect.”* The “null hypothesis” is represented by a simple point estimate of 0, and the observed effect by the mean in a normal (or other appropriate) distribution.

In contrast, a Bayesian analysis (on right) *tests the relative consistency of the observed effect with two or more hypotheses*. Those hypotheses, not the observed effect, are conceptualized as ranges of values arrayed in relation to their probability in distributions that account for measurement error and any other sort of uncertainty a researher might have. The relative probability of the observed effect with *each* hypothesis can then be determined by examining where that outcome would fall on the hypotheses’ respective probability distributions.

I left out why I like the latter better. I was after as neutral & accessible an explanation as possible.

Did I succeed? Can you do better?

## Reader Comments (97)

Dan,

How about:

https://doi.org/10.1007/s13164-018-0421-4

Fun question!

The way I do it is to start with the Bayesian approach, and then explain what gets left out to arrive at the Frequentist approach.

First, you need an intuitive, graphical way to describe conditional probabilities. I use Venn diagrams where the area of each region is proportional to the probability. Then the conditional probability P(A|B) is the fraction of area B that is covered by A.

Then it's fairly easy to show that P(H|E) = P(E|H)P(H)/P(E). This is just a simple statement about areas. We interpret E as the evidence seen, and H as the hypothesis we want to prove or disprove. Then P(H|E) is the probability of the hypothesis given the evidence was observed, P(E|H) is the probability of observing the evidence given the hypothesis, and P(H) and P(E) are the prior probabilities of the hypothesis being true and the evidence being seen.

The thing on the left is what we want, but how do we fill in the things on the right? The first one we can have a go at - we need a 'statistical model' for each hypothesis that tells us how likely different observations are if our conjecture about how the world works is true. Some people start with a common statistical model that describes all the possible hypotheses, others consider the model to be uncertain and part of the hypothesis. Both Bayesians and Frequentists treat this bit pretty much the same (although it is not without its own controversies).

The P(E) bit has an uncontroversial meaning, but it's usually unknown and very hard to estimate. However, there is a neat trick we can do to get rid of it. We write the equations for two different hypotheses, and divide one equation by the other. The P(E) then cancels out.

P(H0|E)/P(H1|E) = [P(E|H0)/P(E|H1)] [P(H0)/P(H1)]

The alternative hypothesis H1 can be either something specific, or simply "H0 is not true".

This is where the first major problem arises, because for a vague statement like "H0 is not true" most statistical models are pretty useless at answering the question "What is the probability of the evidence being observed if H0 is not true?" The truth could be ***anything***; any insane rule, any strange weirdness is allowed. If your hypotheses about some parameter are x=0 and x>0, the latter includes possibilities like x=10^(10^(10^100)) and bigger! You've got infinitely many unknowns and no way to pick between them. How can you even calculate that?! The answer is, you can't.

Bayesians usually try to pick a specific H1 that they've got a statistical model for rather than saying "not H0" (and then having to give up). And then they leave open the question of whether there might be a better alternative hypothesis out there. Frequentists just drop all the terms involving H1 as unknowable, and trust that this is not one of those rare awkward situations where this will come back to bite them.

Finally, we come to the last term: the P(H) or P(H0)/P(H1). This is the problem of the priors, and is the biggest hole in the Bayesian approach because there is no objective method for assigning a value to it. Bayesians acknowledge the gap, and either submit intuitively reasonable guesses and assumptions, pick something that seeks to minimise the distorting effect, or leave the question unanswered and only tell you how much the evidence should move you without saying where you should start or where you will end up.

By contrast, the Frequentists again dismiss it as unknowable and ignore the problem.

Thus, the Frequentists are using something like P(H0|E) = P(E|H0) (or at least that it is hand-wavingly proportional to it), and saying that if the observed outcome is very unlikely under the null hypothesis, then the null hypothesis is very unlikely given the evidence. They cover some of the possible ways this can go wrong by separately examining other quantities like the 'power' and 'bias' of a test. The Bayesians are including a bunch of extra variables, but then having to limit the questions they can ask to exclude some, and then arbitrarily guessing at and hand-waving about the remaining values because they've got no objective and rational way to assign them.

The Bayesians are being a bit more explicit about where and how they're fudging the mathematics, but they're both facing the same basic problem. It's Hume's problem of induction. It can't be solved with mathematics, and fundamentally our methods only work because the universe happens to be unreasonably cooperative and regular. "The eternal mystery of the world is its comprehensibility."

NiV,

Did you read the link I dropped above - and if so, how have you modified your belief in things like: "Frequentists just drop all the terms involving H1 as unknowable, and trust that this is not one of those rare awkward situations where this will come back to bite them" and "...Frequentists again dismiss it as unknowable and ignore the problem"?

"Did you read the link I dropped above - and if so, how have you modified your belief ... ?"I skimmed through it, but didn't parse it in detail.

I didn't notice anything to suggest I was wrong. Should I have?

NiV,

I think someone reading your explanation first, then reading that Colling & Szucs paper would lower their belief in your explanation.

Having read Colling & Szucs paper first, then reading your explanation, I don't appreciably modify my belief in direction of your explanation. This is because your explanation seems to be of the form: viewing frequentism as a modified form of Bayesian inference, one notices that frequentists make certain unsafe systematic assumptions. But, frequentism is not a modified form of Bayesian inference (as is spelled out in Colling & Szucs, and certainly elsewhere). Hence, your conditional is undercut by having a dependent clause that you don't support but that Colling & Szucs provide evidence against.

So, the question becomes, why are you explaining frequentism as if it is a modified form of Bayesianism with additional (and not always safe) assumptions?

"So, the question becomes, why are you explaining frequentism as if it is a modified form of Bayesianism with additional (and not always safe) assumptions?"Because to understand the differences between them you first have to put them into a common frame of reference, and it's easier to understand the Frequentist approach starting from the Bayesian framework.

It's like trying to explain the difference between Newtonian physics and Special Relativity. You start by explaining Special Relativity, and then say Newtonian physics is the approximate limit of behaviour at speeds much lower than the speed of light, or equivalently, the limit you'd get if you made the speed of light infinite.

Newtonian physics is not - historically or conceptually - a modified version of Special Relativity. But if you want to understand the differences between the two, then describing Newtonian physics as the low-speed limit of relativity is far easier than starting with Newton and then trying to introduce all the conceptual leaps and counter-intuitive changes that Einstein added.

The problem is that the underlying philosophy behind Frequentist null hypothesis testing is normally not discussed. It's one reason why people misunderstand it so often. Why do they consider p-values, the probability of an event given the null hypothesis, when what you actually want to know is the probability of the null hypothesis being true? They don't say. They just tell you the method, as if it gives you the answer to the exam quesrion. And then afterwards they kind of note in passing, like as a minor footnote of no special importance, really, don't worry about it, that p-values don't in fact tell you the probability of the hypothesis being true, and it's all a bit more complicated. And then they stop, and don't tell you what comes next.

However, if you look first at the Bayesian formula, and then note that all the bits they leave out are precisely the bits nobody knows how to calculate, it all suddenly makes sense! The reason they use p-values as a proxy for the truth of the hypothesis is that they can't generally formulate the alternative hypothesis or the priors in a way that makes mathematical sense.

There's more to it than that, of course. But if you want a 10-minute summary of the *essential* core difference for a smart person, it's a good place to start.

--

And of course I was hoping to start a discussion. I'm glad that bit worked!

Most scientists would like to know the probability (P(H)) for their hypothesis H, given their data, assuming nothing else. Unfortunately, this isn't possible.

Frequentists keep the "assuming nothing else", discard any hope for P(H) and opt for something else instead that is still useful. Bayesians try to keep P(H) and discard any hope for "assuming nothing else".

Frequentism gives you (if done correctly) something that will "

objectivelytrack truth". Bayesianism doesn't guarantee that, but instead gives answers that are (apparently) more intuitive and easier to manipulate and interpret.However, I wonder if scientists had used Bayesianism extensively for nearly 100 years (as they have frequentism), if they'd be upset with Bayesianism's ability to lead them astray. Being apparently easy to understand isn't always a good thing. Consider the discussion in Colling and Szucs about the stopping rule - it's nuanced in Bayesianism even though it is often thought to not be. And there's always the problem of the priors.

"Frequentists keep the "assuming nothing else", discard any hope for P(H) and opt for something else instead that is still useful. Bayesians try to keep P(H) and discard any hope for "assuming nothing else"."Yes, I'd say that was a fair summary. :-)

"However, I wonder if scientists had used Bayesianism extensively for nearly 100 years (as they have frequentism), if they'd be upset with Bayesianism's ability to lead them astray."Oh, scientists using Bayesian methods extensively complain about its shortcomings, too! I've been involved in debates on the relative merits of uniform priors versus non-informative priors - which even scientists used to the Bayesian approach get confused about. You might also be interested in Lindley's paradox, if you've not come across it before.

In practice, all statisticians I've ever come across use *both*, as they deem appropriate, and always have. Fisher (probably the leading Frequentist) praised Bayes and regarded his updating method as uncontroversially the right approach for estimation. It was primarily the

interpretationof the result as the true, inferred probability of an unobserved parameter (Fisher called it 'inverse probability', but it's known as the posterior distribution now) rather than an estimate subject to error, particularly in the case of uniform priors, that he regarded as a mistake. The debate is long and subtle, and I don't think it's possible to do justice to it briefly. This can only be a starting point.@NiV-- Put me in the "leave the question unanswered and only tell you how much the evidence should move you without saying where you should start or where you will end up" camp (see pp. 390-91).

A problem with Bayesianism that I think is underappreciated is that a Bayesian result usually assumes that the new evidence is completely independent of anything that one used to generate one's prior.

Indeed. Very true.

Another is that the statistical model by which one calculates the probability of each outcome under each hypothesis is also potentially open to question; a part of the hypothesis, and subject to updating as new evidence comes in. If your current model says that the evidence seen is highly unlikely under *any* of the available hypotheses, then maybe it's the

modelthat's wrong. (It's where the models come from in the first place.)If you're science curious, would you rather see frequentist or Bayesian results?

Understanding science curious as Dan has spelled it out, I'd guess that SC folks would rather see frequentist results. I'm basing this on the feeling that frequentism gives one of encouraging surprise of a certain type - a "maybe I'm completely wrong about this" kind of surprise that is offered by frequentist statistics objectively tracking truth independently of one's priors. Bayesian statistics never say you were completely wrong. They imply no evaluation on what you should have believed prior to seeing the experiment. To SC folks (high enough in OSI to understand this much about Bayesianism and frequentism), this aspect of Bayesianism could be a buzz-kill. Which might just make them less SC.

Here's a test for the frequentist v. Bayesian methodologies:

-----------------------------------------------------------------------------------

"Recent Census Bureau projections indicate that racial/ethnic minorities will comprise over 50% of the U.S. population by 2042, effectively creating a so-called “majority–minority” nation. Across four experiments, we explore how presenting information about these changing racial demographics influences White Americans’ racial attitudes.

[..]... Whites exposed to the racial demographic shift information preferred interactions/settings with their own ethnic group over minority ethnic groups; expressed more negative attitudes toward Latinos, Blacks, and Asian Americans; and expressed more automatic pro-White/anti-minority bias..."

http://groups.psych.northwestern.edu/spcl/documents/Craig%20&%20Richeson%202014%20PSPB.pdf

----------------------------------------------------------------------------------------------------------------------------------------------------

That's why the entire American right opposed Johnson's Civil Rights legislation ending European preference in immigration. More diversity inevitably leads to civil strife.

But the really interesting part is that, spooked at the effect of its projections (really common sense requiring no demographic forecast) The Census Bureau has now stopped posting projections for the year minorities will cross 50%. That was the "prior" in 1965, and it has now translated into observable frequencies - that's where all these "divergences" and "polarization" come from.

Trying to use "science curiosity" as explanatory variable is like counting angels dancing on heads of pins - nobody knows, few care. Survival instinct is however an overriding consideration.

http://groups.psych.northwestern.edu/spcl/documents/Craig%20&%20Richeson%202014%20PSPB.pdf

Better link - sorry formatting isn't always automatic.

PS if all else fails, kindly follow link to article given in NYT

https://www.nytimes.com/2018/11/22/us/white-americans-minority-population.html

PPS Johnson's immigration legislation has received less attention than the other twin catastrophes of his presidency, hyperinflation leading to monetary collapse and the Vietnam war, but in the long run it's proved more destructive than either. And keep in mind said legislation concerned legal immigrants, not migrant invaders - on the latter, the Right firmly backs Col. Lind (Rtd, USMC): https://www.traditionalright.com/the-view-from-olympus-invasion/

Ecoute -

That's why the entire American right opposed Johnson's Civil Rights legislation ending European preference in immigration. More diversity inevitably leads to civil strife.Really? How are you defining civil strife? How are you measuring diversity?

Is the US more or less diverse than 25, 50, 100, or 200 years ago? Is there more or less civil strife?

Which countries have the most civil strife - those with the most diversity or the least diversity? Seems to me that there is no consistent causal relationship between diversity and civil strife. Seems to me that the degree of civil strife in a society is a function of many factors. Why do you single out that one factor? Are you cherry-picking? Is there any connection between your singling out that one factor as causal for strife, and your determination that antisemitism had nothing to do with motivating a man who made comments about killing Jews to murder Jews as they worshiped in a synagogue?

What evidence do you use to make your claim?

@Jonathan-- on your problem w/ Bayesianism: See

Kahan, D.M. The Politically Motivated Reasoning Paradigm, Part 1: What Politically Motivated Reasoning Is and How to Measure It. in Emerging Trends in the Social and Behavioral Sciences (John Wiley & Sons, Inc., 2016).; and

Kahan, D.M. Laws of Cognition and the Cognition of Law. Cognition 135, 56-60 (2015)

Dan,

I read the PRMP papers, so won't read again as they are already incorporated in my priors (and wouldn't want to increase their impact merely by rereading!).

That Laws of Cognition paper is new to me. How should I decide how much of what's in this paper is based on things I've already incorporated into my priors (having read 37 other papers by you)? I'll read it anyway, on off chance it contains new things...

Also, there's evidence that humans aren't, and probably can't be, Bayesian:

https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1006572

And, remember,

ought implies can.Another issue I have with Bayesianism is how hard it is for people to conceptualize even the approximate odds of a very improbable event. I think of this as a bias due to our use of the common probability scale of [0..1], and how difficult it is for people to conceive of the difference between 0.00001 and 10^-50 (such an issue might be less of a problem with an information metric like surprisal). In Bayesian terms, this difference is huge - and counters the common belief that biased priors eventually wash out (not in the lifetime of the universe!). It's likely that, if this isn't appreciated, wide use of Bayesianism would lead to an increasing embrace of pseudoscientific theories.

Here's a good read:

Bayesian Reasoning for Intelligent Peoplehttp://tuvalu.santafe.edu/~simon/br.pdf

@Jonatah-- the Laws of Cognition paper discusses mechanisms other than PMR that complicate formation of truth-convergent LRs

Jonathan - Dan's article is orders of magnitude better than the lemon you linked (with due deference to Mr DeDeo). Dan shows a touching concern for ultimately getting to The Truth, a concern wholly absent from the other paper. Though I wish he hadn't preceded it by the term "converging on", which reminds me of a horrible time spent trying to understand path integrals, until one day I finally gathered up all my courage and asked "why do we try to corral all these infinities into convergence when we already know the particle will travel in a straight line?" and was promptly answered with "we don't understand these infinities, but we can make a stab at calculating them". That permanently ended my interest in string theory, but it was the honest answer.

If DeDeo wanted to re-name his paper "Bayesian Reasoning for Honest People" he would accept betting on his belief that Hillary Clinton would win in 2016 with 99% probability. If you claim to understand your Bayesian calculation you bet on it. For the record, DeDeo quotes approvingly Wang, but fails to mention Wang is an honest man who did bet on his own 99% prior by saying if Clinton didn't win, he, Wang, would eat an insect on live television. And he did.

Dan - unless I miss something in your article, I don't see how a search for truth CAN be reconciled with PMR. Don't you inevitably end up with "Cleopatra was black"-type inanities?

Here is NYU savant Kwame Appiah:

"...Dick Hernstein, (who was a psychologist at Harvard) ..was interested in the connection between race and intelligence, and he thought that once you thought that there was a connection between race and intelligence, you were invested in a view which has unfortunate social consequences, independently of whether it was true, and so there he was talking about the ways in which you could be motivated, in that case intellectually motivated, by an attachment to a thesis, to do things that had bad effects."

http://the-orb.org/2017/09/20/interview-with-kwame-anthony-appiah/

I love "independently of whether it was true"!

Here's a pro-Bayesian, pro-climate change consensus (trigger warning: 97% figure), anti-cultural cognition (and other untamed Bayesianism) paper that says that science communicators (notably Dan, cited) just don't get Bayesianism properly:

https://onlinelibrary.wiley.com/doi/full/10.1111/tops.12173

Jonathan -

A simple statement about consensus hides multiple complex and overlapping evidential signifiers, including but not limited to the reliability of the communicator, the sources providing the consensus information, and their degree of independence.Yes, that. Studying the effect of providing information on the "consensus" in a "lab-based" context is of very limited value, IMO.

@Jonathan-- it's a good paper. I don't feel my understanding of Bayesianism, or my use of it to help delineate the presence & consequences of motivated reasoning, is challenged by it. Indeed, in other works, Hahn has recognized that endogeneity of prior & likelihood ratio prevents Bayesian updating from being truth-convergent.

Jonathan -

I'm listening to The Ezra Klein Show with @TuneIn. #NowPlaying http://tun.in/tiQXTH

Haidt.

I'm finding him increasingly annoying. My most annoyed moment came at about 58 minutes in, where - without any apparent sense of irony - he blames triggering-happy, self-victimizing leftists (who supposedly always say "Punching down always good, punching up always bad") for triggering the rightwing to be hateful towards what the left has to say.

Reminds me of when people argue that "skeptics" wouldn't be so "skeptical" if those name-calling, capitalism-hating, pro-childstarving, pocket-lining, data manipulating, dogma-mongering "alarmists" would just stop being mean and calling them "deniers" and stop talking about a consensus (because they can't comprehend their appeal to authority and ad populum fallacy, doncha know).

link drop:

https://phys.org/news/2018-11-science-knowledge-shifted-religious-political.html

Unfortunately, can't find non-paywall version. So, in it's place, will continue my usual rant:

https://www.newscientist.com/article/mg24032052-900-time-to-break-academic-publishings-stranglehold-on-research/

Joshua,

Am ambivalent about Haidt. I agree that he does seem to go overboard with his social commentary, but his science seems to be quite good. Moral foundations was one of the studies that got an OK from Many Labs 2. Also, I agree with him that there are problems with having a left dominated academia in the social sciences (regardless of the cause): https://psyarxiv.com/e4xw3/.

Been wondering ever since Haidt cleverly named the pro-right foundations "binding" and the pro-left foundations "individuating" whether he is mediating his own behavior and fame as a form of social corrective. Who knows, maybe he was partially responsible for that "traditional liberal" moniker that gets my goat.

Dan,

About that second Hahn et al "How good is your evidence..." paper. This brings up another of my pet peeves with Bayesianism, related to one above.

Bayesians know about surprisal, but still seem to prefer to eye-ball prob-scale [0..1] effects, and then suffer bias from that examination. Hahn shows off some very nice charts which have a prob-scale [0..1] dimension, and then says stuff somewhat like:

See, they get closer - isn't that so nice!. But the surprisal scale (-log prob), which has the advantage of having actual physical support (its entropy!, its actual information!) and other mathematical niceness, Bayesian inference doesn't push things closer together at all. This is because Bayes theorem in the log world is perfectly linear:-log(P(H|E)) = log(P(E)) - log(P(E|H)) - log(P(H))

If we're some surprisal (information/entropy) distance apart, and we both modify our beliefs subject to evidence we both deem (hopefully accurately) independent of our priors like perfect Bayesians, we end up exactly as far apart in surprisal terms as before. Suddenly, inference from pretty pictures doesn't provide a warm fuzzy.

This is nearly the same as that issue I raised above with 0.00001 vs. 10^-50. But, turned around a bit. If I'm 0.99 sure of X and you're 0.9999 sure of X, we probably get along much better than if I'm 0.1 sure of X and you're 0.9 sure of X (assuming X is important enough to quarrel about). But, if we were true Bayesians all the way, we shouldn't! We should have about 10 times more animosity towards each other at 0.99 vs. 0.9999 than at 0.1 vs. 0.9. Hahn (and others) appear to me to be arguing that normatively we should be Bayesians, and then drop Bayesianism in favor of our prob-scale bias to determine our social/practical belief distance. Although, to be fair, some Bayesians argue instead that we should be perfect Vulcans and communicate entirely by infinite precision mathematical quantities.

My favorite interpretation is that since we're probably much friendlier at 0.99 vs. 0.9999 than at 0.1 vs. 0.9, then that means we're not very good at the conversion between beliefs and numeric values. Maybe this is related to your Likert-scales-found-wanting finding...

Jonathan - I don't understand your assumption on difficulty in comparing low-probability events.

See here:

https://www.theguardian.com/society/2018/nov/26/rare-cancer-linked-breast-implant-used-by-millions-women-lymphoma

If I were considering having such implants and were told the cancer risk is (per UK estimate) 1 in 30,000 I would probably ignore it. But if, as per Australian estimate, the risk is closer to 1 in 1,000, I probably wouldn't go ahead. Nor do I think such a calculation is uncommon.

Related, another reason I didn't think much of the DeDeo paper: re-formulating the problem as a bet, which he refuses to do, clarifies matters enormously. How much would you pay to insure your home against space meteorites (assuming such insurance is available). That risk is obviously smaller than 1 in 30,000 as in the implants. Would you pay a quarter? All insurance pricing is based on some Bayesian calculation. Frequentist odds aren't as important.

Ecoute,

This Searle quote from the DeDeo paper comes close to one issue I have:

I feel similarly to Searle here - I can't determine what odds I'd need in a life against a quarter bet, and it seems that no odds would satisfy me. I can try to explain this, but I'm not sure the explanation matches with the actual reason. Certainly, part of it has to do wit DeDeo's point about that being the last bet I'd lose (although, think of the marketing possibility - you'll never lose another bet again!). Another explanation might be that my uncertainty about extreme odds/probabilities/values clouds my ability to judge the fairness of such bets to be within my comfort zone. I would probably instead pay to stay out of situations where such a bet is forced. But, how much?

(BTW - this doesn't mean I think Searle is rational - he's been accused of things that, if true, would suggest otherwise. Or, perhaps he is super-rational in a Machiavellian sense.)

Also, as you probably suspect, "All insurance pricing is based on some Bayesian calculation" does not increase my ease with Bayesianism, but instead decreases my (already low) trust in insurance. But, as Hahn says, that might be a purely Bayesian calculation on my part! Dan's probably right here instead - if it were socialized (even partially) insurance of some type instead of entirely commercial (hence left instead of right leaning), I'd have much higher trust, provided I can manage to keep Republicans away from screwing it up. As of earlier this month, my trust in that last bit has increased.

Oh - this is an idea for a test to differentiate pure Bayesianism from cultural cognition: take some partisan issue such that subjects diverge further when exposed to scientific claims (such as climate change or nuclear power), and see if they claim as well after diverging that they already were completely familiar with that claim. If purely Bayesian, they must either not budge or say that the claim is somehow new to them. If instead Dan is correct that they are corrupted Bayesian, they will both diverge and say the claim is completely familiar (hence should already be accounted for in priors). Although there are alternative explanations to Dan's for that, at least we can rule out pure Bayesianism.

@Jonathan-- I think we make such bets 100's or 1000's of times per day. E.g., did you drive to work today? I haven't read the paper-- maybe it makes/anticipates this point.

Dan,

"I think we make such bets 100's or 1000's of times per day. E.g., did you drive to work today?"I suspect that people don't frame activities such as driving to work as a bet, although it is. It's probably the lack of such framing that allows them to make the drive. And then they text while driving, again without framing that as a bet. I suspect that in these cases, people consider that the appearance of control that they have in the situation is such that they think the risk of an accident is ~0 - or, rather that they don't even think of the risk at all because of the appearance of control. That people take such risks does not imply that they calculate such risks, and certainly not that they do Bayesian calculations. It also doesn't imply that they would take those same risks if those behaviors are framed as bets in ways that nullified the appearance of control.

Does "socialized" insurance mean subsidized insurance? If yes, it's not insurance at all, and neither Bayesian nor frequentist probabilities are relevant.

On climate change, today Krugman (who oddly used to be a good economist) comes up with a classic - discarding probabilities altogether and instead formulating the problem in Manichaean terms:

"..While Donald Trump is a prime example of the depravity of climate denial, this is an issue on which his whole party went over to the dark side years ago. Republicans don’t just have bad ideas; at this point, they are, necessarily, bad people."

https://www.nytimes.com/2018/11/26/opinion/climate-change-denial-republican.html

I hope I'm not in breach of cultural cognition rules if I call this ()&*^%$.

"...discarding probabilities altogether and instead formulating the problem in Manichaean terms"Coincidentally, am reading about just that now:

The Binary Bias: A Systematic Distortion in the Integration of Information"...at this point, they are, necessarily, bad people"Yes, certainly sounds both Manichaean and prejudicial. Hmmm... wherever have I seen such Manichaean prejudice about outgroup members before?Jonathan - from your Binary Bias article:

" Highly numerate

individuals have less-distorted probability functions

(Patalano, Saltiel, Machlin, & Barth, 2015) and may be

less susceptible to the binary bias."

But just to take the latest post here: there is absolutely no doubt Krugman is highly numerate - he got his doctorate at Tech before admissions / graduations were polluted by political correctness. Why, then, is he sounding like some mad prophet of the "Repent, sinners, the end of the world is nigh" variety? Some overwhelming Bayesian prior is obviously clouding his judgement.

And if you have managed to remember the answer to your question "...Hmmm... wherever have I seen such Manichaean prejudice about outgroup members before?" I'd like to see it.

There's a cottage industry of attempts at fitting Bayesianism to empirical human probability judgments. Here's a new one:

https://psyarxiv.com/af9vy/

I'm finding these fascinating, although sounding a bit like epicycles:

This paper addresses specifically the conservatism bias. Which hadn't occurred to me when thinking about how bad people are at near 0 or near 1 prob estimation. Perhaps we know we're bad, and stay away.

Or, for fans of just-so-evo stories, perhaps we reserve near 0 and near 1 probs categorically for specific semantic (non-numerical) purposes because these probs haven't been encountered in natural environments (organisms won't likely encounter enough data to classify something that close to 0 or 1 based on rational empiricism alone) often enough for our innate prob estimation to have needed to evolve to accommodate sufficiently exactly. Consider the Gran Sasso neutrino event of 2011 discussed in that DeDeo paper: "Most physicists have very strong beliefs in special relativity; they form these on the basis of both experiments, and also on the general coherence of the theory itself—what William Whewell first called “consilience”". That second coherence-based adjustment of belief is what I'm referring to, and it's that which prevented scientists from falling into the pseudoscientific trap that some journalists of the event fell into - because those journalists' appreciation of the coherence of relativity wasn't strong enough to move fast-than-light travel into that close-to-0 category that one can't escape with evidence alone. (Also the scoop premium in journalism and man-bites-dog stories thing, certainly. Any good just-so-evo story needs friends.)

"Here's a pro-Bayesian, pro-climate change consensus (trigger warning: 97% figure), anti-cultural cognition (and other untamed Bayesianism) paper that says that science communicators (notably Dan, cited) just don't get Bayesianism properly"Excellent paper! Thanks for the trigger warning, by the way, but actually I approve of their use of the example because they do the analysis of how people modeled as Bayesians ought to react to it properly.

"If we're some surprisal (information/entropy) distance apart, and we both modify our beliefs subject to evidence we both deem (hopefully accurately) independent of our priors like perfect Bayesians, we end up exactly as far apart in surprisal terms as before."Per the usual Bayesian assumptions, yes. (And I agree with your point.)

But I'd note the caveat that this assumes all parties also use the same statistical model to calculate the likelihood function. People using different models will calculate different likelihood ratios and jump by different amounts. And the model itself may be subject to being updated by the evidence. In the real world they're not a given. We have to deduce those from observation, too.

And if you change your

model of the worldas a result of new evidence, that would require you to re-evaluate all thepreviousevidence you've seen, and it's effects on your priors!Since your model of the world has to start off with unsupported priors too, there's no more guarantee that people will converge on models than that they converge on subject matter beliefs derived using them.

Textbooks start with the simplified 'lies to children' version, and then slowly fill in the gaps. Since everyone drops out of studying the subject at some point, virtually everyone is left with some level of the 'lies to children' story. Our understanding of Bayesian inference is no exception.

NiV,

Yes - excellent point (again) about models and their dependencies - which you've made before (and I agreed with before). However, I still don't think Dan is convinced of this - my impression is that he (still) thinks that belief divergence necessarily implies corrupted Bayesianism. I think it implies neither Bayesianism nor corruption, but certainly an uncorrupted model-accommodating Bayesian network could work this way as well.

I think that the independence test I proposed would give a clue here. If evidence is presented that might not be independent of subjects' priors - such as scientific consensus arguments - and belief movement (in either direction) based on this evidence is positively correlated with claims that the evidence is familiar to the subject, then I think this would be a problem for Bayesianism, even the corrupted varieties that Dan has proposed, since even these should filter out familiar evidence already incorporated into priors. Seems it would instead imply a more affective explanation - one isn't updating priors, calculating likelihoods, or any other calculation so much as reacting to a stimulus one perceives as a threat. One could go further and test if, when repeated, this reaction follows the typical gradual habituation curves that repeated exposure treatment has on other fears. If it doesn't stop immediately (suppose the subject was lying about familiarity initially - they would be the second time they hear the same argument), then I think Bayesianism is out as an explanation, even the model-accommodating network variety.

Jonathan - I liked the Hahn et al article very much also (and I also think use of the 97% is justified in this case) but I don't see how your interpretation of worldview revisions applies. See poll data:

http://climatecommunication.yale.edu/visualizations-data/ycom-us-2018/?est=happening&type=value&geo=county

Essentially, this says people may or may not believe the climate warnings, but they're definitely sick and tired of hearing them. Two-thirds of all people NEVER want to discuss the topic again, and an overwhelming majority (77%) think that media should mention it at most once a month. I hear the incoming House majority plans to make a major issue of it - which naturally cheered me up no end.

An apparently unrelated datum on cultural cognition, today's victory of Carlsen over Caruana. I know many chess experts; all of them supported Norway's Carlsen, even though Caruana was playing for the US. That level of play is beyond me, but I also rooted for Carlsen - somehow Caruana struck me as an alien interloper. Here's a brief video:

https://www.youtube.com/watch?v=N3YHvvivozM

Ecoute,

"Two-thirds of all people NEVER want to discuss the topic again, and an overwhelming majority (77%) think that media should mention it at most once a month."Do you really parse that data that way, or you just baiting? To me, it looks like 2/3 never discuss it, and 77% hear about it at most once a month in the media. I don't see anything normative in those questions. Too bad this map doesn't show reps vs. their constituents.

As for the incoming House majority - they apparently have Trump derangement syndrome to thank:

http://www.pewresearch.org/fact-tank/2018/11/29/in-midterm-voting-decisions-policies-took-a-back-seat-to-partisanship/

Perhaps they should send him a nice Christmas card and an ugly sweater (sign of affection).

Dazzling. https://www.youtube.com/watch?v=7JwlOjMI420

Jonathan - as you can tell I was reading fast while watching the final game, so if I mis-read I apologize. Will re-read later. But I cannot be suspected of "baiting"!!!

Jonathan, thanks!

"If evidence is presented that might not be independent of subjects' priors - such as scientific consensus arguments - and belief movement (in either direction) based on this evidence is positively correlated with claims that the evidence is familiar to the subject, then I think this would be a problem for Bayesianism, even the corrupted varieties that Dan has proposed, since even these should filter out familiar evidence already incorporated into priors."Possibly. You would have to be very careful about how you tested that, though.

For example, it could be argued (assuming one accepted the 'ad populam' heuristic) that for any claim there will be some unknown proportion of affirming and disagreeing arguments floating around the debate-o-sphere, and every time you come across one, it's a data point regarding the frequency of each. If you hear ten arguments and they all make the same claim, with none against, that strongly suggests that the majority of the arguments circulating support that claim. If you come across six for and four against, you would more likely conclude the issue is contested. So even hearing evidence that you've heard several times before in every detail could arguably be taken as additional confirmation. The evidence itself is repeated, but the multiple events in which it is transmitted to you are supposed independent.

I personally don't doubt that human reasoning is at best a heavy approximation to Bayesian updating consisting of a bunch of "good enough" heuristics - both evolved and learnt through experience. The choice is based on a trade-off between computational effort and accuracy, implemented on a non-numeric neural-net-type computer made of meat. It would be silly to think it was going to be perfectly Bayesian - it just has to be close enough for users to usually survive and have offspring. However, I do also think it's usually a hell of a lot better approximation than the simplistic textbook models many theorists propose! Humans very often do model multiple mechanisms of uncertainty and unreliability in information sources in considerable detail, including the potential for non-independence.

Not always, of course. Some people assume that if they're told 97% of scientists agree on something, that's extremely unlikely to happen coincidentally. Others consider the possibility, for example, that most scientists haven't looked at the data themselves, but get their beliefs from what other scientists say, same as the rest of us. (And obviously if they happen to know the actual figure direct from the source survey was 82%, maybe even less, yet all these 'scientists' keep citing the 97% figure like they believed that was what the evidence said, they do more than just "consider" it!) The amount of detail people put into their mental models varies considerably - and related to their motivation and preconceptions, of course - so the first thing you need to do in any experiment to understand how people think is to ask them detailed questions about what precise definitions, models, and mechanisms they're assuming. If you try to do some statistical experiment to tease out details of their mental mechanisms assuming everyone defines terms like "expert" or "global warming" or "recent" or "significant" or "average" or even "temperature" the same way, you're quite likely to get muddled, meaningless results.

Some people assume that if they're told 97% of scientists agree on something, that's extremely unlikely to happen coincidentally. Others consider the possibility, for example, that most scientists haven't looked at the data themselves, but get their beliefs from what other scientists say, same as the rest of us. (And obviously if they happen to know the actual figure direct from the source survey was 82%, maybe even less, yet all these 'scientists' keep citing the 97% figure like they believed that was what the evidence said, they do more than just "consider"Perhaps for "some people."

But my guess is that most people accept a scientific consensus as a useful but not dispositive decision-making heuristic in most cases, except when they are "motivated" otherwise - in which case they get into endelss arguments to quantify the magnitude of the consensus (the outcomes of which are, of course, easily predictable in association with ideology), or about the value of consensus, etc..

"But my guess is that most people accept a scientific consensus as a useful but not dispositive decision-making heuristic in most cases, except when they are "motivated" otherwise..."Yes, Joshua. That's what I said. :-)

Some people are

"motivated"to take it as dispositive.I wish there was consensus on the consensus number. For instance, there's:

https://www.skepticalscience.com/global-warming-scientific-consensus-intermediate.htm

which cites several different studies, including ones like Doran 2009 and Anderegg 2010, which get to the famous 97% figure with climate scientists most active, not papers. At least, when a popsci article mentions the 97% number, it should cite the source (so, kudos to skepticalscience here). But, that's a pet peeve of mine (have lots of them, making up for all those cats I don't have) - popsci should always cite (and link) its (hopefully non-paywalled) sources so the curious public can take a look for themselves. The public might learn something, and thereby find meaning in learning, at hopefully higher rates than the meager "learning" figures in:

http://www.pewresearch.org/fact-tank/2018/11/30/members-of-both-parties-find-meaning-in-family-but-differ-when-it-comes-to-faith/

Guess we should be thankful learning broke the top 9.