follow CCP

Recent blog entries
« A Tale of (the Tales Told About) Two Expert Consensus Reports: Death Penalty & Gun Control | Main | Wisdom from Silver’s Signal & Noise, part 2: Climate change & the political perils of forecasting maturation »
Wednesday
Jan022013

Chewing the fat, so to speak...

I've already exhausted my allotted time for blogging in answering interesting comments related to the post on Silver's climate change wisdom. I invite others to weigh in (but not on whether Mann is a great climate scientist; see my post update on that).

In particular, I'd like help (Larry has provided a ton, but I'm greedy) on what is right/wrong/incisive/incomplete/provocative/troubling/paradoxical/inherently contradictory etc. about my statement, "Gaps between prediction and reality are not evidence of a deficiency in method. They are just evidence--information that is reprocessed as part of the method of generating increasingly precise and accurate probabilistic estimates." Also the questions of (a) how forecasting model imprecision or imperfection should affect policymaking proposals & even more interesting (given the orientation of this blog) (b) how to communicate or talk about this practical dilemma. (Contributions should be added to that comment thread.)

Two more things to think about, complements of Maggie Wittlin:

1. Who is afraid of obesity & why?  Maggie notes "new meta-analysis finds that overweight people (and, with less confidence, people with grade 1 obesity) have a lower risk of mortality than people with BMIs in the 'normal' range" and wonders, as do I, how cultural outlooks or other sources of motivated reasoning affect reactions to evidence like this -- or of the health consequences of obesity generally.

2. Forget terrorism; we're all going to die from an asteroid. Maggie also puts my anxiety about magnitude 7-8-9 terrorism into context by pointing out that the size/energy-releasing-potential of asteroid impacts on earth also follow a power-law distribution.  Given the impact (so to speak) of civilization-destroying asteroid collision, isn't preparing to protect earth from such a fate (however improbable) yet another thing that we need to do but are being distracted from doing by OHS's rules on removing shoes at airport security-screening stations?! I could do some research but Aaron Clauset's spontaneous & generous supply of references for the likelihood of "large" terrorism attacks makes me hope that some other generous person who knows the literature here will point us to useful sources.

PrintView Printer Friendly Version

EmailEmail Article to Friend

References (1)

References allow you to track sources for this article, as well as articles that were written in response to this article.
  • Response
    Response: osal

Reader Comments (16)

Love your work, but please read the abstract of that obesity paper more carefully. It clearly states that obesity does produce higher mortality rates, but merely being overweight does not. That suggests that our current definition of "normal" and "healthy" is too narrow, it doesn't tell us that our definition of "obesity" is in error. (As someone who was rejected from the military for being too thin, I'm not surprised by this finding. The health and diet fields are loaded with ideology, IMHO.)

Here's the actual conclusion: "Conclusions and Relevance Relative to normal weight, both obesity (all grades) and grades 2 and 3 obesity were associated with significantly higher all-cause mortality. Grade 1 obesity overall was not associated with higher mortality, and overweight was associated with significantly lower all-cause mortality. The use of predefined standard BMI groupings can facilitate between-study comparisons."

January 2, 2013 | Unregistered CommenterErik

@Erik:

Motivated reasoning on my part; I was out the door to McDonalds ... damn.


I was just being very very very very sloppy... Thanks.

January 2, 2013 | Registered CommenterDan Kahan

Erik-- If I'm reading the study correctly, it does suggest that our definition of "obesity" is over-inclusive. As you note, Grade 1 obesity was not associated with a higher mortality risk; it instead appeared to be associated with a lower mortality risk than normal BMIs. (Our "normal" definition may also be over-inclusive on the low end; I don't think it's clear how much of an effect that might have.)

January 2, 2013 | Unregistered CommenterMW

No worries, I'm not terribly well informed about the obesity wars, but I smelled a rat. :)

While I'm primarily a historian of the physical sciences, I've been around long enough to have seen the BMI index, as well as the measurement methodology, be revised. The current version seems to penalize very muscular people (especially women), rating them as "overweight" when they're just very strong. I suspect that's a reason that the "overweight" category does better in health outcomes than one would expect. But I haven't seen the whole paper yet and perhaps they compensated for this effect.

January 2, 2013 | Unregistered CommenterErik

MW

it instead appeared to be associated with a lower mortality risk than normal BMIs.

Where are you getting that? What I see is that grade one obesity is not associated with higher risk. That doesn't equal is associated with lower risk.

Fascinating study, Dan. Thanks for pointing it out. I wonder how the authors might relate their study to the (I believe widely accepted) association between higher BMI and risk for cardiovascular disease and diabetes? I suppose those associations might also be contingent on above level one obesity, but I think not (I will check into that with an epidemiologist friend/student who does research on the association between diabeties and obesity).

Do you have any links to discussion w/r/t the limitations of this study (such as the limitations of "all cause mortality" that might related to my question)? The discussion included in the paper itself is frustratingly brief.

January 2, 2013 | Unregistered CommenterJoshua

Joshua -- The Results section of the paper's abstract says the summary hazard ratio for grade 1 obesity was .95 (95% CI, 0.88-1.01) relative to normal weight, which suggests lower risk, although I haven't read the study carefully.

January 3, 2013 | Unregistered CommenterMW

@ MW You're right, now I'm being a bit sloppy. The study does suggest that the "obesity" category is over-broad, and sweeps in BMI ratings that don't appear to be related to higher mortality.

All of which is to say, if mortality is the only criteria by which BMI categories have been established, they've been established badly. But the historian in me suspects that more went into making these categories than merely mortality risk, and perhaps they'll remain useful once those other criteria are accounted for.

January 3, 2013 | Unregistered CommenterErik

Dan, I would say your definition needs an explanatory conditional to answer Larry's point.

It needs to be understood that evidence can at first validate and then invalidate a method. I cannot remember the author, though I beleive Joshua linked it one time. Statements are always open to being falsified in science. So, it is a kind of faith to accept any science as proven true.

"Truth" requires a framework, and new evidence can change the framework such as to cause a re-orientation rather than an outright falsification. This is how most people "see" science.

I offer ""Gaps between prediction and reality are not evidence of a deficiency in method. They are just evidence--information that is reprocessed as part of the method of generating increasingly precise and accurate probabilistic estimates.These increasingly precise and accurate probabilistic estimates reflect the increased sophistication of the model and a better understanding of the phenomena under study."

It is that understanding that leads to a shift and a considered falsification of the prior model; otherwise, the result is more likley to be an incremental improvement rather than a fundamental shift.

January 3, 2013 | Unregistered Commenterjohnfpittman

"Gaps between prediction and reality are not evidence of a deficiency in method. They are just evidence--information that is reprocessed as part of the method of generating increasingly precise and accurate probabilistic estimates."

Which method are you talking about? The scientific method generally, or the method of those applying science in a particular case?

The usual process is observation, hypothesis formation, hypothesis testing, model formulation, model validation and verification, and then policy projections. You gather data and make guesses as to what's going on. You gather more data and try to eliminate all the alternatives. You build a quantified model able to make predictions based on the theory, document the assumptions, limitations, and uncertainties, and then gather more data to test and confirm both the predictions and the limits. When you have a validated model that you have confirmed can make verifiable predictions over the range of inputs of interest, with a useful accuracy, then you can use to to project the consequences of different policies. You then hand these over to the policymakers, who weigh costs and benefits, and decide what to do.

In the early stages of this process, you will propose many different hypotheses and models that will turn out to be wrong. That's fine, and normal for science. But it's not normal science to take these interim hypotheses and models and apply them in the policy-making process, as if the science was settled and the conclusions were certain. And if a theory/model has been presented as the outcome at the end of this process, and then it is shown that its predictions were wrong (i.e. outside the stated uncertainty bounds), it suggests something has gone wrong during the validation process. It's always possible that a wrong hypothesis/model passed many checks and tests, and possible that a correct model can get it totally wrong by freakish chance, but neither is very likely. Given the known fallibility of human institutions and behaviours, by far the most likely hypothesis is that something has gone wrong in the process, and it hasn't been applied correctly. This has happened often in the history of science.

Climate science is sometimes quite open about it. They say: "The approaches used in detection and attribution research described above cannot fully account for all uncertainties, and thus ultimately expert judgement is required to give a calibrated assessment of whether a specific cause is responsible for a given climate change." Unequivocal attribution is not possible. The approaches used so far have not been able to account for all uncertainties yet, and thus have not shown that observations are consistent with the proposed cause and inconsistent with any physically plausible alternative cause. The conlusion is not a scientific output (meaning shown by methods designed to circumvent or eliminate any subconscious possible researcher bias), but is instead based on the educated opinion of people doing the research.

The problem is that given the high political and economic costs of reducing CO2 emissions, only a much more unequivocal conclusion would make immediate action the right policy. But scientists are human too, and want to see the world a better place, as viewed from their own particular political perspective. They are more risk-averse to environmental risks generally, and want to eliminate even the possibility of disasterous climate change. That's perfectly normal. And there is an understandable temptation therefore to downplay the uncertainties and the lack of scientific validation, and act as if it was far more solid than it is. Politically that would be more effective.

And obviously all the other political viewpoints assess the risks and the importance of the uncertainties differently. They're only human, too.

But this is just the sort of situation science is designed for. The aim of good science is to design an experiment to demonstrate a conclusion in spite of all the observer's biases. We go to all the extra expense of doing double blind experiments because it's necessary. Scientists are biased, without necessarily being dishonest, and their biases can affect experimental outcomes without anybody realising.

Good scientists know they're biased, and science may be defined as that set of measures we can take to avoid fooling ourselves. They seek out people with different biases to challenge their opinions. They document everything carefully, so others can check it, and so they can trace what went wrong if it doesn't pan out. They're precise and methodical, so sloppiness doesn't introduce unnecessary noise that can hide errors and problems. They pay close attention to anomalies and unexpected results. And if they make a prediction based on a hypothesis, and observation contradicts it, they take it very seriously.

It might be nothing, and easily explained away, but the availability of easy explanations may mean the prediction is not falsifiable, which is itself a problem. Especially if you only noticed that fact after it was contradicted. Bias: it's everywhere.

January 4, 2013 | Unregistered CommenterNiV

"Gaps between prediction and reality are not evidence of a deficiency in method. They are just evidence--information that is reprocessed as part of the method of generating increasingly precise and accurate probabilistic estimates."

If one does not address the range necessary to have faith in a model, one is not doing good science. By the above, all models, no matter how far from reality, could be considered “good”.


Let’s look at the statement from NOAA that was brought up in:

Wisdom from Silver’s Signal & Noise, part 2: Climate change & the political perils of forecasting maturation

“..NOAA: Special Supplement to the Bulletin of the American Meteorological Society
Vol. 90, No. 8, August 2009
Pg 24 “..Near-zero and even negative trends are common for intervals of a decade or less in the simulations, due to the model’s internal climate variability. The simulations rule out (at the 95% level) zero trends for intervals of 15 yr or more, suggesting that an observed absence of warming of this duration is needed to create a discrepancy with the expected present-day warming rate…”….”

NOAA makes the case that if the model falls outside of expected values at 95%, the model would be in discrepancy.

If the model is in discrepancy, then the assumptions the model is based on are incorrect. If the model is found to be incorrect, one needs to create a new working hypothesis and state what would falsify this hypothesis. The entire basis of science is being able to falsify as it is not possible to fully prove a positive.

January 4, 2013 | Unregistered CommenterEd Forbes

MW -

The Results section of the paper's abstract says the summary hazard ratio for grade 1 obesity was .95 (95% CI, 0.88-1.01) relative to normal weight, which suggests lower risk, although I haven't read the study carefully.

I think that there is a reason why the authors are so careful to (repeatedly) word their conclusions the way that they do (that mild obesity is not associated with higher mortality).

January 5, 2013 | Unregistered CommenterJoshua

@Ed: Yes, you certainly deserve a good answer. I asked Larry to help me with this w/ you specifically in mind. He did a bit. I also tried to help myself. Some of the points were along the lines NiV suggests. Basically, though, I think there is a difference between testing a hypothesis about how the world works ("atmospheric CO2 concentrations will affect temperature") & testing a model that forecasts how, given the world works in a particular way, things will be in the future conditional on (a), (b) & (c). Being "proven wrong" is taken for granted by the latter (model testing); one welcomes the experience, b/c it generates information used to "update" the model w/ the aim of improving it. *Bad modeling* doesn't calibrate; just treats its power to "retrodict" as license to engage in dogmatic claims about what to do in order accomplish whatever (that's economics in Silver's view). Good modeling predicts, obeserves, calibrates based on "how wrong." The unsatisfying part of this, in my view thinking of how to address your point about "wrong predictions," is that at some point models that miss the mark pass over a line, one that divides Bayesian-evolutionary-business-as-usual & "hey, we should re-examine how we think the world works!" That's what I want help with b/c it is clearly a question that my 2 sentences beg & that I wish I had a 2-sentence or even 2 -paragraph answer for. (Larry thinks the line is the one at the edge of a Kuhnian paradigm shift; I think the sort of frustration/embarrassment I'm describing happens often in "normal science" times, too)

@NiV: if you agree with me, could you pls revise the two sentences that you & Ed don't like? (you might want to go look at the exchange between me & Larry, if you haven't already)

January 5, 2013 | Registered CommenterDan Kahan

Dan,

As I said earlier, it depends on what "method" is being referred to. The statement you made is entirely unproblematic when applied to hypothesis testing and model development. You make proposals, test them, some of them will fail, which will allow you to propose better hypotheses and models.

The issue arises from the context, in which it is being applied here to what are being presented as calibrated and validated models suitable for policymaking. Climate models have not yet been shown to be capable of making accurate long-term predictions of climate. In the context of validated models, an observation falling outside the predicted range is a problem.

If you want a succinct modification to your two sentences that would satisfy me, I would insert something like the phrase "...regarding unvalidated models not yet suitable for policymaking..." somewhere in them. I obviously can't speak for Ed or Larry on this.

---

"Basically, though, I think there is a difference between testing a hypothesis about how the world works ("atmospheric CO2 concentrations will affect temperature") & testing a model that forecasts how, given the world works in a particular way, things will be in the future conditional on (a), (b) & (c)."

OK, there are a number of issues to consider. Almost all serious climate sceptics would agree that CO2 concentrations will affect temperature. However, there are a significant number of other factors that will affect temperature too. Some of these are known and well-quantified, some are known but poorly quantified, and it is almost certain that there are many others that are still unknown. The climate system is incredibly complicated.

The factors affecting temperature are commonly split into "forcings" which are external inputs directly influencing temperature, and "feedbacks", which are weather parameters that change depending on the temperature as well as affecting the temperature. So you have to first add up all the forcings, and then multiply the result by a number related to the total of the feedbacks.

This is further complicated by the fact that even this nice linear forcing/feedback model is a simplification. There are time lags due to heat transfers in and out of the deep ocean, different constants applying at different timescales, some of the constants probably not being constant, but depending on other inputs, feedbacks responding with different strengths to different sorts of contributors, and the possibility of what is called "internal variability", in which random chaotic weather affecting such factors as average cloudiness could drive significant variability over longer timescales without any changes in external inputs. The week-to-week weather is acknowledged to be effectively random - there's no law of physics says it has to average out over specific longer periods.

So the hypothesis is that doubling CO2 on its own adds about 1.1 C to the average temperature, and that the combined feedbacks roughly triple this to 3.5 C. (i.e. 2/3rds of the modelled warming is not the direct effect of CO2 at all, but the feedbacks.) Since we have so far seen about a 40% increase in CO2 which is half a doubling, we ought to expect about half that amount of warming, which would be 1.5 C, except that we have only observed about 0.6 C, the discrepancy being explained by additional negative forcings from (unmeasurable) inputs other than CO2 that are cancelling some of the warming out.

Clearly, this is a much more complicated and difficult hypothesis. Some input variables we can't measure. Some input variables we probably don't even know about. And some of the more definite predictions made by the models turn out to be wrong, contradicted by observation. One of the most notable being that the biggest feedback in the models is increased water vapour. Warmer weather evaporates more water, and water vapour is a greenhouse gas, causing more warming. The prediction is that more water vapour should decrease the lapse rate in the tropics, leading to additional warming high up in the tropical troposphere - the so-called "tropical hot-spot". It isn't there. And according to some observations, long-term water vapour hasn't increased. That's not the only problem, either.

So you see, it's not simply a matter of whether CO2 affects temperature, it's whether it shows up against the background noise of all the other things that affect temperature. That's a very hard question, and one I do not believe science has an answer to yet. Climate models are very useful tools for aiding understanding, but are not yet at the point where they can be relied upon for prediction. They are grossly underspecified and poorly constrained, in a domain where spurious correlations abound. Some aspects are known to be wrong; the question being whether the rest is "good enough". Given the uncertainties about the basics, tuning the models is no more than an exercise in curve-fitting, in my view.

I know you didn't want to turn this blog into part of the climate debate, and I'll cheerfully desist if you like from this point on, however I felt we would be talking somewhat across purposes if you think the question is simply whether CO2 affects temperature. This is only just scratching the surface, but I thought it might help you understand our position (and why your statements were perceived as a problem) if you knew a little more of what the argument was really about.

January 5, 2013 | Unregistered CommenterNiV

Dan,

I just thought of an easier way to try to explain a part of the issue.

If you take any fixed output function and a list of sufficiently many random, unrelated input functions, you can approximate the output function with some linear or non-linear combination of the input functions.

As you gather more data and extend all the functions, you find that the model diverges, and fails to predict the new observations. But you can tweak the coefficients, add a few extra input variables, and get it to fit again. Information is reprocessed as part of a method aimed at generating increasingly precise and accurate probabilistic estimates. The problem is, they don't converge. As you proceed, the predictions keep failing, you keep tweaking the model, and the prediction intervals you report either don't work or don't shrink. The model never gets validated, but never gets rejected, either. It remains constantly in development; in a state of flux.

At what point do you give up, and why do you do so? Is such an approach falsifiable?

And under what circumstances would you recommend trusting a model for setting high-stakes policy? Would you trust an unvalidated model like this, if it was the best thing available?

January 6, 2013 | Unregistered CommenterNiV

Dan

The UK MET has released a new temp prediction base on their model. The below link shows previous predictions.

http://somethingtododownthepub.com/

The above link has 4 MET projections from 2007, 2009, 2011, and the latest that was just released by the MET ( also at http://www.metoffice.gov.uk/research/climate/seasonal-to-decadal/long-range/decadal-fc )

None of the prior 3 show any predictive power worthy of the name, and all run hot. For the latest model, it does seem that the MET has moved to projecting a drop in temp over the next 10 years.

How much "fiddling" does one do to a model that is consistently wrong before one challenges the basic underlying assumption of the models that there are major positive feedback's to an increase in CO2? Without feed-backs, the models can not show that an increase in atmo CO2 levels cause anything other than a very minor increase in temp.

I think this goes to the heart of the topic of your blog post. There has to be a point when one has to challenge the basic assumptions of a model it the model is consistently wrong.

January 6, 2013 | Unregistered CommenterEd Forbes

I remember a simpler time when scientists used to say,

"If your prediction is wrong, your hypothesis is wrong." (Feynman.)

What happened?

January 7, 2013 | Unregistered CommenterBrad Keyes

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>