The uncertainty in climate modeling

"I'm just a model whose intentions are good, Oh Lord, please don't let me be misunderstood," Nina Simone may as well have sung. Models are fundamentally necessary in all sciences. They exist to synthesize knowledge, to quantify the effects of counteracting forces, and most importantly, to make predictions–the evaluation of which is at the heart of the scientific method. If those predictions prove accurate, the models–and more importantly, their underlying assumptions–become more credible.

In an observational science such as the study of climate (or cosmology or evolution), we don't get to do controlled laboratory experiments. We only have one test subject, the planet Earth, and we are able to run only one uncontrolled experiment at a time. It would be unfortunate indeed if this experiment had to run its course before we could say anything about the likely results!

Climate models are amalgams of fundamental physics, approximations to well-known equations, and empirical estimates (known as parameterizations) of processes that either can't be resolved (because they happen on too small a physical scale) or that are only poorly constrained from data. Twenty or so climate groups around the world are now developing and running these models. Each group makes different assumptions about what physics to include and how to formulate their parameterizations. However, their models are all limited by similar computational constraints and have developed in the same modeling tradition. Thus while they are different, they are not independent in any strict statistical sense.

Collections of the data from the different groups, called multi-model ensembles, have some interesting properties. Most notably the average of all the models is frequently closer to the observations than any individual model. But does this mean that the average of all the model projections into the future is in fact the best projection? And does the variability in the model projections truly measure the uncertainty? These are unanswerable questions.

It may help to reframe the questions in the following way: Does agreement on a particular phenomenon across multiple models with various underlying assumptions affect your opinion on whether or not it is a robust projection? The answer, almost certainly, is yes. Such agreement implies that differences in the model inputs, including approach (e.g. a spectral or grid point model), parameterizations (e.g. different estimates for how moist convective plumes interact with their environment), and computer hardware did not materially affect the outcome, which is thus a reasonable reflection of the underlying physics.

Does such agreement "prove" that a given projection will indeed come to pass? No. There are two main reasons for that. One is related to the systematic errors that are known to exist in models. A good example is the consensus of chemistry models that projected a slow decline in stratospheric ozone levels in the 1980s, but did not predict the emergence of the Antarctic ozone hole because they all lacked the equations that describe the chemistry that occurs on the surface of ice crystals in cold polar vortex conditions–an "unknown unknown" of the time. Secondly, the assumed changes in forcings in the future may not transpire. For instance, concentrations of carbon dioxide are predominantly a function of economics, technology, and population growth, and are much harder to predict than climate more than a few years out.

Model agreements (or spreads) are therefore not equivalent to probability statements. Since we cannot hope to span the full range of possible models (including all possible parameterizations) or to assess the uncertainty of physics about which we so far have no knowledge, hope that any ensemble range can ever be used as a surrogate for a full probability density function of future climate is futile.

So how should one interpret future projections from climate models? I suggest a more heuristic approach. If models agree that something (global warming and subtropical drying for instance) is relatively robust, then it is a reasonable working hypothesis that this is a true consequence of our current understanding. If the models fail to agree (as they do in projecting the frequency of El Niño) then little confidence can be placed in their projections. Additionally, if there is good theoretical and observational backup for the robust projections, then I think it is worth acting under the assumption that they are likely to occur.

Yet demands from policy makers for scientific-looking probability distributions for regional climate changes are mounting, and while there are a number of ways to provide them, all, in my opinion, are equally unverifiable. Therefore, while it is seductive to attempt to corner our ignorance with the seeming certainty of 95-percent confidence intervals, the comfort it gives is likely to be an illusion. Climate modeling might be better seen as a Baedeker for the future, giving some insight into what might be found, rather than a precise itinerary.

It's exhilarating to see the fruits of climate research achieve such prominence in the media, political debate, and concerns of industrial and municipal stakeholders. As scientists, though, it's incumbent upon us not to mislead the lay audience by blurring the line between methodological investigation and end products ready for consumption.

I should begin by disclosing that as a former project scientist at the National Center for Atmospheric Research, I was tasked with thinking about how to combine data from different climate models into probabilistic projections of regional climate change. This notwithstanding, I wholeheartedly agree with Gavin that these kinds of probabilistic projections aren't appropriate for risk analysis and decision making under uncertainty and won't be for a long time.

The ideal scenario in climate modeling is a situation where all the models are equally good: They each account for all the physical and chemical processes we deem necessary to describe Earth's climate; they all perform satisfactorily down to the finest resolved scale (currently, we have more confidence in the behavior of these models at the continental scale than at the local scale); and we have enough observed data (from climate records, ice-core samples, volcanic eruptions, and other natural phenomena) to prove that their simulations are consistent with what the real experiment (i.e., climate on Earth) is showing us. At that point, we could take the models' projections at face value, weigh the results of every model equally, and use their range to bracket our uncertainty, at least under a given emissions scenario. I'd also be out of a job. Luckily for me, we're far from that point.

As a statistician, I attempt to make sense of multiple models with a probabilistic treatment, which weighs models differently based on their different performances. Say I have data on average precipitation for the last 30 years in the Southwest United States, as well as simulations from 20 different climate models of current and future precipitation in the same region, and I want to know what the expected change in precipitation will be at the end of this century under a specific emissions scenario. I can try to account for the fact that the different models have shown different skill in simulating current precipitation. I can try to formalize the idea that model consensus is generally more reliable than individual predictions. Ideally, I can factor in the similar basic design of all the models and look at other experiments carried out with simpler models or perturbed parameterization to gauge the reliability of their agreement when compared with alternative modeling choices.

Although challenging, and, at present, only at the stage of methodological inquiry, there's value in interpreting what multiple models produce by trying to assign them probabilities. The alternative is that people will use ensemble averages and ensemble standard deviation as a measure of uncertainty. These quantities are reassuring because they can be easily explained and understood. But when I compute a mean as my estimate and a standard deviation as its uncertainty, I'm assuming that each model is producing independent data, and I'm relying on the expectation that their errors will cancel each other out.

More complicated is a probability distribution function, which characterizes the range of possible outcomes, assigning relatively higher or lower probabilities to subintervals, and may distribute the probability asymmetrically within the range. Such analysis is also more informative, but should I go as far as offering the results to the water managers in Southern California so they can think about future needs and policies for water resource management?

I wouldn't offer my probabilistic projections as a definite estimate of the area's expected temperature or precipitation; not in the same way the National Oceanic and Atmospheric Administration could, until recently, provide the public with "climate normals" based on a large sample of observed records of a supposedly stationary climate. It would be deceiving to present these estimates as having the same reliability that's needed to make structural decisions about designing a floodplain, bridge, or dam.

I think, though, that there's valuable information in a "conditional probability distribution function" if that's the best representation of the uncertainty given the data we have nowadays. The caveat is that the range of uncertainty is likely to be larger than what we see in models, which typically use conservative parameterizations. Additionally, we don't know which emissions scenario will occur over the next decades, further widening the potential realm of possible climate outcomes.

Meanwhile, in the real world, I would echo Gavin: Many decisions can be made by looking at qualitative agreement, general tendencies, and model consensus without the need for quantitative description of the uncertainties. Temperatures are warming; heat waves will intensify; sea level is rising; and arid regions will dry further. So planning for worst-case scenarios is only prudent.

Helping vulnerable populations access aid centers in the case of extreme heat events, dissuading construction on coastlines, conserving water resources, and developing drought-resistant crops are adaptation measures we should pursue regardless of the exact magnitude of the changes in store for us.

I work at the model/reality interface and am happy to join you in discussing our relationship with our models. As in any relationship, it is important to distinguish planning what we will do with the models we actually know, from dreaming about what we could do if only we could meet a group of Claudia's "end of the rainbow" models. Seeking a perfect relationship that would bring decision-support bliss might hinder the development of our warts-and-all models, which, while improving, are in no way converging towards perfection. Our models provide our best vision of the future: Can we accept them for what they are and still develop a relationship with them that works?

Gavin notes that the average of many models is frequently "closer to the observations" than any individual model. While true, this doesn't suggest that such an average can support good decision making. Even when the model is perfect, averaging can be a bad idea.

Consider the parable of the three non-Floridian statisticians who cannot swim but desperately need to cross a river. Each has a forecast of the change in the river’s depth from one bank to the other; in each forecast, there's a different point at which the river is so deep that the statisticians will drown. According to the average of their forecasts, it’s safe to ford the river. Even though the average is closer to the actual depth profile of the river than any one of the forecasts, action based on that information is not safe–if they cross the river, they’ll drown. This suggests there is something fishy about the way we interpret "closer to the observations" to mean "best."

I am curious where the idea came from that imperfect computer models, each running on 2008-era hardware, could yield relevant probability forecasts. The desire for decision-relevant probabilities is strong, in part because they would ease the lives of decision makers and politicians. If we had objective probabilities, we could adopt a predict-optimize-relax approach to decision making: We predict the probability of each possible course of events; we optimize our goals while taking into account our tolerance to risk; we act based on our models' vision of potential futures; and then relax, believing ourselves to be "rational agents." Taking such an approach may help us make it through the night, but it's unlikely to achieve our long-term economic or environmental goals, as we are holding our models to a standard they cannot meet.

There are many textbook examples of the predict-optimize-relax approach, and subfields of mathematics dedicated entirely to it, but there are few real-world examples. Given information from past automobile accidents, auto insurance companies can accurately model losses from future accidents using only the age and gender of a large group of drivers. But there are few, if any, examples of purely model-based probability forecasts supporting good decision-making in problems like climate, where we can't learn from our mistakes. Even the successes of everyday engineering such as bridge and aircraft construction owe much to the use of safety factors, which engineers have discovered and refined over time by watching similar projects fail repeatedly. Given only one planet, this option is not available to us.

Today's climate science can support an assess-hedge-monitor approach to risk management in which we assess our vulnerability along plausible pathways through the future, hedge and reinforce weaknesses wherever viable, but also monitor and maintain the flexibility to revise policy and infrastructure as we learn more.

Climate science will be of greater relevance to policy and decision making once it dispenses with the adolescent pickup lines of "climate-proofing" and "rational action" and focuses instead on achievable aims of real value. To provide a number (even a probability) where science has only qualitative insight is to risk the credibility of science.

Can we shift the focus of climate science from "what will happen" towards "where is our vision of what will happen weakest," "how can we best improve it," and "where are we most vulnerable"? Can we clarify what information from our current models we expect will hold as our models improve, and what things we expect to change? Given the urgency of the problem, climate science will always be developing methodology in real time. How can we best clarify that fact to the consumers of climate science so that they can view advances in the science as a good thing, not as a broken promise, probabilistic or otherwise? Can we embrace the inconvenient ignorance of science as we face the inconvenient truths of climate change?

Policy makers and planners now accept the reality of man-made climate change and are turning their thoughts toward adaptation. Should they make their decisions now, or wait to see if better climate models can provide more precise information? The modeling community strives continuously to improve their models, but experience also teaches us not to hold our breath for quantum leaps in model performance, or in our understanding of how to constrain future projections with historical information. Modeling centers have worked incredibly hard on this during the past decade, yet the range of projections hasn't changed much.

There's no obvious reason to expect more than gradual progress, and besides, some planners will not have the luxury of waiting. For example, regional flood defenses and reservoir capacity may only have been designed to address risks consistent with the current climate. Developments in major infrastructure like this have to be planned long term and decided according to time frames which may be determined by other drivers of risk (e.g. urban development), as well as climate change. So what do Claudia Tebaldi's Southern California water managers do if their planning cycle dictates that they need to make a decision now?

Presumably, they act on the best information available. Some of that information, such as future emissions and land-use change, isn't determined by climate-system processes. But much of it is, including changes in the frequency and variance of precipitation, the proportion of dry days, and changes in river flow and runoff. For that part, some collaborating set of experts–in this case climate scientists, hydrologists, and statisticians–are best placed to provide the information.

Should those experts refuse the task (citing, for example, concern about their scientific reputations) and leave the water managers to assess the risks themselves, based on their less expert assessments of climate factors? No. The relevant experts should take responsibility for their part of the decision process, ideally offering a probability distribution, which describes the relative risk of different possible outcomes.

A common criticism of probability distributions is that because they can't be verified, they can't be trusted. Indeed, the first part of this statement is true, but the second doesn't follow because we can't think about probabilistic climate predictions in the same way we think about probabilistic weather forecasts. In principle, the latter can be tested over many historical forecast cycles and adjusted to be consistent with observed frequencies of past weather behavior (the "frequentist" interpretation of statistical forecasting). Climate probabilities, however, are essentially Bayesian: They represent the relative degree of belief in a family of possible outcomes, taking into account our understanding of physics, chemistry, observational evidence, and expert judgment.

The scientific credibility of a probability distribution function therefore depends not on having near-perfect, error-free models, but on how well the methodology used to sample the uncertainties and convert the model results into probabilities summarizes the risk of alternative outcomes. This is by no means an easy hurdle, but one I believe we are capable of overcoming.

Of course, the probabilities must be accompanied by sensitivity analyses that educate users not to interpret the probabilities to too high a degree of precision. But provided this is done, they can be an effective means of support for decisions that have to be made soon.

However, I agree with Lenny Smith that room should be made for changing forecasts in the future. Climate scientists and statisticians need to work closely with planners to assess, for example, the relative costs and benefits of committing to a long-term investment immediately, compared with an alternative approach of adapting in stages, given the hope that less uncertain information will emerge in the future.

Finally, it's important to note that planners already deal with uncertainty–the wide range of future emissions scenarios being a prime example. So even if a climate scientist did (wrongly) claim to be able to quantify the risk of exceeding some critical threshold to some excessive degree of precision, the planner would still be faced with a family of different answers, corresponding to different socioeconomic assumptions. For this reason, I find far-fetched the idea that a planner is going to rush off with a climate scientist's probability distribution and make an erroneous decision because they assumed they could trust some percentile of the distribution to its second decimal place.

This doesn't mean that we can relax. But given the importance of the problem, perhaps we can be less frightened of telling users what we believe–and perhaps we can credit them with the intelligence not to interpret our current beliefs as everlasting promises. The issues require careful communication, but hey, we can make space in our diaries for that.