Formal risk assessments and nuclear arms control: exploring the value of modern methodologies

This fault tree is incomplete (a realistic fault tree for this problem would have many more branches), and we do not claim that it is definitive or accurate. (Another team might draw a significantly different tree with other event descriptions.) However, a concrete example makes the advantages and disadvantages of using risk assessment methods clearer and easier to understand.

To illustrate the interpretation of this tree, the top level of the tree (an OR gate) postulates three ways by which use of nuclear-armed HBGVs could arise: human or technological error; or deliberate use based on a miscalculation of the perceived threat; or intentional use in response to an actual threat. Here, the difference between error and miscalculation is essentially whether the nature of the event would be evident to an informed observer almost immediately (error), versus only in retrospect, with additional background or context (miscalculation). Each of these paths is then further decomposed into its contributing factors. For example, “error” could be either human error, or technological error. Technological errors in turn (as modeled using an AND gate) are shown as requiring both a false alarm and failure to detect that the alarm is false. Human error could, if desired, be further subdivided into various possible causes (fatigue, time pressure, drug or alcohol use, mental illness, etc.). However, that level of detail is suppressed here. Similarly, “miscalculation” is here modeled as requiring a situation of high tension (which could arise due to either strategic or tactical considerations), and a false belief that there was an imminent threat, and failure to refute that false belief (e.g., due to lack of communication). Finally, intentional use of nuclear-armed HBGVs could in turn be subdivided into various possible reasons for such use—for example, because of pressure to act (“use-it-or-lose-it”) or intentional escalation of a conventional (non-nuclear) conflict.

Note that the methods used for assigning numerical values to the various events in the fault tree (and the uncertainties about those values) may differ. For example, some could perhaps be estimated based on historical data (such as failure of detection), while others could be estimated only judgmentally (e.g., the likelihood that foreign governments have a false belief). However, using the subjectivist theory of probability, both could be estimated quantitatively (at least in principle).

If the probabilities of some events in the fault tree seem too difficult to estimate accurately, some could perhaps be ranked on a purely ordinal basis—that is to say, rating some as more likely than others. Even this process can make critical assumptions explicit and facilitate discussion and debate if differences of opinion emerge. For example, some analysts might argue that the availability of nuclear-armed hypersonic weapons would improve survivability of a country’s nuclear capability, thereby reducing the probability of preemptive attack in situations of high tension. Conversely, others might argue that the availability of nuclear-armed hypersonic weapons would make nuclear use more likely—for example, because of the greater maneuverability of such weapons (compared to intercontinental ballistic missiles), or if the use of non-nuclear hypersonic weapons on the battlefield reduces the psychological barriers to use of nuclear hypersonic weapons. Although use of risk-analysis models might not provide immediate answers to key questions, even this simple hypothetical fault tree shows that it can highlight those questions, help to differentiate among similar scenarios, encourage analysts to make their assumptions explicit, and help to identify areas of disagreement or need for further information.

Risk assessment: advantages and disadvantages

One advantage of risk assessment is obvious: The mere act of scenario-structuring can provide clarity. For example, even without quantifying the probabilities of the events in a fault tree or an event tree, the effort of constructing the tree can highlight events that had been previously overlooked or underestimated. Likewise, the thought process can help to identify relationships among events (for example, the key role of situations of high tension in miscalculations, in the hypothetical fault tree above). And for some applications, a full and complete risk assessment may not be needed; rather, the use of risk assessment “tools” to perform a partial analysis might be sufficient.

In a wide variety of fields, risk assessment is used as a basis for improved risk-reduction decisions. Even without well-defined robability estimates, scenarios afford the opportunity to rank-order risks. After the most important risks have been identified (based on likelihood, severity, or some combination of both), the risk assessment also supports identifying which aspects or elements of the scenario(s) are most important. After these dominant contributors have been identified, one can then review the various details involved in those scenarios to identify opportunities for risk reduction (for example, methods to improve communication between potential arms-control adversaries, or more reliable ways to detect threats).

In cases where risk reductions cannot be achieved unilaterally, risk assessment information would also allow negotiators to focus on mitigating or managing the highest-order risks, while possibly paying less attention to scenarios that (while theoretically possible) may not be important contributors to the overall level of risk. These types of insights, although relying on quantitative analyses, can often be summarized qualitatively, making the results more broadly accessible. Consequently, risk assessments provide a relatively reproducible method that can help to define leadership decision pathways, giving negotiators a range of options and illustrating ways to assess their outcomes. In addition to informing arms-control negotiators and their staff about the most important risks, risk assessments can also provide a communication platform for negotiators. For example, risk assessments could help both arms-control negotiators and their staffs to understand how best to represent the threats they perceive, or how particular measures might increase or decrease the likelihood of particular events in the risk assessment.

Alternatively, risk assessments could be used as input to further analysis. For example, risk assessment could estimate the likelihood of particular outcomes or actions in a game-theoretic analysis of competition or cooperation between nations or could help prioritize the most important uncertainties for future research or intelligence-gathering. To achieve these goals, it is important to keep risk assessments up to date as circumstances change, as so-called “living risk assessments” (Goble and Bier, 2013), “running estimates” in the military context (US Army, 2019), or simply ongoing situation assessment. Risk assessments may need to be updated for reasons ranging from the advent of novel technologies, to a change in leadership in an adversary nation, to transitory changes in circumstances (such as local skirmishes). The importance of maintaining a living risk assessment may be even greater in the context of arms-control negotiations than in other domains, since different parties may have changing understandings of the risks over time, and a once-agreed-to risk assessment may come to be viewed as obsolete over time.

There are, of course, also potential disadvantages of risk assessment. First, the logic and assumptions behind an engineering-style analysis may not always be straightforward and easily understood, especially for subject-matter experts who may be knowledgeable about the problem at hand but not familiar with risk methods. For example, in the hypothetical fault tree, assumptions about adversary actions appear in multiple parts of the fault tree (under both intentional use and miscalculation). It is arguable that greater understanding might result from putting actions and behavior by a particular group of nations into a single framework rather than having them scattered throughout the tree. In principle, however, these different types of analysis (e.g., engineering-style risk methods along with more traditional narrative methods) could perhaps be used in tandem, as in so-called “plural analysis” (Brown and Lindley, 1986).

Risk assessment can also lead to mistaken interpretations in the face of well-known decision maker biases. For example, in a phenomenon known as “probability neglect” (Sunstein, 2002), either extremely small probabilities can be overweighted (Kahneman and Tversky, 1979), or such low-probability events can be neglected as essentially impossible. People are also often insensitive to the magnitude of a bad outcome—for example, 1,000 deaths may be viewed as almost as bad as 100,000 deaths (Slovic et al., 2013, on “psychic numbing”). Moreover, risk assessments themselves can have biases. This problem is perhaps more extreme for open systems such as arms control, where most actions involve highly uncertain human behavior, and probabilities cannot be easily estimated based on past failure data or component testing.

In addition, especially on arms-control issues, the scenario approach to risk assessment can generate enormous numbers of possible pathways. Overly numerous or detailed scenarios can generate detailed insights, but could also distract from recognizing important overarching qualitative aspects of the problem. Such elaborate analyses can also be difficult to use as a basis for decisions in real time (as in a crisis situation, or while actively engaged in arms-control negotiations), suggesting that such analyses are perhaps better done offline, far in advance of when the resulting insights or recommendations are likely to be needed. They can then be boiled down to higher-level and less-complex risk assessments that are easier for decision makers to digest and use.

Another concern is that a decision maker might take the results or insights from a risk assessment more seriously than is merited (for example, if other sources of information and expertise, such as the insights from area studies, are not adequately considered in the risk assessment). It is well-known that risk assessments can cause people to fixate on the parts of the analysis that can be easily quantified (e.g., the probability of false alarms) rather than parts that are difficult to quantify (Brown, 2002), and to focus on the scenarios that are shown in the analysis rather than on those that were inadvertently omitted (Fischhoff et al., 1978). This is problematic in analyzing systems that are highly complex or open-ended, as with “non-engineered” systems where human actions play a key role, in which even a reasonably complete enumeration of important scenarios may be impractical or difficult to achieve (Perrow, 1984). For this reason, in situations of great complexity, Leveson (2012) argues against attempting to apply probabilistic risk assessment, and instead recommends both eliminating risks where possible and reducing complexity (making risks clearer and easier to understand and mitigate).

Care must also be taken if the risk assessment (e.g., fault tree) is shared with the adversary in arms-control negotiations, to ensure that important information or assumptions are not inadvertently revealed. In addition, the use of risk assessment could paradoxically serve to derail negotiation instead of supporting it. For example, disagreements about minor aspects of the risk assessment could distract attention from overarching goals, or could be deliberately exploited to delay negotiations.

Despite these potential shortcomings, including possible oversimplification or even misuse of results, the potential benefits of risk assessment justify attempting to develop and apply it to support the broader objective of advancing international security and, where useful, achieving arms-control agreements. It is also important to recognize that methods exist for mitigating some of the pitfalls just mentioned. For example, formal methods of scenario identification and structuring (e.g., Kaplan and Garrick, 1981; Kaplan et al., 2001) can reduce problems of incompleteness; plural analysis (Brown and Lindley, 1986) can ensure sufficient attention to scenarios that are difficult to quantify; formal methods of expert elicitation can improve the calibration of probability estimates, and reduce the pitfalls of groupthink (Janis, 1982); and precursor analysis can use data on “near misses” or other undesired events to help inform risk assessments. There is literature on how best to communicate risk results (including uncertainties) to decision-makers, potentially mitigating difficulties such as probability neglect (Bier, 2001; Solomon, 2022).

A path forward could include a risk assessment test case for nuclear arms control

There may be considerable value in undertaking detailed risk assessment of the issues related to nuclear arms control for two key reasons. First, the structural formality of these methods provides a way to make explicit the often-implicit assumptions of interlocutors. Second, risk methods provide a pre-analyzed framework into which new information can be integrated as it becomes available, and that framework can be useful in getting multiple team members (or even adversaries) thinking along consistent lines. Without arguing for replacing the more qualitative types of analyses that have been done or for basing decisions primarily on risk assessment, we do believe that structured, engineering-style risk assessments would complement the qualitative types of analyses that have been done in the past, yielding ways of viewing the problem that can supplement and inform the more intuitive understanding of decision makers. Under some circumstances, risk assessment could even be used in near real time (for example, to support arms-control negotiators), despite the complexities of a scenario-based approach. Once a detailed analysis has been completed, the results could be simplified or condensed in a manner that would make them useful and usable in such situations.

The hypothetical fault-tree analysis of HBGVs presented here is intended only to illustrate these potential benefits, falling short of the type of compelling “existence proof” that might persuade potential decision-makers or arms-control negotiators of the utility of a formal risk-assessment process. In addition to fault trees and event trees, numerous additional methods of risk assessment, both qualitative and quantitative, could be explored (and some have already been applied to problems of nuclear-arms control). Examples are pathway analysis, war gaming, red teaming, game theory, agent-based modeling, exchange analysis, stability analysis, and network analysis. The strengths and weaknesses of many of these are discussed in Risk Analysis Methods for Nuclear War and Nuclear Terrorism (National Academies of Sciences, Engineering, and Medicine 2023, ch. 6). All are ways of modeling or understanding the relationships between two or more nations or organizations. Causal methods recently applied in the fields of statistics and economics could also be beneficial in arms control—for example, by providing automated methods of compliance verification. Chaos theory and other methods of understanding complex systems could help to understand whether a given security or arms-control regime contributes to stability in arms-race dynamics.

A desirable step forward would be for one or more organizations (either governmental or non-governmental) to select a specific issue or problem (such as the impact of HBGVs on arms control) and sponsor a project to explore the applicability of risk-assessment methods and develop the sort of risk assessment suggested here. Such an analysis would be resource-intensive and take considerable time to develop and would perhaps best be done by a dedicated group of analysts, working closely with (or at least informed by) decision makers or past arms-control negotiators. Given recent and ongoing geopolitical changes at a level not seen in many decades, it may be time to apply modern techniques of risk assessment to nuclear arms control to identify new ideas, improve understanding, and provide useful insights–increasing the security of all parties to negotiations. (The advance of “big data” and artificial intelligence/machine learning may also help to support the types of analyses suggested here.) Such a study could serve not only as an “existence proof” of the efficacy of the approach (if successfully demonstrated), but also as a way to ensure that end-users (for example, arms-control policymakers and negotiators) are aware of the types of insights that could be provided by this type of analysis.