Wargames as experiments: The Project on Nuclear Gaming’s SIGNAL framework

Editor’s note: A review of the online SIGNAL wargame is available here.

What can we learn from the Peloponnesian War that will help us deal with the intersection of cyber and nuclear conflict? What does World War I teach us about the role of nuclear-capable hypersonic weapons in warfare? The short answer to both questions is—probably not much. While extensive scholarship on nuclear warfare and strategy exists, empirical data on the impact of emerging technologies such as cyber weaponry, advanced intelligence, surveillance, and reconnaissance tools, and precision-guided munitions are lacking. Given the importance of policy-relevant research to equip decision-makers with tools to understand nuclear deterrence and reduce the likelihood of nuclear war, the scarcity of data available to analysts remains a critical gap.

Already, scholars are looking to non-traditional data sources to address this challenge. To further that cause, the Project on Nuclear Gaming (PoNG) has embarked on experimental wargaming as a scalable, data-generating process to explore human decision-making during conflict escalation in a controlled environment.

Wargames have historically been used for modeling and simulation of warfare to inform tactical, operational, and strategic decision-making and for education and training. Traditional seminar or matrix wargames, however, are limited in terms of their scientific utility. The same characteristics that make them useful for producing innovative insights and illuminating surprise tactics and strategies—their flexible structure, open-ended human adjudication, and relatively small and often “elite” participant pools—do not lend themselves to the generation of data that might be statistically analyzed.

PoNG’s experimental wargame, SIGNAL, is designed to complement the qualitative information derived from traditional wargaming methods through the creation of a virtual environment that will generate data to support academic scholarship and guide policy-relevant inquiry. While experimental wargaming is not a magic bullet, such an approach provides a new means to explore the complexities of the modern battlespace.

SIGNAL. The SIGNAL experimental wargame comes in two forms: a board game and an online game. Each is a three-player, turn-based strategy game designed to study decision-making concerning the use of conventional and nuclear weapons. The research objective is to explore how weapons capabilities shape player behavior and the resulting impacts on strategic stability and the potential for escalation in a crisis. The board game serves as both a source of experimental data and a testing ground for the ruleset and game dynamics that were subsequently built into the SIGNAL online platform.

In SIGNAL, players choose courses of action that a hypothetical nation might follow, using economic, military, and diplomatic means. Players may choose to cooperate with or compete against the other players in the virtual world to grow their infrastructure and gain resources while avoiding loss of home territory to conventional militarized invasions or nuclear war. The game proceeds in rounds, with players maneuvering their forces within a map-based environment.

In both the board and online game variants of SIGNAL, players work simultaneously, often negotiating agreements such as trade deals or military alliances, and “signal” to others by placing tokens on hexes (units of land or water). The signaling token is accompanied by the staging of cards (face down) that reflect actions like building cities, sending troops to neutral territories, or engaging in military operations, including the use of conventional or nuclear arms. There is a price associated with signaling; players pay for each card placed, regardless of whether the capability is exercised to reflect the “costly signaling” or “credible commitments” in the existing theoretical literature on deterrence. Players may then exercise their staged actions, but only at locations where signaling tokens were previously placed.

During this action phase, players take a series of turns, starting with a randomly chosen player to ensure that any given player does not have a constant first-mover advantage. After five rounds, players are ranked on a relative scale based on their infrastructure, resources, and territorial integrity to determine the winner—with a player’s cumulative performance captured on a leaderboard. Although win/loss statistics are recorded, it is the patterns of play in the game given specific experimental conditions that are useful for scientific analysis.

The online game follows the same ruleset and sequencing as the board game while allowing for automated collection of player decisions, demographic data, and chat feeds used in diplomacy. A key element of the online game is scalable data collection—expanding the wargaming participant pool from perhaps a dozen individuals in the traditional seminar-based format to thousands of online gamers. The game is specifically designed to allow for the flexible introduction of treatment variables—for example, different types of weapons capabilities, numbers of players, and maps—and for data analysis on the back-end. The online SIGNAL wargame just launched, and all are welcome to login and play.

The benchmarks. A major advantage of seminar-based wargames is the ability to observe how players make decisions while tracking the discussions taking place. They also provide an opportunity for players to investigate challenges and solutions generated from gameplay. To explore how experimental wargaming complements traditional approaches, PoNG also conducted a seminar wargame that allowed for unstructured, adjudicated play among teams of senior policy-makers. This game incorporated the resolution of actions taken by each team via a “white cell”—a control team that manages the progress of the game—while players confront the same strategic problems faced by players of the board and online versions of SIGNAL.

Additionally, a survey experiment offers a control method for PoNG’s experimental gaming approach, using existing survey tools often implemented in social science inquiry. In the surveys, respondents are asked a series of questions on the basis of scenarios that closely approximate the strategic decisions faced by players within the board and electronic variants of the SIGNAL wargame. This experiment allows the team to compare our results against the existing literature on questions concerning nuclear use and conflict escalation, and to assess the relative effect of the data-generating process on player decision-making.

The benefits. Experimental wargames offer several benefits when compared with existing approaches. Unlike traditional wargames, these new methods allow for increased reproducibility in wargaming in a systematically varied environment. This can help to bridge the “complexity-scarcity” gap, where scenarios are too complex to model from first principles and data from actual events are too sparse to draw conclusions. With repeatable games and enhanced control over the experimental conditions, researchers can isolate the variables that they hypothesize may be causal and examine the effects empirically.

In comparison to computer-based or game theoretic modeling, the data collected via these games includes human decision-making and can thus be analyzed to explore factors such as personality, cognition, and experience. Experimental wargames also offer an alternative to a number of challenges inherent in survey experiments. For example, the interactive and role-playing aspects of a game environment tend to engage and draw in subjects to the scenario, while a survey may not offer the same immersion. Further, as subjects learn the nuances of a game, their strategies and methods of play may change, enabling adaptive behavior that would be difficult to capture in a survey.

The challenges. While experimental wargaming approaches offer some significant advantages over traditional war games, there are a number of methodological challenges associated with their use. First, particular subject pools may introduce bias, based on their age, gender, education level, income, or other socio-demographic characteristics. As a result, game designers must assess the impact of these factors—including the military and domain expertise of the players, on outcomes compared against traditional war games involving experienced policy-makers—if they are to use the data for generalizable inference. Second, the determination of appropriate win conditions, player numbers, and design aesthetics represents a challenge for game designers as they strike the appropriate balance between “playability” and external validity. In simple terms, games that are too basic fail to capture the complexity associated with the variables of interest to researchers, while games that are too complex become impossible to play. Moreover, the game design and ruleset may influence the experimental outcomes. Ensuring scientifically sound data requires testing various laboratory effects, including the number of players in a game, the costs associated with various actions, win conditions, initial game conditions, and environment design, among others.

Being aware of these challenges, PoNG has employed a multi-method approach—including the use of board and online wargaming formats with seminar-based exercises and surveys as benchmarks. This choice reflects our view that experimental wargaming benefits from cross-method comparisons and transparency, in terms of game design and the data derived from the experiment. Given the nascent stage of the use of tailored experimental wargames as a means for scientific inquiry, we may well learn that there are significant challenges yet to be overcome. Conversely, we may learn that even simple experimental wargaming tools can yield powerful insights about conflict and warfare. In either case, we will learn a great deal.

The difficulty of addressing international security challenges without empirical data has led researchers to explore additional tools for tackling questions surrounding the impact of cyber warfare, nuclear deterrence, sub-conventional or gray zone warfare, and emerging technologies on conflict escalation. The decision calculus in today’s strategic environment may be affected not only by the introduction of new weapons technologies, but also by the integration and intersection of these capabilities. These changes in technology have contributed to uncertainty on the part of policy-makers concerning the effects—with some arguing that these developments have complicated nuclear deterrence and potentially make nuclear conflict more likely. Where data are scarce, experimental wargaming in general and PoNG’s SIGNAL framework in particular have the potential to provide an additional tool in the methodological toolkit for both policy-makers and academics as they seek to augment lessons of history with new approaches to thinking about future conflict.