Conflict simulation, peacebuilding, and development

Wargaming and analysis: reflections on the 13th annual NATO OR&A conference


On Wednesday, War on the Rocks published a piece by Jon Compton on the obstacles on the road to better analytical wargaming. As Jon tends to be, he was blunt in his assessment:

As an advocate for, and practitioner of, analytical wargaming within the [US] Department of Defense, I’ve witnessed some good things emerge over the past few years, but also enough poor practices to reinforce, to no small extent, the criticisms of gaming made within the operations research community. Peter Perla’s 2016 call to improve wargaming, while widely read and commented on at the time, was mostly ignored by wargame practitioners among federally funded research and development centers, universities, and defense contractors, who, frankly, seem largely content to continue on with business as usual.

Several agencies within the Defense Department, particularly within the Office of the Secretary of Defense and the combatant commands have now seen the effectiveness and impact of a complete analytical process that incorporates wargames and are now beginning to consider how they might do the same thing. The notable exception to this interest has been among more traditional wargame practitioners in the wargame community. To date not a single federally funded research and development center, contractor, or educational institution that purports to provide a wargame service has shown the slightest interest in providing a complete analytical solution that incorporates wargames, nor have they shown interest in analytical ownership of the outcome. To my knowledge, none has even been interested enough to ask what the requirements are. Apparently there is enough demand elsewhere to keep the wargame community busy, and if the reports I’ve read generated from recent wargames are any indication, the BOGSAT is alive and well.

Coincidentally, the day the piece appeared was also the final day of the 13th annual NATO Operations Research & Analysis conference, held this year in Ottawa (full programme here), with a few of the PAXsims crew in attendance. What insight did the conference offer into the issue?


Ben Taylor (DRDC) at the start of the conference.

First of all, wargaming was very well represented. It formed one of five conference streams, the others being methodology, data-driven analysis, operations, and strategic decision-making. A total of nine presentations on wargaming were delivered:

  • Sue Collins (NATO ACT) gave an excellent presentation on NATO analytical wargaming (SAS-139). She started by making the argument for wargames, but also highlighted several serious challenges (such as rigour, cognitive overload, data collection, game analysis). She discussed where there might be value in automating the process. She also pointed to the value of more effective data visualization, as well as techniques for obtaining greater rigour. The Connections professional wargaming conferences got a prominent shout-out. She concluded with a request for participants to share their insights, issues, and concerns with the NATO SAS-139 panel.
  • Matthew Stevens (Lessons Learned Simulations and Training) presented on simulation-based humanitarian training. He noted that humanitarian organizations do not typically red team their operations, and thus make avoidable mistakes. Furthermore, military wargames tend to portray civilian actors poorly: they are often treated as injects or random events, and rarely does a game give any insight into their concerns and decision-making. He discussed a valuable approach to modelling civilian behaviour this based on 6 key steps: identify lessons to be learned; identifying relevant actors and stakeholders; identify the goals and motivations of those stakeholders; identify the kinds of decisions they take and actions they make; understand the systems and contexts that shape their behaviour and choices; and finally model those systems.
  • Dani Fenning (NATO ACT) addressed analytical wargaming in support of NATO military deterrence options, focusing on the Rising Bear matrix game. Sadly I had to miss some of this, since I was scheduled to run a game of AFTERSHOCK: A Humanitarian Crisis Game at a local high school.
  • Rudy Boonekamp (TNO) outlined how a crowdsourced game could be used to generate insight into radicalization processes, in the context of a larger analytical project on opponent modelling. Psychological research provides insights, which can then be studied through experimentation using gaming techniques. The Opponent Immersion Game is an online, narrative selection (“choose your own adventure”) game in which elements of the narrative can be experimentally manipulated to assess causal relationships. Players provide feedback on their perceptions through dialogue with non-player characters and similar mechanisms. Following his presentation, audience questions addressed how artwork/graphics choices might affect gameplay, participant rewards, ethics approval, and extending the methodology to examine attitudes of other actors. (Coauthors: Vladimir Hazeleger, Lucia Tealdi, Helma van den Berg, Bob van der Vecht.)
  • Håvard Fridheim (FFI)  addressed wargaming reachback support to the headquarters planning process. Military headquarters lack reliable, processed information due to limited ,forward-desployed OR&A capability. Moreover, HQs are limited wargaming expertise and capability. Is it possible to reach back to offsite wargaming? They provided several examples of how this might be done. They also outlined a number of challenges: wargaming must offer real benefits, there may be HQ skepticism regarding the method, and games must be developed and run in a timely fashion that synchs with the planning rhythm. Headquarters may be unwilling to give up some control over the process to external providers. Classification and communication issues need to be overcome. At this point, these are a set of ideas that could be pursued in the Norwegian context. Comments from the audience emphasized the importance of trust in making reachback processes work, with one commentator suggesting that it might be too much to fight the “value of wargaming” and “value of reachback” battles at the same time. Another audience member commented on the difficulties of a remote wargaming site having adequate situational awareness. An parallel point was made about the value of participating in a wargame, versus simply reading about the results. (Coauthor: Stein Malerud.)
  • Pilar Caamaño Sobrino (NATO STO DMRE) looked at combining qualitative and quantitative wargaming approaches to support NATO analysis. She suggested that the largely qualitative outputs of many wargaming leads to problems of inadequate rigour. The combined use of qualitative and quantitative games, coupled with modelling and simulation, improves the quality of data and findings. Qualitative, human-in-the-loop  games (matrix, seminar, red teaming) can be enriched with (M&S) visualization tools. Quantitative (software-inp-the-loop) wargames can be assisted with M&S tools that expand exploration of the problem space. She discussed application of this approach in the case of an A2/AD simulation study, for the development of deployment plans, and for autonomous counter-maritime IED systems. (Coauthors: Alberto Tremori, Lucia Gazzaneo, Wayne Buck.)
  • Abderrahmane Sokri (DRDC) addressed a possible analytical wargaming approach to cyber deterrence. He discussed the logic of deterrence in general, and then applied some of this to potentially gaming cyberattacks. Members of the audience suggested that his approach would be enhanced by a more nuanced representation of the cyber realm, as well as a more complex treatment of deterrence.
  • Koen van der Zwet talked about applying OR game theory analysis to military cooperation, with a focus on logistics This was really a multi-actor optimization question, however, rather than application of a wargaming tool. (Coauthor: Wouter Noordkamp)
  • Mike Larner (Dstl) spoke on “wargaming for strategic decision-makers in the UK”—but since that formed part of the strategic decisions making panels, I missed Mike’s presentation.

Finally, one of the conference keynote addresses was by Stephen Downes-Martin (Naval War College). Stephen emphasized the analytical and ethical responsibility of analysts and sponsors, as well as the possibilities for malpractice. He noted the effect—sometime adverse—that small group dynamics can have on wargames and analysis, especially when actors have preferred outcomes that they would like the game to validate. These are issues he has raised before, notably in his important “Three Witches of WargamingNaval War College Review (2014) article, as well as his February 2019 Connections North presentation.

The non-wargaming panels I attended were good too. I very much enjoyed the keynote by Peter Singer (New America Foundation) on the impact of new technologies. The keynot by Christine Fox (Johns Hopkins APL) was awesome by all accounts, but I missed that as I watched the students of Robert Borden High School deal with a simulated earthquake in Carana.

On the third day of the conference, participants had the option of taking part in a full-day multi-domain wargame, or participating in a wargaming best practices workshop.

The former was run by Altan Ozkil (Atilim University) and Levent Berke Çapli (NATO):

The wargame is played by two teams Red and Blue with four sub-groups, with approximately two persons each sub-group. The four sub-groups represent the Tactical (Battalion) Commander, Joint Staff, STRATCOM Office and Cyber Command. Finally, the city is a green/white cell which is a non-playable character reacting to the team’s activities. All of these groups need to coordinate their activities both before and during the conflict.

The workshop was delivered me, with assistance from Stephen Downes-Martin, who skillfully role-played a problematic boss or sponsor. You’ll find the slides for the wargaming workshop here. I also provided an annotated two page bibliography of useful readings on serious wargaming.



As to Jon’s point in War on the Rocks, he’s absolutely correct to emphasize that wargaming must form part of a broader, data-driven, methodologically-sound, and results-oriented analytical endeavour, rather than used as a stand-alone tool. Overall, I think both the workshop and the majority of the presentations at the NATO OR&A conference did this.



Several times in my presentation I highlighted the WWII work of the Western Approaches Tactical Unit, the (predominately female) Royal Navy analysis unit that undertook both all-sources/multi method analysis and games-based professional training. If a small group working with 1940s technology on the upper floor of the Exchange Flags building in Liverpool can get it right, surely we can too?

NATO ORA watu.jpg

One of the readings I recommended was Becca Wasser’s recent New York Times Magazine piece on women in professional wargaming. By my count, around 15% of the wargaming workshop participants were women, as were 30% of the conference wargaming presenters. That’s likely far better than it would have been a few years ago, but also indicative that there is a way to go.

Finally, thanks are due to the organizers. Everything ran very well—even the weather. It  is a shame there wasn’t an even larger contingent of Canadians present, since it was an excellent opportunity to network with NATO and other colleagues. It was also an opportunity to enlighten even more folks from the Department of National Defence and other government departments on the value of OR&A, including serious gaming.

The conference proceedings will eventually be made available through NATO STO. 

2 responses to “Wargaming and analysis: reflections on the 13th annual NATO OR&A conference

  1. RockyMountainNavy 13/10/2019 at 4:47 pm

    Sadly, I feel that NATO may be moving ahead of the US in the use of wargaming for analysis. Even after CONNECTIONS 19 I sense that the Europeans are far more embracing of “innovative” wargaming like Matrix Games and the like. I see that American wargaming is bifurcated between commercial publishers (emphasis on playability) versus “professional wargamers” with too many of the later stuck on hex-and-counter or veering towards Modeling & Simulation.

  2. sunray42 12/10/2019 at 5:24 pm

    Well done Rex and Ben, as you say a pity more of us from JWES could not attend, we were however running wargames and being careful to make sure we got the analysis bit done !

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: