The recent Military Operations Research Society (MORS) special meeting on professional gaming set itself the following task:
The meeting will produce initial content for a Professional Gaming Practitioner’s Handbook and bring together members of the community of practice to consider best practices, design, existing applications and appropriate analytic methodologies in an effort to codify the fundamentals of game design and analysis. The meeting is designed for information exchange and participant exposure to professional practice.
The main conference started off on Monday with several plenary presentations.
George Akst (Senior Analyst, Marine Corps Combat Development Command) highlighted the value of wargaming as a midpoint between large exercises and operations research analysis. Wargames are, he suggested, are generally a single/deterministic (n=1) approach to a stochastic problem, illuminating one possible plausible scenario. He pointed to the value of wargaming in identifying capability gaps, developing doctrine, and experiential training and learning. They can also help narrow the scope of problems for subsequent (OR) analysis. He also noted weaknesses in many wargames: analytical follow-through, adjudication rules and procedures. Operations analysis can address some of these shortcomings by bringing additional analytical rigour, including sensitivity analysis. Analysts should be integrated into the process early to make the combination of wargaming and OR analysis most effective. I thought it was a useful presentation on the strengths and weaknesses of analytic gaming, although I would prefer to see as gaming as a tool in the analytical toolset, rather than something performed by the gaming tribe to which OR analysts must somehow relate.
Robbin Beall (Head, Campaign Analysis and Modeling at Assessment Division, Office of the Deputy Chief of Naval Operations) stressed the growing appreciation of wargaming as a tool to help identify innovative approaches (and to weed out those that may be less useful). She also identified three areas of concern:
- More attention need to be devoted to innovation in gaming and game design. Designers need to be smarter, effectively pinning players into the puzzle they are expected to address. A cadre of good game designers needs to be fostered within DoD. This year’s US Navy Title X game is an example of greater innovation, with a traditionally large and monolithic game being broken into a series of smaller, more highly focused games.
- Adjudication remains a challenge. If a game has weak adjudication, the game fails and participants leave with a wrong impression.
- Despite the current renewed emphasis on wargaming, it needs to be remembered that wargaming is only one tool in the analytical toolbox. Games are not always the best approach.
E.B. Vandiver III, former Director of the Center for Army Analysis, delivered the keynote address. He focused many of his comments on CAA experience and some of the problems of wargaming. He summarized some of the most common objections to wargaming: it is too subjective, too qualitative, they aren’t repeatable, they learning effects overwhelm functional effects, and they are too time-consuming and resource intensive.
He also discussed the development of a training wargame at CAA to train junior analysts with little or no background in military history or the military decision-making process. The initial version was too complex, so they designed a new, simpler, faster, and more strategic game—but never ran it, because they were tasked to develop a front end analysis for a new Korea operations plan (building in part on the prior game development).
He also discussed using games in 2006-07 to analyze the security force requirements of the Iraqi government in the context of the US withdrawal from that country. The questions asked required specific answers: force size, deployment, and so forth. A computer-assisted, open player game was developed, based on research, COIN doctrine, data on violent acts, and assumptions vetted by the sponsor. The processes highlighted the value of participation from the actual theatre, that there was a need to address speed, efficiency, and errors in the game process; and that the game really required a precursor training game. The game was later refined and used for drawdown risk assessment and a range of other questions. It was even modified top examine Afghan drawdown risks.
Overall he suggested that many or most of the shortcomings of gaming could be mitigated.
Given the abject failures of the Iraqi security forces in recent years, it would have been useful to more fully discuss why the games Vandiver described fell so short of anticipating these shortcomings. I would argue that the poor performance of the ISF has more to do with leadership, patronage, corruption, morale, sectarian polarization, and internal politics and similar social and political “intangibles” than it has with capabilities, deployment, or formal organization. I asked Vandiver about this, and he responded that the games had not explored political context and effects, or even force motivation and morale. To my mind, that’s rather like wargaming without physical terrain or weather or visibility effects, and underscores Beall’s point about the need for more innovative approaches to gaming. (As one colleague noted in a side-comment, hobby wargame rules have long addressed morale issues, so it isn’t exactly an impossible challenge.)
Mark Gallagher (Studies and Analyses, Assessments and Lessons Learned/A9, USAF) emphasized the human-in-the-loop, adversarial character of wargaming, and suggested it ought to be seen as part of analytics. He distinguished it from military exercises. He argued that the “single output” nature of wargaming didn’t necessary limit its usefulness. The particular decision points in games can be examined, for example.
Bill Lademan (Director, Wargaming Division, Marine Corps Warfighting Laboratory) made a very lively presentation in which highlighted in the ways “next generation” wargaming could be done better. He noted that the Marine Corps will be developing a state-of-the art wargaming center (adjacent to their intel facilities, to enable access to TS/SCI SAP material). He stressed the need to “wargame at the speed of thought,” rather than being handcuffed by developing powerpoint presentations. Knowledge needs to be “maneuvered, not managed,” in a way that effectively addresses problems. He emphasized human intellectual input into wargaming, and was critical of relying on computers. Analysis, he suggested, was all about the decomposition of a problem and striving for precision. Wargaming, he argued, was about many variables, and what emerges from their dynamic interaction with player decision-making: “what comes out the soup” of a complex situation.
Addressing the purposes of the MORS special meeting, Lademan expressed the view that wargaming should not be preoccupied with achieving rigour and repeatability, in which the manipulation of data becomes a substitute for wisdom.
He pointed at length to the Vietnam War, where “analysis replaced strategy” and failed to appreciate that the war was not about kill ratios but rather about winning over people. Later, the bodycount became the analytical measure of success. The disagreed that wargaming ‘was an analytical methodology,” arguing it was an assessment methodology. Analysis, he argued, was about exact conclusions, while wargaming (and assessment) is about a judgment. “Assessment is to analysis the way philosophy is to science,” he suggested. He disagreed that wargaming struggled for acceptance. Wargaming, he argued “isn’t broken”—and efforts to fix it with greater supposed analytical rigour would be counterproductive, since it could squeeze out creativity. Because of human learning and cognition, wargames are never truly repeatable. They’ll also never be rigorous, and models will never be adequate.
It was a powerful and stimulating presentation, and a credit to the bluntness for which the Marine Corps is known. It was also something of a riposte to the earlier OR-focused comments by Akst and Beall. In a dialectic sense I suppose the clash of views of useful. Or, perhaps, Lademan’s comments only reinforced the view in some of the OR community that gaming is a sort of nebulous witchcraft claiming insight into the free will and the human soul. Towards the end of his comments he offered more conciliatory comments, calling for a productive union of wargaming and analysis where each offset the weaknesses of the other.
I enjoyed it a great deal, but I’m concerned by the absolutism of some of it. Analysis and gaming, I think, have very fuzzy boundaries. Drawing sharp dichotomies between quantitative and qualitative analysis is counterproductive. Sharpening tribal divisions around these issues only contributes to hedgehogism.
Wargaming for analysis within DoD (from the back of the room).
After lunch an OSD panel discussion examined “wargaming for analysis within DoD.” The session was chaired by Mike Ottenberg, and featured the quad chairs for the current initiative on wargaming and innovation: COL Mark Gorak (OSD CAPE), Jacob Heim (OSD Policy), COL Neil Fitzpatrick (JS J8 SAGD), and CDR Phil Pournelle (ONA). Among the many points that were made:
- Small games are often more useful than large, complex ones for some purposes. (Phil)
- “Blackbox” modeling and simulation can alienate players from a game, as they are unable to understand cause-effect connections. (Phil)
- The professional military education process needs to develop wargaming experience and skills. (Phil)
- The style and design of a game needs to be selected according to the problem to be addressed. (Phil)
- In the past, the senior leadership hasn’t necessarily seen their questions and priorities being addressed by wargaming. (Mark)
- Given the natural biases of the services and the combatant commands, how can gaming help to develop DoD-wide perspective and priorities? (Mark)
- Wargaming needs to explore left and right boundaries of issues. (Mark)
- When inputting data into the new (OSD CAPE-managed) wargame repository, please put greater thought into significant key insights generated. (Mark)
- SAGD is high demand for POL-MIL games, for clients such as the NSC. (Neil)
- The next DoD-wide wargaming summit is scheduled for early November.
- There appears to be less capacity within D0D for multi-move, adjudicated games with an active Red side. (Jake)
- The DEPSECDEF is especially interested in games that offer insight into key challenges as well as programmatic issues. (Jake)
- How wargaming initiatives by the services will be integrated with DoD-wide efforts? (Mike) The first step is dialogue, and trying to understand what is going on out there, and who is doing what. The DoD wargame repository is one important element of this. (Neil) Most current games address the “big five” potential adversaries—it is important to examine how to develop insights and synergies across those wargames. While resisting the temptation to impose a single set of metrics to assess the utility of games, there is a need to think about how to differentiate a good game from a bad one. (Mark)
- How should best practices be recognized, taught, and promulgated within DoD? (Mike) We don’t want to be in a situation where a formal standard is established and wargamers are “certified.” Instead we need to invest in analysts and PME. (Phil) We also need to also educate sponsors. (Jake)
- One audience comment challenged the anti-quantitative thrust of Lademan’s earlier comments. Phil warned about the false precisions of much M&S work.
Following these plenary sessions, we broke into several working groups:
My hotel roommate (and former fellow UVic wargaming club alumnus) Brian Train and I served as cochairs/facilitators for WG8, along with Joe Saur (Georgia Tech Research Institute), Eric Greenburg (JHUAPL) and Clyde Smithson (JHUAPL). This was primarily intended as a continuation of yesterday’s training course intended to further develop wargame design skills by actually designing a wargame.
Working Group 8 at work.
On Wednesday we started the day with another plenary address by Robbin Beall, this time on “challenges in design and execution of wargames.” She identified several of these:
- insufficient initial research
- lack of situation awareness of similar or complimentary supporting analytic efforts
- lack of authoritative data on capabilities
- flawed game design
- overly broad scope leading to shallow conclusions
- lack of innovation
- failure to focus players
- unsound basis for adjudication
- linked to shortcomings in initial research
- conclusions reached on anecdotal evidence
- lack of qualified subject matter experts
- overly simplistic wargaming tools
- team/player issues
- player fear of failure
- game too short for team cohesion
- one or two strong personalities dominate game play
- no objective post-game evaluation of game effectiveness
I thought her points were excellent. Her final point is a particularly important one: far too much of the serious games community—wargamers included—are far too willing to assess the value of the game based on anecdotes, game enjoyment, or the designers’ own confirmatory self-evaluation.
She also discussed the synergies between wargaming and quantitative analysis. Game objectives, she suggested, should be set to utilize the particular strengths of wargaming. Game designs should focus players on a limited number of dilemmas unresolved by previous analysis. Game execution should inform players with what is known from previous analytic efforts. Game adjudication should mine what goes on in the analysis world to support adjudication decisions. Finally game lessons should be determined and new ideas should be incorporated into quantitative analysis.
The implicit thrust of some of her comments seemed to be to frame wargames as adjunct to quantitative operations research, addressing those areas that quantitative analysis could not easily answer. I would tend to view things rather more broadly, arguing for qualitative research as a more equal partner, and also suggesting that findings are most robust when they are triangulated by variety of methods. Indeed, much of the discussion at MORS has struck me as akin to the quantitative vs qualitative skirmishes that afflicted political science a decade ago, but which in that case have now largely been superseded by widespread appreciation of mixed and plural methods.
Interestingly, the one wargame she did praise—a simple, apparently largely conceptual game that modeled a basic guns/butter or kinetic/nonkinetic tradeoff—doesn’t appear to have been particularly rigorous by OR standards or anchored in research and data, but rather intended as a spur to discussion and reflection.
Much of the rest of the day was spend with the working groups. In WG8 the participants divided into two groups:
- One group (Marcus Tregenza, Shawn Zackey, Christophe McCray, and Bob Turner, aided by WG cochair Eric Greenburg) set about designing a naval platform and technology acquisition game.
- The other (Stacie Pettyjohn, David Maxwell, John Montonye, Stephen Mackey, aided by Brian Train and I) decided to develop a game that would explore how ISIL balances its strategic options, and how it might respond to coalition efforts in Iraq and elsewhere.
In the afternoon there was also a games expo. PAXsims had a display booth, featuring AFTERSHOCK, ISIS Crisis, and various other simulations I’ve run at McGill and elsewhere.
The MORS game expo.
AFTERSHOCK, with Brian Train in the background displaying his many game creations.
I also had an extensive discussion with Stephen Ho (Dstl) about the potential impact of llamas on modern special forces operations. This is clearly an area ripe for exploration through good game design.
Special forces combat llamas, one of the great military topics yet to be gamed.
The final afternoon we all reassembled to hear brief-backs from each of the working groups.
- WG2 (wargame objectives). The group emphasized the importance of initial problem identification and research design. They also addressed educational wargame objectives. There was substantial discussion of the relationship with game sponsors, and what to do about sponsors who don’t know what they want, who want a game to address a non-gameable problem, or who are inclined to micro-manage. A long list of “wargame pathologies” were identified.
- WG3/7 (game design, development, and execution). They reviewed the typical game design process: concept development; research; identification of game elements; building components, prototyping, and initial rules concepts; playtesting; finalizing components… and only after all this, the game itself. Feedback and critical evaluation needs to be continuous throughout all steps of the process. They also noted that analysis must be persistent through the design process, and that execution considerations need to be incorporated into the design and development process.
- WG4 (data collection, analysis, and tools). The group ran through a revised and abbreviated version of the ZEFRA wargame, which was used to spur a discussion of data collection and analysis issues. Among other issues, they argued that the analysis team needed to specify player requirements (qualifications and other characteristics); the value of a GICOD (“good idea cut-off date”); to consider constraints, limitations, and assumptions; and the need to regularly review collection during the event. The game design and game analysis teams should be an integrated part of an overall project team, rather than entirely separate. The Data Collection and Management Plan (DCMP) should specify what data will be collected on each issue and sub-issue of interest; where in the scenario the data might be generated; what methods and tools will be used; and when during the game such collection needs to occur. If data looks wonky, corrective action should be taken sooner rather than later. A list of tools that can assist in data collection and analysis can be found here.
- WG 5 (adjudication). The group reiterated Robbin Beall’s point that adjudicators should “do no harm.” Adjudicators need to be facilitators too, need to communicate with players to reduce frustration. Adjudication issues are often intimately tied to game design. Adjudicators need to be well trained and prepared, aware of the dangers of overtasking. Players must feel their choices make a difference. Overall the WG suggested it was hard to identify universal best practices.
- WG 6 (aligning games with other studies). This group explored how to best integrate various decision analysis methods with wargames, especially in the context of various “wicked problems.”
As for our own WG 8 (quick turn-around game design) presentation, Bob Leonhart presented some overall impressions from our collective game design efforts. He noted how time pressures sparked considerable energy and enthusiasm. He also underscored how much one learns about a topic from designing, and not simply, a game. The two game design groups then presented their games.
In Buying Victory: Budget Battle Wargame the Red and Blue sides invest in hulls and various (surface, subsurface, and air) technologies. Mature technologies are cheaper and safer investments, while potentially more effective future technologies involve more time, risk, and resources. The combat power of the Red and Blue fleets then confront each other three times during the game to determine the overall winner.
Buying Victory: Budget Battle Wargame
Countering ISIL: The Board Game pitted ISIL against a US-led coalition. ISIL has six possible lines of kinetic operation:
- In the Anbar/Fallujah/Ramadi area, against the Iraq security forces
- In the Baji/Takrit/Samarra area, also against the Iraqi security forces
- In the Irbil/Kirkuk area, against the (Iraqi) Kurdish Regional Government
- In northeastern Syria/Kobane/Hassakeh, against the Syrian Kurds
- In Aleppo province, against various rival Syrian armed groups
- Around Damascus, against the Syrian Army.
It also has two possible non-kinetic lines of operation:
- Building governance, which generates resources and helps to respond to potential governance challenges in the events deck.
- Building prestige, which helps attract recruits and facilitates international terrorism.
The design was partly inspired by the solitaire States of Siege games by Victory Points Games, although in our case the system is adversarial and provides players with a far more complex set of options and constraints.
Players start the game by selecting several game objectives from a list of possible options. Subsequent game play is very straightforward, consisting of four sequential phases:
- Event phase, in which an event card is drawn. This may present the players with challenges or opportunities, or otherwise affect game play.
- Resource phase. ISIL gain recruits and resources from control of territory, effective governance of the “caliphate,” and prestige which attracts supporters and donations. The coalition gains material resources at a steady rate, but political capital is only slowly replaced.
- Card selection phase. The players select five cards to play from a large deck of possible options. Most have a cost associated with their play: resources and/or recruits for ISIL cards; resources and/or political capital for the coalition.
- Card play phase. The players take turns playing their cards, each of which has an effect (and possibly an associated die score to succeed). Some cards may be played to augment the effects of other cards, or complicate those of an opponent. Still others represent key decisions or investments in major initiatives, which are prerequisites to the play of other cards in the future. When both players have run out of cards or choose to pass, the next turn begins.
We took a video of one turn of game play, which you’ll find below. Overall I thought it was an excellent design, and if time allows I may put some further work into developing it, in conjunction with other members of the team.
The MORS special meeting on professional gaming ended with thoughts from the synthesis group, members of which had floated from WG to WG during the meeting. With regard to WG8, the synthesis group (and in particular, Richard Phares, who had been the primary synthesis group spy in our midst) quite rightly pointed to the differences between wargame design for serious purpose versus wargaming for fun. They also noted that game designs are living things that can evolve over time, the key linkage between research and game mechanics, and the critical value of repeated playtesting.
Overall I thought it was an excellent conference. While I regretted missing out on the discussions in the various working groups, I very much enjoyed the design work in WG8, and certainly benefitted from the excellent plenary sessions. MORS—and even more so chief organizer Scott Simpkins (JHUAPL)—did outstanding work. I look forward to the wargaming handbook that should eventually emerge from this effort.
PAXsims 