PAXsims

Conflict simulation, peacebuilding, and development

Daily Archives: 14/09/2012

Vogt: A Methodology to Assess UrbanSim Scenarios

Brian Vogt has kindly allowed us to upload a copy of his recently-completed MSc thesis at the Naval Postgraduate School on “A Methodology to Assess UrbanSim Scenarios.” In it, he examines whether the choices and feedback mechanisms in the counter-insurgency software UrbanSim actually support its intended learning objectives:

Turn-based strategy games and simulations are vital tools for military education, training, and readiness. In an era of increasingly constrained resources and expanding demand for training solutions, the need for validated, effective solutions will increase. Appropriate performance feedback is an important component of any training solution. Current methods for designing and testing the performance feedback provided in turn-based simulation are limited to well-structured problems and do not adequately address ill-structured problems that better replicate problems facing military leaders in today’s complex operating environment. This thesis develops and explores new methods for assessing the feedback mechanisms of turn-based strategy games. Using UrbanSim, a game for training strategic approaches to COIN operations as an exemplar, this thesis developed and explored two unique methods for evaluating the reward structure of the UrbanSim scenarios. The first method evaluates different student strategies using a batch-run method. The second method uses a reinforcement-learning algorithm to explore the decision space. These scenario evaluation methodologies are shown to be able to provide insights about a game’s performance feedback mechanism that was not previously available. These methodologies can be used for formative evaluation during game scenario development. Additionally, these evaluation methodologies are generalizable to other training and education games that focus on ill-structured problems and decision-making at discrete intervals.

Brian offers some excellent insight into feedback mechanisms in military training games in general, as well as in the specific case of UrbanSim. He runs large numbers of iterations of possible strategies in the game, so as to assess whether UrbanSim rewards (with success) what it is supposed to (doctrinally), and whether the “reward signal is strong enough for the learner to differentiate between optimal and non-optimal strategies.” He finds that:

From the perspective of evaluating the fielded UrbanSim scenarios, it appears that the unstated, but assumed, training objective of rewarding students that conduct exclusively legal actions is properly rewarded. The training objective of emphasizing the doctrinal principle of ‘‘Clear, Hold, Build’’ did not stand out very clearly. However, it appeared to be in the range of acceptable solutions. The fact that the Build, Build, Build strategy was also in the range of acceptable solutions is not desirable because it reinforces the notion that you can be successful if you ignore the enemy and allow them to operate and you can still be successful in the scenario. The 4th training objective that wants the students to demonstrate that a mixture of lethal and non-lethal actions is better than exclusively lethal or non-lethal was not supported. Non-lethal actions were more strongly rewarded than the mixed approach and the lethal actions. This may be closely tied to the fact that the enemy units in the scenario do not affect the simulated environment enough to replicate the danger of ignoring enemy units operating in the area of operation.

As the thesis notes, the study is all about the feedback and rewards inherent within the game itself, and not about how it might be used instructionally. Obviously, it is better if the cues that the games provides to a player most closely align with course content and educational objectives. However, understanding how the game may, at times, misalign or send unclear signals is also extremely useful from an instructional point of view, allowing corrective action to be taken by an instructor (or even providing discussion opportunities for post-game hotwash and critique).

Brian mentioned he would welcome feedback on the thesis, his methods, and findings, so be pleased to add comments below.

* * *

Biographical note: Brian Vogt was commissioned an Armor Officer in 1996.  He served as a tank platoon leader, support platoon leader, and tank company executive officer in 1st Cav Div, Ft Hood.  Then, following the Armor Officer Advanced Course, he served as a Brigade Plans Officer in 2nd Inf Div, Camp Casey, Korea.  Subsequently he was a brigade current operations officer in 3BCT, 1AD, Ft Riley.  Took command of C/1-13AR in Baghdad in June 2003.  In August 2004, took command of HHC, 3BCT, 1AD for the second deployment to Iraq.  Upon redeployment in April 2006, became a FA57, Simulations Operations officer and worked in TCM-Virtual Training Environment at Ft Leavenworth.  Following CGSC in June 2010, he started classes at the Naval Postgraduate School in the MOVES Institute.  He graduates in September 2012 and will be stationed at Ft Eustis, VA.

Forthcoming Simulation & Gaming articles on peacebuilding

Several more articles from the forthcoming special issue of Simulation & Gaming on peacebuilding have now been made available in advance by SAGE, including our own short introduction to the collection:

Rex Brynen and Gary Milante, “Peacebuilding With Games and Simulations.”

Simulations and games can offer valuable insight into the management of conflict and the achievement of peace. This special symposium issue of Simulation & Gaming examines several such approaches, used in both educational settings and to prepare practitioners to deal with the concrete challenges of peacebuilding. In the introduction, the authors offer some brief thoughts on the how and why of simulations and games-based approaches, scenario choices (abstract, fictional, and real world), intended audiences, and design approaches. They also address the question of how games might (or might not) contribute to policy making in this field.

Tucker B. Harding and Mark A. Whitlock, “Leveraging Web-Based Environments for Mass Atrocity Prevention.”

A growing literature exploring large-scale, identity-based political violence, including mass killing and genocide, debates the plausibility of, and prospects for, early warning and prevention. An extension of the debate involves the prospects for creating educational experiences that result in more sophisticated analytical products that enhance preventive policy action. This article details an attempt to bridge the theory to practice gap. It describes the role of a simulation COUNTRY X within the educational contexts of both a graduate course in prevention of mass killing and genocide at Columbia University’s School of International and Public Affairs (SIPA), and a practitioner training workshop designed for regional conflict early warning analysts in Africa. The authors review educational theory describing problem-based learning and apply it to a web-based educational simulation. Using a recent training of professional conflict early warning analysts as their case study, they explore several assumptions regarding the utility of simulated environments as educational tools in moving from theory to practice. Use of the simulation resulted in active and engaged participation by learners, increased capacity for well-reasoned perspective taking, and improved analytical confidence in complex scenarios.

Richard B. Powers and Kat Kirkpatrick, “Playing With Conflict: Teaching Conflict Resolution Through Simulations and Games.”

Playing With Conflict is a weekend course for graduate students in Portland State University’s Conflict Resolution program and undergraduates in all majors. Students participate in simulations, games, and experiential exercises to learn and practice conflict resolution skills. Graduate students create a guided role-play of a conflict. In addition to an oral debriefing, students wrote a debriefing report following the Description, Interpretation, Evaluation (DIE) model of debriefing. The written debriefing report gave all students an opportunity to reflect, analyze, and evaluate their experience in depth. The use of two facilitators allows one to facilitate while the other observes and rests, makes 2 points of view available for the debriefing, and offers a model for resolving minor disagreements between them. Trust among students increased across the weekend as evidenced by an increase in cooperative choices and estimates of the likelihood that others would cooperate in the TAKE-A-CHANCE game, a version of PRISONER’S DILEMMA. Most reported having fun while they learned about themselves, interpersonal conflict, and some large-scale social conflicts.

Julian Schofield, “Modeling Choices in Nuclear Warfighting: Two Classroom Simulations on Escalation and Retaliation.”

Two classroom simulations—SUPERPOWER CONFRONTATION and MULTIPOLAR ASIAN SIMULATION—are used to teach and test various aspects of the Borden versus Brodie debate on the Schelling versus Lanchester approach to nuclear conflict modeling and resolution. The author applies a Schelling test to segregate high from low empathic students, and assigns them to hard case positions in three simulations to test whether high empathy students can engage in tactic bargaining and whether low empathetic students are necessarily as escalation prone. He has a bipolar nuclear simulation that is an easy case for the Brodie set of assumptions about nuclear war, avoidance, and Schelling-esque tacit bargaining. He expects the system structure and high empathy leader selection to contain escalation, despite the temptation of relying on accelerated Single Integrated Operational Plan solutions and the counterincentive of diminished tacit bargaining through decapitation attacks. The second simulation is a multipolar nuclear simulation set in the near future of Asia, and emulates the Borden-esque logic of nuclear war as artillery exchanges, with a Lanchester square law logic encouraging rapid escalation, coupled with a selection for the most autistic leadership. The author expects rapid nuclear escalation under these structural and decision-making conditions. His conclusions are anecdotal, but seem to indicate, from student feedback during class discussions, that the failure to model fear may be a factor in undermining successful tacit bargaining by players, suggesting that Borden rather than Brodie better conceptualized nuclear conflict. Therefore, peace is about restraining war initiation, as there are great pressures for escalation once war is initiated.

These and other forthcoming articles can be found here.

%d bloggers like this: