Brian Vogt has kindly allowed us to upload a copy of his recently-completed MSc thesis at the Naval Postgraduate School on “A Methodology to Assess UrbanSim Scenarios.” In it, he examines whether the choices and feedback mechanisms in the counter-insurgency software UrbanSim actually support its intended learning objectives:
Turn-based strategy games and simulations are vital tools for military education, training, and readiness. In an era of increasingly constrained resources and expanding demand for training solutions, the need for validated, effective solutions will increase. Appropriate performance feedback is an important component of any training solution. Current methods for designing and testing the performance feedback provided in turn-based simulation are limited to well-structured problems and do not adequately address ill-structured problems that better replicate problems facing military leaders in today’s complex operating environment. This thesis develops and explores new methods for assessing the feedback mechanisms of turn-based strategy games. Using UrbanSim, a game for training strategic approaches to COIN operations as an exemplar, this thesis developed and explored two unique methods for evaluating the reward structure of the UrbanSim scenarios. The first method evaluates different student strategies using a batch-run method. The second method uses a reinforcement-learning algorithm to explore the decision space. These scenario evaluation methodologies are shown to be able to provide insights about a game’s performance feedback mechanism that was not previously available. These methodologies can be used for formative evaluation during game scenario development. Additionally, these evaluation methodologies are generalizable to other training and education games that focus on ill-structured problems and decision-making at discrete intervals.
Brian offers some excellent insight into feedback mechanisms in military training games in general, as well as in the specific case of UrbanSim. He runs large numbers of iterations of possible strategies in the game, so as to assess whether UrbanSim rewards (with success) what it is supposed to (doctrinally), and whether the “reward signal is strong enough for the learner to differentiate between optimal and non-optimal strategies.” He finds that:
From the perspective of evaluating the fielded UrbanSim scenarios, it appears that the unstated, but assumed, training objective of rewarding students that conduct exclusively legal actions is properly rewarded. The training objective of emphasizing the doctrinal principle of ‘‘Clear, Hold, Build’’ did not stand out very clearly. However, it appeared to be in the range of acceptable solutions. The fact that the Build, Build, Build strategy was also in the range of acceptable solutions is not desirable because it reinforces the notion that you can be successful if you ignore the enemy and allow them to operate and you can still be successful in the scenario. The 4th training objective that wants the students to demonstrate that a mixture of lethal and non-lethal actions is better than exclusively lethal or non-lethal was not supported. Non-lethal actions were more strongly rewarded than the mixed approach and the lethal actions. This may be closely tied to the fact that the enemy units in the scenario do not affect the simulated environment enough to replicate the danger of ignoring enemy units operating in the area of operation.
As the thesis notes, the study is all about the feedback and rewards inherent within the game itself, and not about how it might be used instructionally. Obviously, it is better if the cues that the games provides to a player most closely align with course content and educational objectives. However, understanding how the game may, at times, misalign or send unclear signals is also extremely useful from an instructional point of view, allowing corrective action to be taken by an instructor (or even providing discussion opportunities for post-game hotwash and critique).
Brian mentioned he would welcome feedback on the thesis, his methods, and findings, so be pleased to add comments below.
* * *
Biographical note: Brian Vogt was commissioned an Armor Officer in 1996. He served as a tank platoon leader, support platoon leader, and tank company executive officer in 1st Cav Div, Ft Hood. Then, following the Armor Officer Advanced Course, he served as a Brigade Plans Officer in 2nd Inf Div, Camp Casey, Korea. Subsequently he was a brigade current operations officer in 3BCT, 1AD, Ft Riley. Took command of C/1-13AR in Baghdad in June 2003. In August 2004, took command of HHC, 3BCT, 1AD for the second deployment to Iraq. Upon redeployment in April 2006, became a FA57, Simulations Operations officer and worked in TCM-Virtual Training Environment at Ft Leavenworth. Following CGSC in June 2010, he started classes at the Naval Postgraduate School in the MOVES Institute. He graduates in September 2012 and will be stationed at Ft Eustis, VA.