Conflict simulation, peacebuilding, and development

Daily Archives: 18/02/2015

Ellie Bartels on Research Design for Gaming


I’ll be giving a talk later today on how I use social science case methodology to think about game design. For those who are not able to attend, I wanted to post both my slides and a brief summary of my talk. This is part of an ongoing research effort, so feedback and thoughts are very much appreciated!

MORS Gaming COP Game Design from Social Science

There has been quite a lot of recent interest in expanding the use of gaming while ensuring that games are rigorous so they have a positive impact.  Traditional instruction on game design, such as NWC War Gaming Handbook or Peter Perla’s Art of Wargaming, stresses the need to make design choices in a thoughtful way in order to achieve game objectives, but does not provide much specific help translating objectives into choices about game roles, rules, and environments. More tools to help gamers think through design choices and communicate the potential impact of these choices on findings can help bridge this gap.

Recent work by other wargamers has discussed tools to apply more rigorous techniques to analyzing game results (see work by Wong and Cobb, Vebber, and Ducharme). However, as I discussed in an earlier post, some recent work conflates how structured the problem examined by the game is with how structured an approach is used to guide game design and analysis. Gaming is well-suited to examining unstructured problems, but to be done rigorously, it needs to be done in a structured way.

The goal then should be to find techniques for structured study of unstructured problems. Vebber and Wong and Cobb both use types of narrative analysis as one such approach, but there is also a role for a more generalized approach that might be useful for more types of games.

To that end, I propose a revision to the traditional design process based on case study methods from the social sciences.  While gaming and social science have been in dialog in national security analysis circles for the past several years, there is still not a well-developed collection of work connecting the two fields. However, because social scientists work on similar types of problems, it is worth considering what we gamers might be able to learn about structuring research and analysis.

Case study methodology is a particularly promising area of social science research design to tap into for gamers. Like gaming, case studies are used to study fairly unspecified problems, so are useful for theory creation and variable identification, as well as theory testing. Case study methods are also designed to focus on the mechanism that connects causes and effects, and are able to document complex causal relationships. As a result, case study methods are easier to apply to the type of unstructured problems we game than more quantitative techniques are.

I argue that we can often think of games as analogous to single case studies that look at variation over time or in comparison to a counterfactual in order to identify the mechanisms that link potential causes to outcomes of interest. While the findings of these approaches are not considered as strong as paired case studies (which are more commonly used in social science research as a result), they have a robust history of producing insights that advance our understanding of complex political, military, and social problems.


Applying the logic of case study research design then allows us to apply best practices from case study design to the development of games’ purpose and objectives; concepts; selection of scenario setting; definition of scenario, rules, and roles; and data collection.  I review some initial thoughts in this presentation, including the need to:

  • Identify common game objectives, such as pattern analysis and variable identification, which can provide ways to categorize games. This can allow us to develop best practices for tackling similar design problems even when games address different problems for different clients.
  • Require designers to explicitly state their understanding of the problem being gamed and how that hypothesis shapes what issues are highlighted or ignored in game design.
  • Encourage designers to clearly define input and outcome variables of interest, particularly the role of player decisions. Designers should also think through what confounding variables may appear in a game design, and how they might shape what can be concluded from the game.
  • More carefully select the scenario setting for games based on what type of analysis is being performed.
  • Consider how inevitable logistical limitations shape the testing environment of games, and how these limits should scope the applicability of game findings.
  • Better tailor data collection to strengthen analysis.

Each of these areas offers potential avenues for further development of more detailed best practices and techniques.

%d bloggers like this: