Conflict simulation, peacebuilding, and development

A critical look at serious game design

Yes, PAXsims does memes.

A new online-first article in Simulation & Gaming by Kristy de Salas (University of Tasmania) et al should be required reading for all serious game designers. In it, she and her colleagues undertook a systematic review of the English-language literature on “gameful interventions to improve behaviour related to environmental outcomes” published between 2015 and 2020. Only original, peer-reviewed articles on digital games were included. With these criteria 52 relevant papers identified and assessed.

What did they find? The article is paywalled, so I’ll excerpt some key findings below.

Regarding the types and contexts of pro-environmental games being developed, our study identified that the environmentally oriented gameful interventions were split between those classifying themselves as gamification – the use of game elements within a non-entertainment context – and serious games – full games designed for a behavioural outcome. While both gamification and serious games aim to influence a player to achieve a desired behaviour, the processes to achieve this outcome are vastly different in gamification and serious games, and clarity in classifying these interventions is important (Coreaxis, 2020). For example, within an environmental context, the intention of serious games is to directly improve long-term pro-environmental behaviours in a target group, whereas gamification aims to alter the attitude of a player – for example, increasing a player’s motivation and engagement towards participating in a short-term pro-environmental activity (Aubert et al., 2018). It is not the intention of gamification to influence long-term behaviour directly.

In reviewing these insights, we learn that designers with desires towards longer-term behavioural outcomes may be relying on a gamification model in the hope of bringing about change, despite the increasingly large volume of literature reporting its failure as an effective long-term strategy (dating back to 2011) (Bogost, 2015). This is of concern, for if we design gameful interventions with inaccurate underlying assumptions (i.e. believing that gamification in itself will bring about long-term change), outcomes will likely be compromised.

…there was a distinct absence of behavioural model informed design, justification for the use of these specific intervention functions, and the assessment of their affordability, practicability, effectiveness, acceptability, safety and equity. This is of particular concern as intervention design theory tells us that it is important to consider these design elements in order to make effective choices about which intervention functions are most appropriate or have the best potential chance of success in bringing about change in a particular context (Michie, Hyder et al., 2011). This lack of reporting on the consideration and application of design theory in our reviewed studies mirrors the limitations identified in other studies (Akl et al., 2010; Alanne, 2016; Alessandra et al., 2019; Battistella & von Wangenheim, 2016; Bodnar et al., 2016; da Silva et al., 2019; Farcas & Szamoskozi, 2016; Hersh et al., 2018; Lämsä et al., 2018; Leal et al., 2018; Mora et al., 2017; Osatuyi et al., 2018). This further reinforces the call for more thorough articulation of the current state of design to ensure best-practice design is being employed to bring about the target change.

Of further concern was the apparent lack of any articulated exploration of the appropriateness of games to their target audience. Designers of these gameful interventions were following the example of others trying to improve environmental outcomes, with only half of the studies describing the inclusion of any specific selection of specific behaviour change techniques to the improvement of either the capability, the opportunity or the motivation of the target audience to achieve the target behaviour or to engage with a game as the delivery mechanism, as is recommended by intervention design theory (Michie, Hyder et al., 2011). Rather, designers’ perceptions of games being efficient and low-cost approaches to achieve pro-environmental outcomes were the driving force for the intervention design rather than being informed by their suitability for the target audience. This finding mirrors that found by others when games are introduced for behaviour change outcomes in various disciplines (see, for example, Sharifzadeh et al., 2020).

Only one of our reviewed studies included the subject-matter expertise of environmental scientists, and only three included the discipline expertise of behavioural intervention designers, with the majority being developed by technological experts. This style of design team composition is consistent with practice recorded in much technology-informed intervention design in which design is often given over to technical developers (Salah et al., 2014); however, it is counter to best-practice user-centred design practice that suggests the need for multidisciplinary expertise in design teams to support the development of useful and usable interventions and systems (Gurses & Xiao, 2006; O’Brien et al., 2003).

Interestingly, despite the technical focus of the majority of gameful intervention development teams as just described, our reviewed papers included little to describe the influences on, and practice of, the technical design process of the interventions. Consequently, comparisons cannot be drawn across the design methodologies of the studies.

The extent of our knowledge from these design descriptions is limited to an understanding that these interventions were designed as mobile and web-based games that included traditional games elements such as points, levels, loot, feedback and badges to incentivise players.

Our reviewed papers did not sufficiently report the reasoning for incorporating specific game elements and difficulties arise therefore in determining the impact of reported game elements on a conceptual level such as the difference between implementing tasks and challenges. This lack of design description inhibits our opportunities to identify those elements of games and their design that have direct impact of the achievement of the targeted behavioural outcomes, a finding that is mirrored by other authors calling for more description in design to better inform the future design decisions of others

A further limitation to gaining a comprehensive understanding of the usefulness of games to bring about pro-environmental behavioural change is that not all reviewed studies undertook an evaluation of their intervention.

For those studies that did include an evaluation of their gameful intervention, a range of outcomes were reported across them, with the largest single proportion (28%) of studies reporting a positive change in either the target behaviour or the achievement of participant motivation towards conducting the target behaviour, followed by equal studies indicating mixed results or no difference (28%).

Difficulties arise in attempts to understand the differences in these reported outcomes as these results are influenced by many factors, including the range of outcomes being measured (including effectiveness, ease of use and usefulness); the nature of the data being collected (including survey and questionnaires, player metrics and interviews) and the range of evaluation tools (including single experiments with no control conditions, randomised control trials, observational studies and focus groups…

They conclude:

In this article, 52 articles reporting on gameful interventions were reviewed to determine the scope of games to support pro-environmental outcomes, the design of these systems and the evaluation of these interventions towards supporting the needs of the target audience. Our review has identified a lack of comprehensive articulation of the behavioural design elements to guide the intervention, including an absence of information regarding the process undertaken to gain an understanding the target behaviour and audience; a lack of justification for the selection of intervention functions and a failure to substantiate the use of a game as an appropriate delivery mode for the intervention.

We further identified that the reviewed gameful intervention designs do not include (or at least fail to articulate) best-practice activities such as multidisciplinary team composition, user-centred design or iterative design and feedback. In fact, the papers yield very little insight into the technical development practices of these interventions.

The reports use a range of primary measures, data collection tools and data sources to report on the outcomes of their interventions. This heterogeneity further limits opportunities for comparison.

In conclusion, our review of these 52 articles reporting on pro-environment gameful interventions has highlighted that despite the reported full or partial achievement of the goals of the interventions across these reviewed articles, we cannot yet be convinced that gameful interventions included in this specific review

•have been designed according to best-practice intervention design – including practices to understand the existing behaviour and the likelihood of changing that behaviour;

• have been designed according to best-practice technology development – including multidisciplinary teams and user-centred design;

• have considered thoroughly why a game is the most suitable delivery mechanism for the intervention;

• have selected evidence-based behaviour change techniques and mapped those to specific game elements within the design to ensure these act as ‘active ingredients’ of the intervention and

• are being evaluated based on best practice and can therefore report confidently on evidence-based outcomes of short-term engagement (in gamification interventions) or long-term behaviour change (in serious game interventions.

We suggest that future articles on gameful interventions should present information regarding their intervention design, and justification for their design choices, both behaviourally and technologically. In doing so, future reports on gameful interventions can better contribute to our body of knowledge on best-practice intervention design and evaluation practices, further contributing to the successful adoption of such interventions and the achievement of positive behavioural outcomes.

It’s a quite scathing critique—and one, I am confident, that applies to almost all serious gaming, including professional wargaming. Only in medical simulation and gaming, I think, do we see somewhat greater attention to some of these issues.

I would add that the problem may be even deeper than this, because I’m not confident that all of the literature on gaming, learning, and behavioural change rests on especially strong theoretical foundations. All too often, when designers or researchers invoke a theoretical paradigm, it is little more than a typology weakly supported by empirical research (such as the oft-cited “learning styles”).

3 responses to “A critical look at serious game design

  1. David Becker 26/07/2022 at 5:57 pm

    I would add that any survey or after action measure done in the 52 games probably does not follow up (6 -12 months later) to see if there is long term change, rather than short term enthusiasm. My experience with non game behavioral changes from standard adult learning is that little sticks long term. I have always hoped that gamification of materials would stick better, but have never seen research to support that.

  2. Timothy Smith 20/07/2022 at 9:00 am

    All ‘digital’ games? — is that a not a biased sample? It’s certainly a skewed pool. Computer games are entertainment products produced by profit-seeking manipulators and their software engineers. These are hard cultures to change. One can ‘contract’ with them to produce designs, but the principal-agent dichotomy still exists. Better to put topical subject-matter experts together with experts in (1) modeling (e.g., systems dynamics), (2) simulation-based learning cycles, (3) Bayesian data-processors, and (4) cognitive psychologists and learning theoreticians (e.g., collaborative/interactive learning), all under expert guidance by experienced interdisciplinary facilitators and have them design a tabletop sim model. But of course nobody in the hyper-superficial ‘gamification’ buzz-land understands anything about scientific methodology, interdisciplinary theory and actually doing WORK. Maybe this article will help encourage practitioners to start developing expertise.

  3. Josh Peery 20/07/2022 at 1:48 am

    I am more impressed by the quantity of those games rather than the quality. No one game is going to “silver bullet” someone’s attitude toward “environmental” issues. However, if there are a lot of that type of game, maybe there is a message getting through. Getting instructors, or even the general public, to accept a “game” as a teaching tool is hard enough. Undermining these games for their academic rigor, in this way, especially when it’s an attitude adjustment, showing a point of view, type game just seems counter productive to the field and continued practice of Serious Game design. Want serious games to be better? Write a best practices article. Research the end users of a game and publish results. As a practitioner I have done what I suggest this to help share my findings with the field. Don’t be smug and judge a bunch of games without attempting to research the questions/doubts the article author poses.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: