A new online-first article in Simulation & Gaming by Kristy de Salas (University of Tasmania) et al should be required reading for all serious game designers. In it, she and her colleagues undertook a systematic review of the English-language literature on “gameful interventions to improve behaviour related to environmental outcomes” published between 2015 and 2020. Only original, peer-reviewed articles on digital games were included. With these criteria 52 relevant papers identified and assessed.
What did they find? The article is paywalled, so I’ll excerpt some key findings below.
Regarding the types and contexts of pro-environmental games being developed, our study identified that the environmentally oriented gameful interventions were split between those classifying themselves as gamification – the use of game elements within a non-entertainment context – and serious games – full games designed for a behavioural outcome. While both gamification and serious games aim to influence a player to achieve a desired behaviour, the processes to achieve this outcome are vastly different in gamification and serious games, and clarity in classifying these interventions is important (Coreaxis, 2020). For example, within an environmental context, the intention of serious games is to directly improve long-term pro-environmental behaviours in a target group, whereas gamification aims to alter the attitude of a player – for example, increasing a player’s motivation and engagement towards participating in a short-term pro-environmental activity (Aubert et al., 2018). It is not the intention of gamification to influence long-term behaviour directly.
In reviewing these insights, we learn that designers with desires towards longer-term behavioural outcomes may be relying on a gamification model in the hope of bringing about change, despite the increasingly large volume of literature reporting its failure as an effective long-term strategy (dating back to 2011) (Bogost, 2015). This is of concern, for if we design gameful interventions with inaccurate underlying assumptions (i.e. believing that gamification in itself will bring about long-term change), outcomes will likely be compromised.
…there was a distinct absence of behavioural model informed design, justification for the use of these specific intervention functions, and the assessment of their affordability, practicability, effectiveness, acceptability, safety and equity. This is of particular concern as intervention design theory tells us that it is important to consider these design elements in order to make effective choices about which intervention functions are most appropriate or have the best potential chance of success in bringing about change in a particular context (Michie, Hyder et al., 2011). This lack of reporting on the consideration and application of design theory in our reviewed studies mirrors the limitations identified in other studies (Akl et al., 2010; Alanne, 2016; Alessandra et al., 2019; Battistella & von Wangenheim, 2016; Bodnar et al., 2016; da Silva et al., 2019; Farcas & Szamoskozi, 2016; Hersh et al., 2018; Lämsä et al., 2018; Leal et al., 2018; Mora et al., 2017; Osatuyi et al., 2018). This further reinforces the call for more thorough articulation of the current state of design to ensure best-practice design is being employed to bring about the target change.
Of further concern was the apparent lack of any articulated exploration of the appropriateness of games to their target audience. Designers of these gameful interventions were following the example of others trying to improve environmental outcomes, with only half of the studies describing the inclusion of any specific selection of specific behaviour change techniques to the improvement of either the capability, the opportunity or the motivation of the target audience to achieve the target behaviour or to engage with a game as the delivery mechanism, as is recommended by intervention design theory (Michie, Hyder et al., 2011). Rather, designers’ perceptions of games being efficient and low-cost approaches to achieve pro-environmental outcomes were the driving force for the intervention design rather than being informed by their suitability for the target audience. This finding mirrors that found by others when games are introduced for behaviour change outcomes in various disciplines (see, for example, Sharifzadeh et al., 2020).
Only one of our reviewed studies included the subject-matter expertise of environmental scientists, and only three included the discipline expertise of behavioural intervention designers, with the majority being developed by technological experts. This style of design team composition is consistent with practice recorded in much technology-informed intervention design in which design is often given over to technical developers (Salah et al., 2014); however, it is counter to best-practice user-centred design practice that suggests the need for multidisciplinary expertise in design teams to support the development of useful and usable interventions and systems (Gurses & Xiao, 2006; O’Brien et al., 2003).
Interestingly, despite the technical focus of the majority of gameful intervention development teams as just described, our reviewed papers included little to describe the influences on, and practice of, the technical design process of the interventions. Consequently, comparisons cannot be drawn across the design methodologies of the studies.
The extent of our knowledge from these design descriptions is limited to an understanding that these interventions were designed as mobile and web-based games that included traditional games elements such as points, levels, loot, feedback and badges to incentivise players.
Our reviewed papers did not sufficiently report the reasoning for incorporating specific game elements and difficulties arise therefore in determining the impact of reported game elements on a conceptual level such as the difference between implementing tasks and challenges. This lack of design description inhibits our opportunities to identify those elements of games and their design that have direct impact of the achievement of the targeted behavioural outcomes, a finding that is mirrored by other authors calling for more description in design to better inform the future design decisions of others
A further limitation to gaining a comprehensive understanding of the usefulness of games to bring about pro-environmental behavioural change is that not all reviewed studies undertook an evaluation of their intervention.
For those studies that did include an evaluation of their gameful intervention, a range of outcomes were reported across them, with the largest single proportion (28%) of studies reporting a positive change in either the target behaviour or the achievement of participant motivation towards conducting the target behaviour, followed by equal studies indicating mixed results or no difference (28%).
Difficulties arise in attempts to understand the differences in these reported outcomes as these results are influenced by many factors, including the range of outcomes being measured (including effectiveness, ease of use and usefulness); the nature of the data being collected (including survey and questionnaires, player metrics and interviews) and the range of evaluation tools (including single experiments with no control conditions, randomised control trials, observational studies and focus groups…
In this article, 52 articles reporting on gameful interventions were reviewed to determine the scope of games to support pro-environmental outcomes, the design of these systems and the evaluation of these interventions towards supporting the needs of the target audience. Our review has identified a lack of comprehensive articulation of the behavioural design elements to guide the intervention, including an absence of information regarding the process undertaken to gain an understanding the target behaviour and audience; a lack of justification for the selection of intervention functions and a failure to substantiate the use of a game as an appropriate delivery mode for the intervention.
We further identified that the reviewed gameful intervention designs do not include (or at least fail to articulate) best-practice activities such as multidisciplinary team composition, user-centred design or iterative design and feedback. In fact, the papers yield very little insight into the technical development practices of these interventions.
The reports use a range of primary measures, data collection tools and data sources to report on the outcomes of their interventions. This heterogeneity further limits opportunities for comparison.
In conclusion, our review of these 52 articles reporting on pro-environment gameful interventions has highlighted that despite the reported full or partial achievement of the goals of the interventions across these reviewed articles, we cannot yet be convinced that gameful interventions included in this specific review
•have been designed according to best-practice intervention design – including practices to understand the existing behaviour and the likelihood of changing that behaviour;
• have been designed according to best-practice technology development – including multidisciplinary teams and user-centred design;
• have considered thoroughly why a game is the most suitable delivery mechanism for the intervention;
• have selected evidence-based behaviour change techniques and mapped those to specific game elements within the design to ensure these act as ‘active ingredients’ of the intervention and
• are being evaluated based on best practice and can therefore report confidently on evidence-based outcomes of short-term engagement (in gamification interventions) or long-term behaviour change (in serious game interventions.
We suggest that future articles on gameful interventions should present information regarding their intervention design, and justification for their design choices, both behaviourally and technologically. In doing so, future reports on gameful interventions can better contribute to our body of knowledge on best-practice intervention design and evaluation practices, further contributing to the successful adoption of such interventions and the achievement of positive behavioural outcomes.
It’s a quite scathing critique—and one, I am confident, that applies to almost all serious gaming, including professional wargaming. Only in medical simulation and gaming, I think, do we see somewhat greater attention to some of these issues.
I would add that the problem may be even deeper than this, because I’m not confident that all of the literature on gaming, learning, and behavioural change rests on especially strong theoretical foundations. All too often, when designers or researchers invoke a theoretical paradigm, it is little more than a typology weakly supported by empirical research (such as the oft-cited “learning styles”).