Carvalho: Virtual Worlds Can Be Dangerous

Gustavo Carvalho, a PhD student in the Department of Political Science at the University of Toronto, has an interesting article appearing soon in International Studies Perspectives on the use of computer simulations in an international relations classroom—in this case, the online educational game Statecraft. He warns, based on use of the game in an introductory international relations course at U of T, that such games may not necessarily be very effective, and may also be less versatile in the classroom than non-digital “manual” games:

Video games have become a hot topic in education. To their proponents, they enhance the interactive and active aspects of learning. In addition, mass-produced off-the-shelf video games promise a cheaper and more convenient approach to education, being quick and easy to set up, in contrast to the extensive time commitment that goes into designing a simulation from scratch. My paper uses our experience with Statecraft, a commercial off-the-shelf IR computer simulation tailored to the educational market, as a proxy to discuss the educational usefulness of commercial strategy video games in general. Our experience recommends that we be cautious and reflective in the use of ready-made games for teaching. More to the point, it is still not clear which benefits, apart from convenience, commercial computer simulations bring to our classes that cannot also be provided by old-fashioned, low-tech customized simulations, whether designed by instructors or in collaboration with students.

The full text of the article is behind a paywall, but for readers who don’t have access to the journal here are some of the major points he makes, together with some comments of my own:

In view of how much they relied on the lectures, it is of some concern that only slightly over a quarter of the students said that the simulation helped them understand IR theories and concepts much (23.91%) or very much (4.35%), while close to a third (32.07%) ranked their experience only as average. Likewise, although a little more than a quarter of the students said Statecraft improved their engagement with the course readings (29.3%), a quarter felt that the improvement was average (25%), and almost half of them felt that it improved their engagement little (23.37%) or nothing at all (21.74%). Not surprisingly, given the status quo bias of Statecraft, only slightly over a fifth of the students said the simulation changed their previous views on international politics (21.19%). In a more positive tone, however, close to half of the students said Statecraft improved much (32.07%) or very much (13.59%) their engagement with the lecture and tutorials, while less than a fifth (19.02%) recorded an average change in engagement.

Much depends, of course, on how a game is integrated into curriculum, and especially how it is briefed and debriefed. That being said, the numbers aren’t very impressive—even more so when one considers that they are self-reported learning effects, which tend to be a more generous appraisal of game effects than objective learning measures (such as impact on test scores). One also needs to consider opportunity costs. It isn’t enough that a game have learning effects—it also needs to have learning effects that are greater or different from those that would be generated by a similar amount of time devoted to lectures, tutorials, readings, films, or other ways of examining the course material. It is an observation that international relations scholars James Robinson, Lee Anderson, Margaret Hermann and Richard Snyder made almost half a century ago in a seminal research article in the American Political Science Review, but which often gets lost amid contemporary enthusiasm for the gamification of learning. (It should be noted, however, that a 2013 paper by Chad Raymond on Statecraft reports much more positive learning experiences—again suggesting that much may depend on how any give game/simulation is used in the classroom.)

While it is often assumed that games-based learning is more attractive to students, Carvalho has some words of caution based on his classroom experience:

In contrast with the expectations of some scholars (Weir and Baranowski 2011:450), my first takeaway point is that computer-based simulations may in fact be unattractive to students, particularly those that do not feel at ease playing video games…. One student summarized this problem poignantly: “(…) I found that I got lost very quickly. Not being used to computer games I had to take more time to get acquainted with the rules and on top of trying to understand what was going on [at] my screen, I had to connect that with what I was learning.”

He also later notes that, based on the results from his class survey,  “commercial video game simulations appeal more to those students who already enjoy playing them outside of the academic environment.”

The author’s second takeaway from the experience relates to the issues of course design and other learning methods raised above:

My second takeaway point is that there is a careful balance to be struck between simulations, particularly video game-based ones, and traditional learning tools, such as lectures and tutorials. Simulations and video games do not replace good textbooks and content material, and they need to be carefully interwoven with lectures if they are to be effective educational tools (Aldrich 2009). Moreover, simulations and video games may also be detrimental to the experience of students who prefer traditional learning methods (Asal 2005:361) or feel uneasy in intense social situations.

His third major point relates to the problem of “realism” versus playability, a constant source of debate among conflict simulation designers:

My third takeaway point is that the trade-off between complexity and playability, important for games in general, is crucial in educational simulations. Game designers may feel tempted to increase the complexity of a game, or the amount of variables and elements that the players need to deal with, in order to make it seem more “realistic” (Sabin 2012:21). This seems to have been the case with Statecraft. Our data suggest that many students had trouble with the number of variables they had to control and with the choices they had to make in every turn of the game.

Gus also discusses the need for an educational game to be easy to play and run:

My fourth takeaway point is that off-the-shelf or commercial computer products may present serious technical challenges to course instructors. In the case of our experience with Statecraft, many software glitches had a direct impact on the performance of the countries, a serious problem for a simulation that relies too much on conflict-based game dynamics. In our case, the bugs and glitches were not serious enough to derail the simulation, but they may have been detrimental to our educational goals in the course, and particularly to the experiences of students that were not gamers to begin with

His fifth major point is a very important one, relating to the way in which any game models the “real world,” and the need to be aware of the potential message this sends to students:

My fifth takeaway point is that, when using commercial video games for teaching purposes, we need to be aware of the concepts and ideas that they either explicitly or implicitly transmit to the players. Game designers may be uncritical when choosing game mechanics or may be more concerned with making the game viable from a commercial point of view. Either way, their choices may not be equally useful in helping the students to better understand political science and IR, and some may actually be counterproductive to the goals of our courses.

In the case of Statecraft, it is difficult to say whether its designers had strong views about IR theories or were attempting to emulate successful games such as those in the Civilization and Age of Empires series.[13] Either way, as a result of their design choices, Statecraft ended up as a tragic caricature of international politics, to the detriment of its pretense realism. Instead of depicting the nuances of international politics, with the real trade-offs behind decision-making and the high costs associated with conflict, the game dynamics behind the simulation pushed the students to behave with the testosterone-infused logic of the stereotypical male gamer, including the “trash talk” and “trolling” that are associated with it. This is supported by the survey, with more than two-thirds of the respondents ranking the level of realism of Statecraft as average and lower, and was highlighted by the nuclear wars that occurred in two of our simulated worlds.

This is a widespread problem. We raised similar concerns a few years ago in a review of the manual, book-and-roleplay IR simulation  International Relations in Action: A World Politics Simulation. We also pointed to the problem of exaggerated levels of international conflict  in our recent review of the computer game Masters of the World. 

One final point in the article that is worth underscoring is how serious games in the classroom can positively affect inter-student dynamics:

…an interesting, and usually neglected, part of the entertainment factor of group simulations such as Statecraft is their social or community-building aspect (Aldrich 2009; Hofstede et al. 2010:830–832). In private conversations, some of my students noted how the simulation had actually brought them together and helped them connect with other students in their tutorials, a welcome change in our current environment of huge (and increasing) class and tutorial sizes.

This is a point my own students frequently make, with the friendships forged in the simulation often enduring for years after (or, in one notable case, resulting in marriage!)

All-in-all, an excellent piece, and well worth a read.