PAXsims

Conflict simulation, peacebuilding, and development

Carvalho: Virtual Worlds Can Be Dangerous

logo-large-330x329Gustavo Carvalho, a PhD student in the Department of Political Science at the University of Toronto, has an interesting article appearing soon in International Studies Perspectives on the use of computer simulations in an international relations classroom—in this case, the online educational game Statecraft. He warns, based on use of the game in an introductory international relations course at U of T, that such games may not necessarily be very effective, and may also be less versatile in the classroom than non-digital “manual” games:

Video games have become a hot topic in education. To their proponents, they enhance the interactive and active aspects of learning. In addition, mass-produced off-the-shelf video games promise a cheaper and more convenient approach to education, being quick and easy to set up, in contrast to the extensive time commitment that goes into designing a simulation from scratch. My paper uses our experience with Statecraft, a commercial off-the-shelf IR computer simulation tailored to the educational market, as a proxy to discuss the educational usefulness of commercial strategy video games in general. Our experience recommends that we be cautious and reflective in the use of ready-made games for teaching. More to the point, it is still not clear which benefits, apart from convenience, commercial computer simulations bring to our classes that cannot also be provided by old-fashioned, low-tech customized simulations, whether designed by instructors or in collaboration with students.

The full text of the article is behind a paywall, but for readers who don’t have access to the journal here are some of the major points he makes, together with some comments of my own:

In view of how much they relied on the lectures, it is of some concern that only slightly over a quarter of the students said that the simulation helped them understand IR theories and concepts much (23.91%) or very much (4.35%), while close to a third (32.07%) ranked their experience only as average. Likewise, although a little more than a quarter of the students said Statecraft improved their engagement with the course readings (29.3%), a quarter felt that the improvement was average (25%), and almost half of them felt that it improved their engagement little (23.37%) or nothing at all (21.74%). Not surprisingly, given the status quo bias of Statecraft, only slightly over a fifth of the students said the simulation changed their previous views on international politics (21.19%). In a more positive tone, however, close to half of the students said Statecraft improved much (32.07%) or very much (13.59%) their engagement with the lecture and tutorials, while less than a fifth (19.02%) recorded an average change in engagement.

Much depends, of course, on how a game is integrated into curriculum, and especially how it is briefed and debriefed. That being said, the numbers aren’t very impressive—even more so when one considers that they are self-reported learning effects, which tend to be a more generous appraisal of game effects than objective learning measures (such as impact on test scores). One also needs to consider opportunity costs. It isn’t enough that a game have learning effects—it also needs to have learning effects that are greater or different from those that would be generated by a similar amount of time devoted to lectures, tutorials, readings, films, or other ways of examining the course material. It is an observation that international relations scholars James Robinson, Lee Anderson, Margaret Hermann and Richard Snyder made almost half a century ago in a seminal research article in the American Political Science Review, but which often gets lost amid contemporary enthusiasm for the gamification of learning. (It should be noted, however, that a 2013 paper by Chad Raymond on Statecraft reports much more positive learning experiences—again suggesting that much may depend on how any give game/simulation is used in the classroom.)

While it is often assumed that games-based learning is more attractive to students, Carvalho has some words of caution based on his classroom experience:

In contrast with the expectations of some scholars (Weir and Baranowski 2011:450), my first takeaway point is that computer-based simulations may in fact be unattractive to students, particularly those that do not feel at ease playing video games…. One student summarized this problem poignantly: “(…) I found that I got lost very quickly. Not being used to computer games I had to take more time to get acquainted with the rules and on top of trying to understand what was going on [at] my screen, I had to connect that with what I was learning.”

He also later notes that, based on the results from his class survey,  “commercial video game simulations appeal more to those students who already enjoy playing them outside of the academic environment.”

The author’s second takeaway from the experience relates to the issues of course design and other learning methods raised above:

My second takeaway point is that there is a careful balance to be struck between simulations, particularly video game-based ones, and traditional learning tools, such as lectures and tutorials. Simulations and video games do not replace good textbooks and content material, and they need to be carefully interwoven with lectures if they are to be effective educational tools (Aldrich 2009). Moreover, simulations and video games may also be detrimental to the experience of students who prefer traditional learning methods (Asal 2005:361) or feel uneasy in intense social situations.

His third major point relates to the problem of “realism” versus playability, a constant source of debate among conflict simulation designers:

My third takeaway point is that the trade-off between complexity and playability, important for games in general, is crucial in educational simulations. Game designers may feel tempted to increase the complexity of a game, or the amount of variables and elements that the players need to deal with, in order to make it seem more “realistic” (Sabin 2012:21). This seems to have been the case with Statecraft. Our data suggest that many students had trouble with the number of variables they had to control and with the choices they had to make in every turn of the game.

Gus also discusses the need for an educational game to be easy to play and run:

My fourth takeaway point is that off-the-shelf or commercial computer products may present serious technical challenges to course instructors. In the case of our experience with Statecraft, many software glitches had a direct impact on the performance of the countries, a serious problem for a simulation that relies too much on conflict-based game dynamics. In our case, the bugs and glitches were not serious enough to derail the simulation, but they may have been detrimental to our educational goals in the course, and particularly to the experiences of students that were not gamers to begin with

His fifth major point is a very important one, relating to the way in which any game models the “real world,” and the need to be aware of the potential message this sends to students:

My fifth takeaway point is that, when using commercial video games for teaching purposes, we need to be aware of the concepts and ideas that they either explicitly or implicitly transmit to the players. Game designers may be uncritical when choosing game mechanics or may be more concerned with making the game viable from a commercial point of view. Either way, their choices may not be equally useful in helping the students to better understand political science and IR, and some may actually be counterproductive to the goals of our courses.

In the case of Statecraft, it is difficult to say whether its designers had strong views about IR theories or were attempting to emulate successful games such as those in the Civilization and Age of Empires series.[13] Either way, as a result of their design choices, Statecraft ended up as a tragic caricature of international politics, to the detriment of its pretense realism. Instead of depicting the nuances of international politics, with the real trade-offs behind decision-making and the high costs associated with conflict, the game dynamics behind the simulation pushed the students to behave with the testosterone-infused logic of the stereotypical male gamer, including the “trash talk” and “trolling” that are associated with it. This is supported by the survey, with more than two-thirds of the respondents ranking the level of realism of Statecraft as average and lower, and was highlighted by the nuclear wars that occurred in two of our simulated worlds.

This is a widespread problem. We raised similar concerns a few years ago in a review of the manual, book-and-roleplay IR simulation  International Relations in Action: A World Politics Simulation. We also pointed to the problem of exaggerated levels of international conflict  in our recent review of the computer game Masters of the World

One final point in the article that is worth underscoring is how serious games in the classroom can positively affect inter-student dynamics:

…an interesting, and usually neglected, part of the entertainment factor of group simulations such as Statecraft is their social or community-building aspect (Aldrich 2009; Hofstede et al. 2010:830–832). In private conversations, some of my students noted how the simulation had actually brought them together and helped them connect with other students in their tutorials, a welcome change in our current environment of huge (and increasing) class and tutorial sizes.

This is a point my own students frequently make, with the friendships forged in the simulation often enduring for years after (or, in one notable case, resulting in marriage!)

All-in-all, an excellent piece, and well worth a read.

7 responses to “Carvalho: Virtual Worlds Can Be Dangerous

  1. J Wells (@introtoir) 12/08/2013 at 3:17 pm

    I went to a talk at ISA in San Francisco put on by a couple of the developers/managers of Statecraft. After about 45 minutes, I was quite underwhelmed. This post and the referenced article are really helpful in thinking more about some of the reservations I had after their talk.

  2. Rex Brynen 12/08/2013 at 3:37 pm

    More broadly, I think Gus’ article points to a number of things that IR simulations often don’t do very well:

    1) They embody or encourage a sort of hyperrealism, with an associated excessive use of armed force, trade embargoes, etc.

    2) They don’t much address domestic political constraints, or bureaucratic politics and process. Some (like Tessman’s International Relations in Action) do require multiple team players to sign off on a decision, although that’s not quite the same thing.

    3) They have rather poor modelling of economic interaction, markets, the private sector, and the international politics of trade, frequently rendering these in a highly statist, mercantilist way.

    This has long been a problem, and indeed has been since the very earliest academic IR simulations (such as the early 1960 Guetzkow INS simulation, which I first played as an undergraduate in the early 1980s).

    You can mitigate a lot of these weaknesses, however, if you encourage the students to undertake a critical analysis of the simulation experience, and then reflect on these strengths and weaknesses in your own debrief of the class.

  3. Brant 14/08/2013 at 3:33 pm

    one of the big takeaways from my dissertation research was that when you introduce some form of interactive digital tool (usually a game, but not always), that the user interface is a major issue. If your audience spends too much time ‘fighting the interface’ or trying to learn it on the fly, then they spend valuable lesson time learning and interface instead of the learning the material. For students who are already familiar with the interface (or one that’s substantially similar) then the buttonology takes up fewer cognitive resources that can instead be focused on the actual material to be learned

  4. Brant 14/08/2013 at 3:36 pm

    To follow Rex’s comments on what IR ‘sims’ don’t do well – they rarely, if ever, model the sorts of mass-mood-altering effects of the media that we see, also. You’d never have had the Spanish-American War, or the 1990s interventions in the Balkans or Somalia without the media’s role in shaping public opinion of the conflicts.

  5. J Wells (@introtoir) 19/08/2013 at 2:17 pm

    You know, it’s interesting, Rex, because from the talk I sat in on, I actually had an opposite set of concerns about Statecraft:

    1. Fighting is simulated as mere paper-rock-scissors. Not that there is anything wrong with PRS, but I would think a computer program developed over several years that includes a relatively sophisticated set of war-fighting units available for production would also include a more nuanced approach to armed conflict. One of the presenters actually said, “We don’t want them playing Risk,” which I appreciate, but I don’t think the answer is to build steering players away from conflict into the game;

    2. Statecraft seemed to be all about domestic politics! The talk went for about 45 minutes covering how players could order their internal politics, invest in domestic economic development, and manage social welfare systems. Finally, I asked what these states actually do to each other, and that led to about two minutes on the PRS system described above. There is also, as far as I could tell no terrorism, civil war, or intervention. There is some espionage and a bit of territorial war-fighting, with the caveat that resorting to nuclear weapons automatically produces bad grades.

    3. There wasn’t enough discussion of IPE for sure, but as I said above, there was a lot on domestic political economy.

    Of course, I have never played the game or even seen it played other than the brief videos they showed at this talk, but I was, and I think it worth repeating, definitely underwhelmed.

  6. Gustavo Carvalho 19/08/2013 at 11:57 pm

    Jeremy,

    the way Statecraft deals with domestic politics or interest groups is more of a sketch than a real simulation of the political dynamics in contemporary pluralistic societies (or, in fact, in any other regime type).

    Statecraft attempts to deal with the push and pull of domestic politics by simulating in each country (even totalitarian ones) a set of interest groups roughly divided by core policy interests (Greens, capitalists, environmentalists, and so on) that reacts to the actions and policies implemented by the students according to these interests. In the end, everything comes down to approval ratings (which can be a motivating factor until the students realize that acquiring territories and military power are the dominant strategies) and an occasion strike or riot (which can have a negative effect for resource production).

    Bureaucratic politics are also very thinly simulated by Statecraft. In theory, the students have different roles which control different bureaucracies in their countries, such as the military, intelligence, production, etc. In practice, the game does not provide any effective mechanism to make those roles work, unless the students themselves take the initiative and role-play them.

    Unfortunately, in our case the students were more interested in gaming the game than role-playing and, when the bombers and ICBMs started flying, everybody stopped paying attention to what was going on domestically. Well, in fairness, at that point some of them didn’t even have a country to control anymore.

    In my opinion, the main problem is that domestic or bureaucratic politics are not “internalized” in Statecraft. They are not inherent to what the students are doing, to their roles, or even to the policies they are implementing. Everything is still dictated by the “omniscient-benevolent-enlightened-dictator-player” with no real input from any possible interested parties, groups, selectorates, etc. As a result, these groups were seen more as a nuisance than a fundamental part of the “rules of the game” of political life in a realistically simulated country. As such, the students quickly learned to “solve” domestic politics by buying the support of the troublesome factions with token policies or specific city improvements.

    There are many other issues worth discussing, some of which I only touched in the paper. As you, Rex and Brant mentioned, the way Statecraft portraits IPE in general, the global or domestic economies, markets, etc. is very problematic. It doesn’t simulate the role of the media properly, it doesn’t simulate knowledge and scientific research well, and so on. It may not even simulate conflict, arguably its core strength, in a thoughtful and realistic way. Digital World Construction may change and improve the game, but there are fundamental issues they need to sort out before they can offer a better product. In fact, one of the major ones is the very idea that Statecraft should simulate the whole of IR. Instead of (poorly) simulating everything, they should focus and try to better simulate fewer issue-areas. Food for thought.

    Naturally, these problems are not particular to Statecraft. My paper attempts to go beyond it to discuss issues that may be common to other types of commercial video games (or even “analog”, pen and paper, simulations). But they certainly clouded our experience with Statecraft.

  7. Rex Brynen 20/08/2013 at 1:01 am

    In fairness to Statecraft I should reiterate a point I made in my earlier pos–namely that a great deal depends on how Statecraft, or any sim, is integrated into the course, and how it is briefed and debriefed.

    A game can’t do everything, and it certainly can’t do everything well. However, the things it omits (or distorts, or poorly portrays) can themselves be very teachable moments if the debriefing/discussion/related course assignments are designed appropriately. For example, the tendency to war among UofT students could be used to generate a lecture or discussion on why the game behaviour might differ from actual state behaviours.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: