The following post was contributed by Jonathan Keller (James Madison University). For an earlier PAXsims summary of Gustavo Carvalho’s forthcoming article, see here.
* * *
The Pedagogy of Statecraft
I would like to thank Rex Brynen for the opportunity to join this conversation on his excellent blog. A good discussion has been provoked by the forthcoming International Studies Perspectives article by Gustavo Carvalho entitled “Virtual Worlds Can Be Dangerous: Using Ready-Made Computer Simulations for Teaching International Relations.” The article focuses on one class’s (largely negative) experience with the Statecraft simulation at the University of Toronto and generally casts doubt on the effectiveness of Statecraft as a teaching tool.
As the creator of Statecraft, I read this article with great interest. Statecraft is certainly not a perfect simulation, and I am interested in feedback on ways in which it can be improved. I was disappointed to discover, however, that Carvalho had employed Statecraft in at least four ways that were directly counter to the explicit instructions we provide to professors. These instructions are not arbitrary, but are the result of over a decade (now approaching 15 years) of observing the pedagogical impact of different Statecraft design choices in a variety of IR courses. The instructions were designed to maximize Statecraft’s pedagogical effectiveness and to prevent precisely the sorts of negative outcomes this class experienced. Specifically, in the Toronto class that was the subject of the ISP article, (1) students were not incentivized to learn the simulation rules through the online manual quizzes, (2) the all-important grading system (which encourages realistic behavior) appears not to have been used, (3) countries and student roles were not set up properly before Turn 1 began, leading to widespread confusion, and (4) the instructor materials (lecture outlines, assignments, etc.) that are essential for helping students make sense of their Statecraft experience are nowhere mentioned in the article and appear not to have been used.
ISP has decided to include my rebuttal alongside Carvalho’s article in print. This forthcoming article is entitled “Misusing Virtual Worlds Can Be Dangerous: A Response to Carvalho.” This article provides context regarding Statecraft’s design, instructions for use, and pedagogical intent that were missing from the Carvalho piece, so that readers may gain a more complete picture of what Statecraft was intended to do and how it is designed to work.
I encourage PaxSim’s readers to read this rebuttal, which should be available soon on ISP’s “Early View” if it is not already. But here I’d like to highlight briefly three pedagogical lessons that the Toronto experience (and my own 15-year history developing Statecraft) suggest regarding the use of simulations.
First, grading criteria greatly affect students’ behavior and should be carefully calibrated to produce the dynamics the instructor wishes to illustrate. The clearest lesson from the early trials of Statecraft (1999-2002, back when it was a purely “paper and pencil” simulation) was that unless students are given incentives to behave like real world leaders, Statecraft will quickly degenerate into entertaining but unrealistic global warfare, with a heavy emphasis on nuclear weapons. One student described an early version of the simulation as “college kids with nukes.” The current Statecraft grading system is a result of this experience, and it gives tangible incentives for students to pursue the range of goals that have historically motivated real world countries (national prestige/distinctiveness, domestic development, cooperation on transnational issues, and imperial conquest), without telling students which of these goals they must pursue. Since Statecraft assigns students to countries using a foreign policy attitudes survey, there will always be a mix of hardline countries, pacifist regimes, and so on. The “Historians’ Verdict” award was introduced specifically to curb unrealistic resort to nuclear warfare, and when used it virtually eliminates nuclear war in Statecraft. In the last 10 years of using the recommended grading system (described in detail in my forthcoming ISP article), about 40% of my “worlds” have avoided war altogether, and only one nuclear weapon has ever been launched. I encourage instructors to tweak the default grading criteria to achieve the type of “world” they want their students to experience, but they should be cautious about diverging too far from these thoroughly tested criteria. The extraordinary bellicosity of the world described in the Carvalho article, together with the omission of any mention of the grading system, indicates that the recommended grading criteria were not used.
Second, precise verisimilitude with the real world should not necessarily be the primary goal of IR simulations. Yes, some degree of realism is necessary in order to illustrate key concepts and replicate the core dynamics of world politics. But if a given run of Statecraft produces outcomes that diverge from real-world outcomes, this should not be an occasion for despair (as the Carvalho article seems to suggest) but presents a golden opportunity for reflection and critical thinking. If a class finds itself locked in conflict spirals and the UN is impotent, the instructor can ask students what factors are driving the conflict and under what conditions these processes are likely to be replicated in world politics. He or she can ask students whether these outcomes approximate the predictions of realists or liberals, and can encourage them to consider whether their classroom experience with an ineffective UN parallels the limitations of the real UN, or whether the actual UN has more influence than the Statecraft version, and why. This is how Statecraft was designed to be used, as evidenced by the many discussion questions and paper assignments (provided to instructors using Statecraft) that ask students to actively critique the assumptions behind the simulation design and compare their classroom experience with their observations of world politics.
Finally, no matter how well designed a simulation is, student learning will be stunted if the simulation experience occurs in a vacuum. It is still the job of the instructor to make clear the connections between students’ simulation experience and class material. Statecraft is intended to be fully integrated into IR courses through lecture, discussion, exams, and paper assignments. (All of these instructor resources, including 39 pages of lecture outlines on 13 different IR topics, are included with Statecraft). It is therefore not surprising that Carvalho’s students—who, based on his article, were not exposed to lectures or assignments making sense of their Statecraft experience—expressed skepticism about the utility of Statecraft as a teaching tool. As Carvalho notes (p. 13), “Simulations and video games do not replace good textbooks and content material, and they need to be carefully interwoven with lectures if they are to be effective educational tools.” On that point, we are in complete agreement.
Hopefully the upcoming publication of Carvalho’s piece and my response in ISP will continue to generate productive discussion on the pedagogy of IR simulations. I believe that the Toronto class experience in spring 2012, when properly understood, offers constructive lessons about the limitations of simulations as standalone teaching tools and the ways in which Statecraft can most usefully be employed.
Jonathan Keller is Associate Professor of Political Science at James Madison University. He received his Ph.D. from The Ohio State University in 2002. His research and teaching interests include political psychology, foreign policy decision-making, U.S. foreign policy, and research methods. His work has appeared in the Journal of Politics, Journal of Conflict Resolution, Journal of Peace Research, International Studies Quarterly, Political Psychology, Conflict Management and Peace Science, and Foreign Policy Analysis.