This isn’t exactly liveblogging from the APSA Teaching and Learning conference—I’m not entirely sure how people synthesize their thoughts, type, and pay attention all at the same time—but I did think that I would offer a few semi-live, semi-random reflections arising from the ongoing discussions in the “simulations and role-play” track at the TLC. A few might even get expanded into fuller blog posts or discussions after the conference, and conference participants are certainly invited to add their own thoughts and comments below.
Simulations and losing control over course content. One participant has noted that, unlike a traditional lecture format, an instructor risks giving up control over course content to the sometimes unpredictable course of the simulation. What do you do if the simulation heads in the wrong direction, and the players then draw the wrong lessons from that?
I’ve seen this happen, at times, in my own Brynania simulation. If anything, some students plot and backstab even more than real-life political actors. Moreover, the highly personal nature of interaction in the sim, coupled with the absence of large bureaucracies, means that students can draw exaggerated lessons about the role of personal contact in diplomacy. This isn’t to say that such things aren’t important—diplomatic skills and idiosyncratic factors do make a difference, particularly at the level of key envoys, ministers, commanders, and coordinators—but it is somewhat less determinative of the broad sweep of foreign policies.
I tend to deal with this, or other gaps and misfits between my virtual sim world and the real one, by raising these issues explicitly in the end-of-simulation debrief, and explaining why things might be different in the real world than in Equatorial Cyberspace. Still, its an issue that I continue to grapple with.
UPDATE: Lukas Neville raises an excellent point in the comments about an inherent problem of short-term bargaining games, namely that the lack of iteration/repetition tends to degrade the real-life importance of credibility and reputation. In other words, there is far more incentive to lie and cheat in a sim that lasts a day or week than there is in a real-life relationship that may last for years. There’s no shortage of work on bargaining that shows that reputation matters, so this is presumably a count time to point it out. Also, one could probably design in-game mechanisms and disincentives that discourage this sort of unrealistic behaviour, and to be frank I probably haven’t done enough of that in Brynania.
Informing students of learning objectives. How fully should you inform participants in advance about what you want them to get out of the simulation? Identifying learning objectives may facilitate learning by allowing students to match what they are simulating with broader issues and questions. On the other hand, there are times when you don’t want to pre-warn people about the dilemmas and issues that will arise, or the mistakes they are likely to make.
I’m a big fan of learning-through-messing-up, especially in a context where no actual people die from civil conflict and humanitarian crisis. One of the things that I do in the Brynania simulation is warn students of their likely shortcomings in advance—”your donor coordination meetings will be more time-consuming and less productive than you hope, and most of you will information-share after the fact than truly coordinate projects before they are implemented”—then watch them repeat those dysfunctions despite their efforts to avoid them. This often leads into a useful post-simulation discussion on why it happened: stove-piping, donor pathologies, bureaucratic politics, national and local rivalries, time-constraints, local politics, and so forth.
I think it is all very context-dependent, depending on the group, the course, and the particular learning objectives and simulation concerned.
Assessing the impact of simulations. There are several papers at the TLC that seek to offer some assessment of the impact of simulations, typically by looking at the effects on student grades. While this is very valuable—and we certainly don’t want to see no, or negative, impact of simulation participation (!)—I would argue that it would probably be a mistake to use this as the sole proxy for simulation effectiveness.
This is because course exams typically test substantive knowledge (of theories, approaches, issues, actors, and relationships), and not more intangible qualities (negotiation and diplomatic skills, verbal and written communication, coalition-building, and generally a feel for the “art” of politics, peacemaking, and development programming). Soapbox warning: I think this is a major deficiency of academic political science in general, which is so focused on the existing corpus of theoretical literature that it provides relatively little sense of how politics operates on the ground. Others may disagree.
To grade, or not to grade? In academic settings, should students be evaluated on simulation participation, on simulation accomplishments, or on some form of post-simulation hot-wash/after-action/lessons-learned/self-reflection project?
The arguments in favour of evaluation include encouraging participation, rewarding hard work, and integrating learning objectives into the simulation. Administrators may also want to see some evidence that those noisy hours spent simulating in the classroom are producing measurable learning outputs. The disadvantages of simulation grading include fairness, and whether grading a simulation encourage the wrong types of competition and participation.
I don’t place a great deal of evaluation weight on the Brynania sim. Only 10% of the course grade is assigned for participation, and another 10-15% for a short post-simulation written debrief. The participation weight is low because it is difficult for me to monitor everything that 100+ students are doing over the week, and because I can’t guarantee that everyone has an equal chance to participate. If there is no peacekeeping operation in a given year, for example, UN DPKO has much less to do than a year when they are mandated to organize one. Similarly, the World Bank team may have little to do beyond generating “watching briefs” if a lasting ceasefire isn’t achieved. I typically have no problem motivating simulation participation, so there is no need to use grade rewards to do this. At times, in course evaluations, however, students have suggested a larger weight be assigned.
With regard to the debrief, I would assign it greater length and weight in POLI 450 if it weren’t the fact that the simulation is at the end of term, my class has just spent a week enthusiastically simulating civil war, and that as a result half of them are likely behind on assignments for other classes. There are limits to my cruelty…