PAXsims

Conflict simulation, peacebuilding, and development

Monthly Archives: March 2009

more simulation news

Given that I work on the Israeli-Palestinian conflict for a living, I can hardly omit mention of this one, buried in a report on the airstrikes in Sudan earlier this year:

IAF airstrike in Sudan hit convoy of weapons destined for Gaza

Haaretz, 27 March 2009
By Yossi Melman, Amos Harel and Barak Ravid, Haaretz Correspondents, and News Agencies

…Meanwhile, in May, an international conference is scheduled to take place in Ottawa, the third of its kind since the end of Operation Cast Lead, which will discuss how to prevent arms smuggling from Iran to the Gaza Strip. 

In addition to host Canada, Britain, Spain, France, Germany, Italy, Norway, Denmark, the U.S. and Israel will also take part. 

Immediately after the conference a “war game” is scheduled to take place in Washington, with the participation of security officials and diplomats from the countries involved. The “war game” will practice a scenario of foiling arms smuggling from Iran to the Gaza Strip. 

It will be interesting to see whether the wargame is intended largely as a political-diplomatic feel-good gesture  to create a sense that there is forward movement on this issue, or whether it is hoped to foster great practical cooperation through identifying (for example) the operational requirements, dynamics, and necessary resources and cooperation mechanisms to interdict smuggling.

If it is the latter, I hope they have someone adequately red-team gaming the Iranians and Hamas, so that the simulation fully reflects the potential perils and complications of interdiction operations.

simulation news

University of Utah students participate in counterterrorism simulation

Deseret News, 27 March 2009.

By Ethan Thomas

Imagine being a high-ranking government official on a day when terrorists execute several attacks on U.S. soil and abroad.

Some law students at the S.J. Quinney College of Law at the University of Utah had to walk a mile in a few of those officials’ shoes Friday in a live counterterrorism simulation at the law school.

The fake attacks included an explosion that rocked the U.S. embassy in Tel Aviv, suicide bombings at the Lincoln Memorial, Washington Monument and Holocaust Museum in Washington D.C., ferry explosions in San Francisco and anthrax scares at several hospitals in the U.S. capital.

And that was just the first hour of the six-hour simulation.

 

 

Badges, fake guns, interrogations and arrests in IRS simulation for students

KOAM TV 7, 27 March 2009.

PITTSBURG, KAN. – April 15 is the tax deadline, and if you were contemplating not paying taxes this year, dozens of students are gearing up to make sure you go down.

On Friday an IRS training program called the Adrian Project was held at Pittsburg State University, giving students a real world simulation including badge and fake gun.

The goal is to help accounting and criminal justice students who are interested in the career.

“It’s a program that gives us a chance to come out to the universities and maybe even the high schools and give the students an idea of what an IRS agent does,” said Toni Weirauch, the special agent in charge.  “We try to give them the full gammet of what we do, as well as they will be doing some arresting and hand cuffs and some felony car stops.”

From the oath to the arrest

The students took an oath, and were fitted with the Adrian Project’s version of a badge and a fake gun.

Then they start from square one, interviewing and interrogating mock suspects of tax evasion.

They investigated accounts where a tax return preparer may try to hide income, whether legally or illegally.

“It’s kind of like a puzzle and that’s what’s real fun to me,” said PSU senior Kyle Spechinger.  “You have to put a lot of pieces together before you can come up with your answer.  It’s cat and mouse.”

 

 

Local residents participate in Poverty Simulation Exercise

Pensacola News Journal, 20 March 2009

Jean Norman
 

Oatmeal or fruit for breakfast? Is there time to read over a newspaper or two? What about visit an elderly neighbor and wish him a happy birthday?

These are the decisions I made early Tuesday morning.

The questions were harder for me later in the day, and for about 55 local residents participating in a Poverty Simulation Exercise at the Langley Bell 4-H Center. Our imaginary worries: When will we get some food? Will I lose my job if I take my sick child to the emergency room? How will we get there?

The poverty simulation, conducted by Michael Gutter, assistant professor at University of Florida and sponsored by the Unite Escambia Poverty Solutions Team, made participants grapple with hard questions in a fast-moving, stress-filled couple of hours that demonstrate the frustrations of trying to survive with few resources and plenty of problems.

 

four seasons of simulations

BSG cast at UN

Well, alright, it is a bit of a stretch to liken Battlestar Galactica to a learning simulation.  Still, I would contend that a thought-provoking piece of film or literature pushes many of the same buttons that a simulation does if it can push viewers/readers to consider the ways the world could be in light of new learning about the way the world is

This seems to have happened on Wednesday night when the UN brought together a very unique panel of policymakers and cast/creators of BSG for a discussion on human rights.  I wish I could’ve attended, but what struck me most about the eyewitness accounts was  the blurring of reality and show (simulation) – like Olmos turning to students and saying “So say we all” – to which the room responded “So say we all”.   I think part of what makes storytelling so powerful is how visionary it can be – is it the same kind of vision we need for policymaking? 

Tonight is the series finale for BSG.  Thanks to the creators, writers and cast for bringing us into their simulation in their quest for Earth, whatever it is.

a force more tedious

game170I agreed to review the pc simulation “A Force More Powerful” (AFMP) by request from Rex, who couldn’t run the game on his Mac (how convenient…).  I was actually looking forward to it, after all, it is a simulation, I enjoy games and simulations of all types, I enjoy playtesting, and I was especially intrigued by the subject matter: nonviolent resistance.  Even with this enthusiasm, I was underwhelmed by the AFMP experience, which saddens me because I really wanted to like it and write a glowing review here.  Here are some observations.

In AFMP you are the organizing “spirit” of a resistance movement in 10 scenarios against the Regime (an intentionally impersonal pronoun reflecting the powers that be in each of the scenarios).  These scenarios range from attempting to peacefully and democratically overthrow a dictator in a little island state, to fighting corruption in a small eastern european city, to protesting a war in a democracy.   I say “spirit” because you are effectively the senior strategist of your movement, though you don’t have a character or a particular role, per se—one weakness of the design to which I’ll return below.  You assign the characters in your organizations various tactics to accomplish policy or regime change in the scenario to fulfill objectives (which you set and prioritize – a good choice by the game designers).

In each scenario you are given a very thorough briefing on the issues, the major groups and actors (characters).  This is presented in a notebook format, with really rough sketches and a lot of handwritten notes.  The interface is actually a really endearing part of the game—the graphic design actually feels like a real grass-roots group operating out of a warehouse, notes written out on scratch paper, organizational charts that might be pinned on a corkboard, graphs that look like fading pens on old overhead transparencies.   In this respect, the feel of the simulation is actually really well done, an example of the designers playing to their relative strengths.  You are also given some rough sketched maps of the island, country, regions, etc. with key objectives identified, though there doesn’t seem to be a lot of relevance to these objectives in actual game play. 

As each scenario develops, you work through up to four phases in a pre-determined timeline trying to reach your objectives outlined above through various tactics (boycotts, mass protests, fundraising, pickets, publishing newspapers, fundraising, sending letters, occupying buildings, fundraising, organizing social events, fundraising, training, upgrading communications…).  There are about 40 different tactics you can use to reach your objectives, though you end up organizing social events and writing letters most of the time, oh, and fundraising—did I mention fundraising?  Those letters need postage, so you spend a lot of your time assigning characters to fundraising—I thought the grant chasing at the World Bank and academia was bad—according to AFMP these organizations spend a LOT of time raising money for their cause, when they aren’t in jail.

I played three scenarios and in two of the three I eventually gave up (on normal difficulty) because more than half of my characters were imprisoned with seemingly little chance of being released by an unsympathetic Regime.  This may be extremely realistic, but one wonders at the pedagogical value of month after month of game time (minute after minute of real life) spent advancing time only to be given the meaningless choice of ineffective hunger strikes or ineffectual letter writing.  Not only are your character’s incarcerated, but you have no way of getting them out—none of the nonviolent strategies at your disposal can be directed at the police or the court system to liberate your characters (no picketing the police, no rallies, there are vigils, but they can’t be used to get anyone out of jail).  So, my experience was generally a tedious march of my little rabble rousers heading off to jail, the occasional freedom bought with a hunger strike and the long, painful death of a movement with no forward momentum. 

So, sadly, this review isn’t real positive, but I still think there is a lot to learn about simulation design from this experience, which I’ll limit to four basic points:

  1. Design from learning objectives
  2. Engagement of the audience
  3. Strengths of your simulation
  4. Playtesting

Design from Learning Objectives

The simulation originated from a documentary series by the same name which I will definitely pick up . Alhough the simulation is (in my humble opinion) weak, the designers clearly know a lot about the subject matter and I want to learn more from them.  Still, much of the knowledge that the designers have is lost in the tedium of the simulation.  Had they designed the simulation from clearly defined learning objectives  at the outset, I think it would be a better product.  As it is, the participants are overwhelmed by a lot of information about actors, organizations and tactics in a kind of sandbox for nonviolent resistance where it is difficult to map any actual activities to the outcomes that follow.  Granted, this might be realistic, but it isn’t a particularly effective way of learning and it is a pitfall I am sure we all risk falling into in our simulation designs.

Engaging the Audience

As I mentioned above, while you, the player, decide all of the strategy for the movement, because you are a kind of Senior Strategist Spirit (no character, no role, no identity) there is very little emotional buy-in to what could otherwise be quite an engaging simulation experience.  You currently watch while your main characters go through the trials and tribulations of non-violent resistance – and you really just watch. The simulation would be much more engaging if the player actually had a role, had an identity, could actually be imprisoned or put under house arrest, if you had to communicate what the movement should be doing through letters or passing notes to prison guards, if you had to balance the political significance of a hunger strike against the real possibility that it might weaken or kill you. Without embodiment in the simulation world, very few of these choices have any emotional resonance with the player and it is disappointing.  The lesson from this is that when possible and practical, actually identifying players with characters, roles, goals and identities can be very useful in engaging them.  Who knows how many more turns I would’ve played of AFMP had my character been in prison and I needed to figure out a way out?

Strengths of your simulation

It is ironic to me that AFMP teaches us that non-violent resistance is often the best response to authoritarianism because states have comparative advantage in the use of violence to reach their objectives—following Sun Tzu, we should choose our battlefields where we have relative strengths, not engage our opponents where they are strongest —yet, the designers of AFMP themselves decided to compete against their opponents (real-time strategy (RTS) games) exactly where RTS are the strongest (graphics, maps, video sequences).  Honestly, I have better graphics and video sequences on my blackberry than AFMP delivers.  This is tragic because they didn’t have to engage their competition in this forum – they could’ve left the cuts to video sequences and the graphics of tanks and military parades to the pros in the retail gaming industry and they could’ve used video footage from their own documentaries, but they chose to meet their competition on the battlefield where they are weakest and, sadly, it shows.   I think the lesson here for us is to remember our strengths in simulation design – we will never have the resources to design the twenty thousand person simulations that the military can use for war games, but we can use our knowledge, training, expertise and interest in the subject to deliver engaging simulations that are fun and interesting to participants.  Of course, maybe our colleagues that run simulations for the government have a different perspective on this?

Playtesting

Lastly, there is no acknowledgement of any playtesting in the 120 page player’s guide, the resistopedia or anywhere in game and I think the simulation plays like one that wasn’t seriously playtested.  I can’t imagine a non-gamer interested in learning about non-violent resistance loading up AFMP and spending more than fifteen minutes trying to figure out the interface, trying to understand what they are doing and what is going on and trudging through the tedium of the simulation to get to any of the nuggets of learning there (and there are many).  I also can’t imagine a gamer playing with the interface and not coming up with pages of suggestions for the developers.  If there were playtesters, they may have been too close to the development process to really be critical.  If there were no playtesters the designers did themselves and the subject matter a disservice, which is a shame.  The lesson is that no matter how realistic or interesting you think your simulation is, it will always benefit from outside, objective critiques, especially from folks that might comprise the target audience – set them down and ask them, honestly, what they think about your simulation.

I wish AFMP was a better simulation, it has a lot of potential to teach about nonviolent resistance and political dynamics in authoritarian settings.  The producer’s note says that it is the beginning of a dialogue, and if that is the case, I urge the designers to spend a lot more time thinking about what they want people to learn from their simulation and how the simulation can best be designed to teach those lessons.  Maybe AFMP2 can deliver where AFMP has fallen short.

when simulations go bad…

Hat-tip to Lukas Neville for pointing out this piece:

Simulations May Be Causing Real Trouble

Computer simulations have introduced some strange problems into reality

Chronicle of Higher Education, 13 March 2009

Sherry Turkle, the noted professor of the social studies of science and technology at the Massachusetts Institute of Technology, has a new book, Simulation and Its Discontents (MIT Press). In it she tracks difficulties that arise when simulation—from virtual-reality chambers for nuclear-weapon testing to computer programs for architectural design—becomes integral in our daily and professional lives. Ms. Turkle elaborated on some of these in a conversation with The Chronicle.

A: There’s a generation that is growing up with the computer as an appliance, and they truly have no understanding of how it works. In my book, I tell the story of a girl who was a power player of the game Sim City. She talked to me about her “David Letterman Top Ten Rules of Sim City,” and rule number 6 was “raising taxes leads to riots” because when she did that, that happened in the game. She didn’t understand that if I had programmed that computer, raising taxes would’ve led to more social services and greater social harmony. She was drawing a set of conclusions about how the world worked based on the simulation. The trouble with that was not that she was using the simulation, but that the simulation wasn’t transparent to her.

Q: So where do we go from here?

A: I want to be part of a process of reconsideration about transparency, and to put some breaks on the ways in which we’re seduced by simulation. Yes, it’s gorgeous. Yes, it’s perfect. But you know, it could just not be true.—David Shieh

Fascinating stuff—I’ll order the book, and report back.

simulations and intelligence analysis

I recently came across a 2007 paper by Warren Fishbein and Gregory Treverton on Rethinking “Alternative Analysis” to Address Transnational Threats, published by the CIA’s The Sherman Kent Center for Intelligence Analysis. The paper itself is a based on a longer series of workshops and associated reports published by RAND (Enhancing Warning for Transnational Threats: Workshop Reports).

In the paper, the authors argue (emphasis added):

One of the greatest challenges of any “alternative” effort is to effectively communicate the message to those who occupy decision-making roles. Decision-makers are buried by both information and tasks. Motivating them to spend time reading sophisticated analysis in general, let alone analysis that queries existing analytic lines, is a considerable challenge. Several presenters at the workshops who had been senior officials in the terrorism area stressed the extent to which information overload had grown in the post-September 11 environment. Moreover, in the transnational domain, many potential key consumers are in middle and lower operational levels, or outside the government, and thus have even fewer contexts for understanding intelligence information. And, of course, in order to carry out “alternative sense-making,” alternative points of view would need to be put before this wide array of harried individuals on a fairly regular basis.

One way to accomplish this is to rethink the concept of the intelligence “product.” Intelligence organizations continue to insist upon written prose and formal briefing as the “gold standard” for disseminating information even though adults rarely retain more than ten percent of what they are “told” either orally or in written form. Instead, more experiential, interactive formats, as discussed at the workshops, might better capture the attention and imagination of intended audiences and strengthen retention of insights.

  • Use of web-logs would give consumers—particularly non-senior consumers without formal feedback processes—the opportunity to tap in from time to time on debates within the analytic community and to pose questions themselves. 
  • RapiSims—rapid simulations enabled by increasingly sophisticated spreadsheet-based programs—would allow consumers to manipulate variables to generate alternative outcomes. Decision-makers could quickly and easily explore a range of possibilities in a way that is more likely to be retained than if presented in a long and dry formal tome. 
  • Half day “gaming” sessions—intentionally kept brief to allow even the most harried to participate on occasion—could help decision-makers experience, at a minimum, the uncertainties surrounding an issue.

I’m a little dubious about RapiSims, to be frank. So much depends on the assumptions embedded (and therefore hidden) within the underlying computational algorithms that it only seems useful to me in cases where the relationship between quantifiable data is itself fairly widely accepted and uncontroversial.

I can see value in the use of well-structured simulations/gaming in exploring and projecting potential developments, and the response of actors to those developments. However, most simulation structures (unlike most well-written intelligence products) only allow one to easily pursue a single linked path of causality, and not examine a series of possible outcomes and the associated drivers and decision-trees. In terms of examining multiple possible paths and their implications, it might well be that red cell analysis, or even a well-moderated open bear-pit type discussion among analysts and policy folks, is a better investment of time and effort. I also wouldn’t underestimate the amount of give-and-take that goes into a good verbal intel briefing, especially when the policymaker on the receiving end knows how to ask hard questions of the briefer.

As noted in my previous posting below, I think that simulations can be a very useful way of engaging current and potential policymakers under certain circumstances. However, I’m wary of treating it all with a sort of faddism, or of denigrating the very considerable cost-effectiveness of other methods.

The same applies outside of the intelligence community context, in training and education settings too. It is important that we ask, when we devote substantial time and effort to a simulation, whether it is worth the opportunity cost of time that would have otherwise be devoted to lectures, seminar discussions, written assignments, or other more traditional methods.

It often is worth it—indeed, that’s the very reason for this website. It isn’t always, however.

(more) wisdom at the Horse of Peas

Tim Wilkie, as usual, has some insightful comments on the practical contributions and limitations of simulations and gaming at his blog, A Horse of Peas. Specifically, he’s commenting on a WSJ op ed by Eliot Cohen on pundits and policy-making that was first flagged over at Opposed Systems Design, and to which he’s added some thoughts. (You’ve got to love the net for making this sort of virtual conversation possible.)

Tim comments:

If a game “clarifies problems or solutions that the insiders have only vaguely or incompletely considered” or provides the opportunity for outside commentators to develop the empathy with decision-makers that Cohen describes, that could be a valuable contribution.

I couldn’t agree more. As I’ve mentioned on the blog before, I was involved in a  simulation of Palestinian refugee negotiations last summer with Chatham House (UK) which involved a range of participants: academic subject matter experts, current and former diplomats, mid-level negotiation advisors, and former senior negotiatiors and officials. The combination proved especially useful: officials and negotiators found themselves presented with technical challenges they hadn’t considered before, while the academic specialists (who tend to focus on best practices) confronted how theoretically ideal outcomes might be constrained by the give-and-take of negotiations. We were also able to put refugees and refugee advocates into the mix too, providing insights and feedback that was lacking from the actual refugee negotiations of 2000-01 at Camp David and Taba, and which were also missing from the most recent Annapolis round of negotiations in 2007-08.

There are some serious challenges with this kind of mix. Senior officials, even former ones, have little time to spare, especially for multi-day simulations. There are potential status issues when you mix retired senior folks and recent PhD graduates. The necessary abstractions from reality (for reasons of “playability”) can both bias the results and devalue it in the eyes of participants. Political sensitivities can make it difficult to get officials and ex-officials from warring sides in the same room. Finally, some folks will simply see the simulation as a rather silly game. We had one participant leave for political reasons, one carefully check that our Hamas players weren’t actual Hamas cadres before boarding their flight to the UK, and a third who decried it all as a “summer camp” and left.

Was it useful? Will it have an effect? It is hard to say—I’m a completely biased observer, since I ran the simulation. However, the UK Foreign Office clearly thought it was valuable enough to ask Chatham House to hold a follow-up workshop on the challenges of implementing a refugee deal a few months later, and feedback from participants has been good.

Review: Tessman, International Relations In Action

Review of: Brock F. Tessman, International Relations in Action: A World Politics Simulation (Boulder: Lynne Rienner Publishers, 2007). 138pp. USD$16.95 pb.

iriaInternational Relations in Action offers all of the rules, background materials, and scenarios required to conduct an international relations simulation in the classroom. The simulation setting is the fictitious world of Politica and its eight major powers:  Paxony, Refugia, Emerjent, Tundristan, Industrael, Islandia, Petropol, and Minerite. These countries vary in terms of history, goals and grievances, political systems (various democracies, a military junta, a declining authoritarian great power, and a theocracy), economy (from industrialized, newly industrializing, and primary resource extraction) and the particular resources that they control. Within each country, students are assigned different roles (chief decision-maker, diplomat, economic advisor, intelligence officer, and opposition leader), each with sometimes slightly different interests.

The twelve scenarios provided in the book include military security issues and alliance politics, ethnic tensions, trade and other economic issues, multilateral institution-building, and even the environmental challenge of global warming. Each can be played as a one-off simulation, or one after the other as a linked series, or all of the scenarios run at once over an extended period. As the author notes, the latter would result in a much richer classroom experience, forcing actors to prioritize interests and enabling issue-linkage and trade-offs, rather than allowing all national resources to be devoted to a single international problem at a time.

The book contains some additional guidance on how long simulations might take (typically, up to one class/week per scenario, and a full term for all twelve), and how role assignments might be made in classes of varying sizes (from 8 to 48 students). Within each scenario chapter there are suggestions to students as to additional academic readings for the general topic being explored. The simulation is aimed at introductory college-level courses in international relations, although it might have some utility in senior high school classes too.

The provision of all materials required to run the simulation in a single, compact , and reasonably-priced booklet has much to commend it. The background materials, scenarios, and rules are all very well explained. The simulation mechanics are simple and clear. Some additional materials are also available on the author’s website, although these could be further developed.

A core of the simulation is the allocation of resources or “factors.” Each turn, teams are asked to decide on how they will allocate these between military strength (“guns”), economic growth (“butter”), or specific actions. Resources used for “guns” may be used for both offensive and defensive military actions. Resources used for “butter” have an impact on economic growth and the availability of resources in subsequent turns. Resources allocated for actions may either be provided to other players in the form of foreign aid, or allocated for specific activities. There is some inherent flexibility in this latter use of resources that enables both players to think outside of the strict constraints of the rules, and allows instructors to easily modify scenarios. Moreover, many scenarios have specific targets that require the allocation of factors to action, for example by expending resources to build a dam, reduce pollution, or prevent a  global epidemic.

There is also a rudimentary trade system built into the simulation, with each state allocated different amounts of three essential trade goods (oil, fish, and minerals). States that fail to secure adequate supplies of each resource suffer a trade penalty to their economes in the subsequent turn.

Bureaucratic politics and the domestic sources of politics are reflected through the assignment of different roles within each country, as well as the requirement that decisions taken by democracies require the support of a majority of team members.

The scoring system used for each scenario guarantees that different countries, and even different actors within a country, confront different pay-off structures. Thus, the simulation enables both conflict and cooperation based on a variety of mixed-sum arrays of interest.

The global political and economic system outlined in International Relations in Action is obviously a highly abstracted one. Without having played through any of the simulations—and hence based solely on a reading of the rules—it would seem that there might be problems with the frequency of international armed conflict in the simulation, despite mechanics that allow defensive alliances, create opportunity costs to buying guns-over-butter, and provide the defender with a 2-to-1 comparative advantage. State-to-state war is very rare in contemporary international relations, with most conflicts being of the intra-state variety. In the simulation, however, there appears to be little to constrain leaders from threatening military intervention over such “low politics” issues as the establishment of an international criminal court, currency crises, or trade relations.

Another area of abstraction that could be problematic is the system of resource allocation . This presumes an almost Stalinist system of centralized economic control, with actors able to massively shift their sectoral allocations from turn to turn. The trade system similarly implies a much great involvement of state officials in trade decisions. In Politica, it is state officials that decide how many fish will be sold and oil purchased and from whom, rather than this being a process of largely capitalist exchange by firms influenced by tariffs and the international trade regime. Economic embargo is much more easily wielded as a weapon of diplomatic influence in Politica than it is in the real world.

Overall, therefore, one weakness of the simulation is that in attributes far too much leverage and latitude for action to states, and fails to fully reflect the limited incrementalism that is often the substance of global politics. If an instructor is not careful, a simulation risks look more like RISK, and less like international relations. There is also the opportunity cost of simulating to be considered: every hour spent in classroom simulation is an hour lost to lecturing. This is not a reason to not use simulations, but it is to suggest that the learning objectives need to be carefully weighed.

Given this, very much depends on how an instructor uses this simulation. As previously suggested in PaxSims, “unrealistic” simulations can serve a very useful purpose too if instructors are careful to brief or debrief participants as to how, where, and why the simulation diverges from actual politics. Were there to be a second edition of this book, it might usefully contain an additional page or two for each simulation describing the manner in which actual international practice deviates from the simulated variety, and why that is so.

If integrated into curriculum in a way that assures that students learn the right lessons, and not the wrong ones, International Relations in Action could be a quite useful pedagogical tool. It is an adaptable system, with instructors easily able to adapt scenarios or generate new ones. It might also be a useful resource for instructors considering how they might develop their own simulations using similar kinds of mechanics.

evaluating learning from simulations

Don asked in comments on an earlier post about how we evaluate our learning.  Specifically, for Carana, the tools for assessing before the simulation and what assessments are done after the evaluation.

Carana is used as part of our four day course for Bank staff on working in conflict-affected and fragile states.  At the beginning of the course, we have participants self-report on their level of expertise in the field through a pre-evaluation.  At the end of the course, we have participants respond in another evaluation.  Questions in the second evaluation are rephrased, but can be mapped to the level of expertise self-reporting from the first evaluation.  We also give participants an opportunity to evaluate Carana, both in writing and in a group discussion where we do reveals on the simulation and discuss the activity.

Based on the two evaluations above, we have some evidence that participants learn from the course.  Of course, the evaluations are self-reported and they are on the entire course, so this evidence may be partially subjective and it is difficult to separate out how much of this progress is attributable to the simulation.  We’ve run the course four times now and every time has been with the simulation, so we don’t have a control group to determine the treatment effect of using the simulation.

That being said, we benefit from a very large learning group at the World Bank with lots of data points for other courses.  In addition to the evaluations above, we’ve had learning experts observe every one of our courses and provide feedback on what kind of learning is being done and how we could improve it.

In June we’ll be finishing a tracer study on course participants where we will follow up with all past course participants and ask them what they learned and what they have used.  Again, admittedly, self-reported.

Lastly, there is a sort of evaluation built into the simulation.  Once the participants have put together their plan for post-conflict recovery for Carana, they go home for the night and the next morning they are rewarded or chastened for their choices and they have the opportunity to respond by adapting their recovery plan.

That answers a few questions about our process in our simulation and course.  Our hands are tied just a bit since we are teaching experienced staff and professionals who won’t tolerate a pop-quiz or a final, like some other simulationists we know (ahem, Rex).  But maybe there are better ways to determine whether or not we are actually teaching with these simulations – what other ways do people evaluate the performance of these simulations in teaching?

%d bloggers like this: