Assessing simulation effectiveness in Brynania

Back in 20120, Michael King and I asked students in my POLI 450 (Peacebuilding) course about the learning effectiveness of the Brynania civil war simulation. I never did get around to do much with the results, so I have posted them below. I’ve highlighted the largest learning effects in green, and the smallest in red.

As you can see, students reported learning in every single area we asked about. Learning was highest in those areas related to process and operational constraints (that is, the bureaucratic politics,  “friction” and “fog of war/peace”)—precisely those areas that it is hardest to explore through conventional lectures and readings. Students also overwhelmingly reported that a week of simulation was more valuable to them than a week of readings or lectures. Very importantly, however, they saw the experiential learning of the simulation as being complimentary with the more conventionally-delivered course material, synergistically improving their understanding of prior course readings. As I noted in a previous post, there were no statistically significant gender differences in learning.

Self-reported learning effects were smallest in those areas that related to the development of personal skills like leadership, time management, or empathy (although even here there was agreement that the simulation had delivered some learning). In addition to data above, we had also asked students to self-assess their skills in such areas both pre- and post-simulation. That part of the study revealed no substantial, statistically-significant changes.

In interpreting the data, some caveats are in order:

• First, this is self-reported learning, not learning that has been assessed by some more objective measure like exam results. There is some evidence that self-reported measures may overstated the effectiveness of simulation-based methods, especially when these are assessed in the immediate aftermath of a simulation that students have enjoyed.
• Second, these findings hold true for this simulation and these (highly-motivated) students. While they do provide further evidence of the general value of simulation methods, other simulations and audiences could well report significantly different effects.
• Third, simulation-based learning cannot really be separated from the broader curriculum in which it is embedded. We know, for example, that the simulation debrief plays an essential role in good learning outcomes. In addition, however, learning outcomes are likely enhanced when the simulation and non-simulation material are designed to compliment and synergize each other.

A final thing we did in the questionnaire was to assess whether simulation participation had any measurable effect of student career goals. Here there were suggestions of a very slight decline in interest in legal or military careers, but generally no significant effect.