evaluating learning from simulations

Don asked in comments on an earlier post about how we evaluate our learning.  Specifically, for Carana, the tools for assessing before the simulation and what assessments are done after the evaluation.

Carana is used as part of our four day course for Bank staff on working in conflict-affected and fragile states.  At the beginning of the course, we have participants self-report on their level of expertise in the field through a pre-evaluation.  At the end of the course, we have participants respond in another evaluation.  Questions in the second evaluation are rephrased, but can be mapped to the level of expertise self-reporting from the first evaluation.  We also give participants an opportunity to evaluate Carana, both in writing and in a group discussion where we do reveals on the simulation and discuss the activity.

Based on the two evaluations above, we have some evidence that participants learn from the course.  Of course, the evaluations are self-reported and they are on the entire course, so this evidence may be partially subjective and it is difficult to separate out how much of this progress is attributable to the simulation.  We’ve run the course four times now and every time has been with the simulation, so we don’t have a control group to determine the treatment effect of using the simulation.

That being said, we benefit from a very large learning group at the World Bank with lots of data points for other courses.  In addition to the evaluations above, we’ve had learning experts observe every one of our courses and provide feedback on what kind of learning is being done and how we could improve it.

In June we’ll be finishing a tracer study on course participants where we will follow up with all past course participants and ask them what they learned and what they have used.  Again, admittedly, self-reported.

Lastly, there is a sort of evaluation built into the simulation.  Once the participants have put together their plan for post-conflict recovery for Carana, they go home for the night and the next morning they are rewarded or chastened for their choices and they have the opportunity to respond by adapting their recovery plan.

That answers a few questions about our process in our simulation and course.  Our hands are tied just a bit since we are teaching experienced staff and professionals who won’t tolerate a pop-quiz or a final, like some other simulationists we know (ahem, Rex).  But maybe there are better ways to determine whether or not we are actually teaching with these simulations – what other ways do people evaluate the performance of these simulations in teaching?