PAXsims

Conflict simulation, peacebuilding, and development

evaluating learning from simulations

Don asked in comments on an earlier post about how we evaluate our learning.  Specifically, for Carana, the tools for assessing before the simulation and what assessments are done after the evaluation.

Carana is used as part of our four day course for Bank staff on working in conflict-affected and fragile states.  At the beginning of the course, we have participants self-report on their level of expertise in the field through a pre-evaluation.  At the end of the course, we have participants respond in another evaluation.  Questions in the second evaluation are rephrased, but can be mapped to the level of expertise self-reporting from the first evaluation.  We also give participants an opportunity to evaluate Carana, both in writing and in a group discussion where we do reveals on the simulation and discuss the activity.

Based on the two evaluations above, we have some evidence that participants learn from the course.  Of course, the evaluations are self-reported and they are on the entire course, so this evidence may be partially subjective and it is difficult to separate out how much of this progress is attributable to the simulation.  We’ve run the course four times now and every time has been with the simulation, so we don’t have a control group to determine the treatment effect of using the simulation.

That being said, we benefit from a very large learning group at the World Bank with lots of data points for other courses.  In addition to the evaluations above, we’ve had learning experts observe every one of our courses and provide feedback on what kind of learning is being done and how we could improve it.

In June we’ll be finishing a tracer study on course participants where we will follow up with all past course participants and ask them what they learned and what they have used.  Again, admittedly, self-reported.

Lastly, there is a sort of evaluation built into the simulation.  Once the participants have put together their plan for post-conflict recovery for Carana, they go home for the night and the next morning they are rewarded or chastened for their choices and they have the opportunity to respond by adapting their recovery plan.

That answers a few questions about our process in our simulation and course.  Our hands are tied just a bit since we are teaching experienced staff and professionals who won’t tolerate a pop-quiz or a final, like some other simulationists we know (ahem, Rex).  But maybe there are better ways to determine whether or not we are actually teaching with these simulations – what other ways do people evaluate the performance of these simulations in teaching?

3 responses to “evaluating learning from simulations

  1. Gary Milante 09/03/2009 at 7:56 pm

    Sure thing, Lukas:

    First, these are the questions we ask at outset of the course:

    Please tell us how much you currently know on each specific topic or concept [The questionnaire includes a five point scale for respondents to self-evaluate]

    1) diagnostics of fragile and conflict-affected situations (FCS), and characteristics that define them
    2) reasons fragility and conflict require a differentiated response
    3) tools to determine appropriate response in various situations of fragility and conflict
    4) custom operational and strategic choices available for FCS
    5) structure and roles of the ”international architecture” of fragility and conflict
    6) policies, procedures, and program approaches of the World Bank
    7) diverse interests of stakeholders outside the World Bank
    8) how economic development agencies link to other international partners
    Then the participants are asked the following questions in their post-course evaluation:

    How do you rate your *new* knowledge level on each of these specific topics/concepts? [Same 5 point scale, emphasis added]
    1) diagnostics of fragile and conflict-affected situations (FCS), and characteristics that define them
    2) reasons fragility and conflict require a differentiated response
    3) tools to determine appropriate response in various situations of fragility and conflict
    4) custom operational and strategic choices available for FCS
    5) roles of partners in ”architecture” of response to fragility and conflict
    6) World Bank policies, procedures, and program approaches in FCS
    7) diverse interests of stakeholders outside the World Bank
    8) linking to areas beyond the Bank’s core competence

    These evaluations are anonymous, however, they are coded so that we can compare the pre-course evaluations to post-course results.

    As mentioned before, the participants are professionals, often already working in these environments and often senior level staff, so the evaluation is about as much as we are able to extract from them – no pop quizzes, mid-terms or finals. This is sufficient for our reporting purposes internally at the Bank, but I am not convinced it tells us as much as we could learn about what participants are learning.

  2. Rex Brynen 09/03/2009 at 12:56 am

    Quite by coincidence, I received this message today from a former POLI 450 student, who now works at the UN: “I have something I’ve been meaning to say for a while now: oh my god the simulation is so close to reality.”

    That’s the sort of evaluation I like!

    In all seriousness, though, I wonder what more critical sim evaluations might be out there, which former participants are loathe to express for reasons of politeness, friendship, etc. If you’re reading, feel free to post them…

  3. Lukas Neville 08/03/2009 at 6:16 am

    Can you elaborate on what kinds of measures/questions you give your students in the pre- and post-simulation “self-report on their level of expertise”?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: