PAXsims

Conflict simulation, peacebuilding, and development

Monthly Archives: October 2010

Game theory, role-playing, and forecasting

Things are rather busy at the moment with your PaxSims editors, and we haven’t had a chance to post much in the way of simulation evaluations, book reviews, or even news as of late. So that our regular readers don’t get bored, therefore, I thought I would instead flag an interesting article that appeared a few years ago in the International Journal of Forecasting 18, 3 (July-September 2002) devoted to forecasting and conflict situations. In it, Kesten Green describes an experimental assessment of the relative predictive effectiveness of unaided judgement, role-playing simulation, and game theory:

Can game theory aid in forecasting the decision making of parties in a conflict? A review of the literature revealed diverse opinions but no empirical evidence on this question. When put to the test, game theorists’ predictions were more accurate than those from unaided judgement but not as accurate as role-play forecasts. Twenty-one game theorists made 99 forecasts of decisions for six conflict situations. The same situations were described to 290 research participants, who made 207 forecasts using unaided judgement, and to 933 participants, who made 158 forecasts in active role-playing. Averaged across the six situations, 37 percent of the game theorists’ forecasts, 28 percent of the unaided-judgement forecasts, and 64 percent of the role-play forecasts were correct.

You’ll find the full article here. They are interesting findings indeed, and you’ll find much discussion of them by other contributors in the journal (you’ll require an institutional subscription to access it all, however). In particular, George Wright, commenting on Green’s findings, suggests that the immersive and iterative dynamic in role-playing tends to bring out a range of experiential and prior knowledge. It also tends to provide the kind of feedback that improves forecasting accuracy.

The poor showing of unaided judgment might improve, of course, if forecasters were making predictions within their own areas of expertise. However, as Philip Tetlock showed in his 2005 book Expert Political Judgment, experts have a remarkably poor record of accuracy in their predictions—not much better, in fact, than dart-throwing (or wheel-spinning) monkeys.

Tetlock first discusses arguments about whether the world is too complex for people to find the tools to understand political phenomena, let alone predict the future. He evaluates predictions from experts in different fields, comparing them to predictions by well-informed laity or those based on simple extrapolation from current trends. He goes on to analyze which styles of thinking are more successful in forecasting. Classifying thinking styles using Isaiah Berlin’s prototypes of the fox and the hedgehog, Tetlock contends that the fox–the thinker who knows many little things, draws from an eclectic array of traditions, and is better able to improvise in response to changing events–is more successful in predicting the future than the hedgehog, who knows one big thing, toils devotedly within one tradition, and imposes formulaic solutions on ill-defined problems. He notes a perversely inverse relationship between the best scientific indicators of good judgement and the qualities that the media most prizes in pundits–the single-minded determination required to prevail in ideological combat.

Having just spent last summer reviewing the predictive accuracy of one particular group of forecasters—intelligence analysts—it turns out that this group, at least, shows better discrimination and calibration than much of the “expert political judgement” Tetlock examines. Part of this might be the thinking styles that the intelligence community favours. Part of it too, however, is probably the collegial process of producing assessments, as well as the iterative process of refining them.

Bringing this all back to simulation design, it suggests the importance of maximizing feedback, interaction, and iteration in game mechanics. From a forecasting and training perspective alike, a “busy” simulation with a range of interaction experiences may be likely to produce more realistic or accurate results than one which reduces its players to a limited number of rigid turns of strategic decision-making (of the Decision A/Result A1/Decision B/Result B2/Game Over variety).

On a different simulation-related note, the poor performance of game theorists in Green’s experiment may also hold out a cautionary note about our ability to use such models in computational simulation (although in fairness Green is as much testing game theorists as game theory—which may well be quite different things).

Kesten Green also has a follow-up paper on the value of “role thinking” that simulation designers will also find useful. You can find a pre-publication copy of it here.

Carana and peacekeeping training

The Daily Nation (Kenya) recent featured a story on peacekeeping training in Africa, using the Carana simulation developed by the Pearson Peacekeeping Centre:

26 African countries have trained troops on peacekeeping

13 October 13 2010 at 20:43The Tactical Operations Staff Course, taught over the past three weeks at the Humanitarian Peace Support School in Embakasi, was designed by Canada’s Pearson Peacekeeping Centre.

This think tank is named for Canadian statesman Lester B. Pearson, who won the Nobel Peace Prize after fathering the first ever UN peacekeeping mission in 1957.

In 2002, the leaders of the G8 — a group bringing together the world’s eight leading economies — met in Kannanaskis, Canada.

Here, Canada pledged to do something more to help build African capacity for peacekeeping, and some months later this course was designed. Since then, Canada has trained 452 military officers from 26 African nations using the Carana scenario.

Major Alan Woolley, the burly and moustached chief instructor, said the scenario was designed for any sort of mission — be it a peacekeeping, development, or police mission. While it is inspired by the histories of places like Sierra Leone and Sudan, he said it was a wise decision to create a fictional country.

“With a fictional country (Carana), no one can get offended,” Maj Woolley told the Nation. “If you use real terrain there’s always the chance someone can get offended at the way a country is represented.”

The Canadian major described the simulation, which is the culmination of the course, as “three bad days in a mission.” Crammed into this short time are all the things that can and do go wrong on actual peacekeeping missions; everything from bandit raids and landmine strikes to humanitarian emergencies.

To enhance the realism, and teach another important skill set, mock news reporters come knocking on the doors of the Command Centre. They slyly take pictures of classified maps, grill senior officers with tough questions, and write stories quoting the students for the ‘press clippings’ released each morning.

“The media aspect has grown to become really a major facet, and the students see it as extremely important,” said Maj Woolley. “If they make a mistake here, nobody dies, nobody gets their career ended. And if they make a mistake here, they will remember it.”

You’ll find the full news report here.

strategic gaming positions at NDU

The Center for Applied Strategic Learning at National Defense University is currently advertising two positions.

The first is for an Associate Research Fellow:

Serves as a one of two Division Directors at the Center for Applied Strategic Learning (CASL), National Defense University (NDU). Under the direction of the CASL leadership, the incumbent will be responsible for all aspects of activities within his/her division. Division duties focus primarily on designing, developing, and executing scenarios/exercises in support of the Office of the Vice President for Research and Applied Learning and the colleges of the National Defense University (NDU). Incumbent will also be responsible for ensuring subordinates design, develop, and facilitate national security exercises in support of Members of Congress, State Governors, the Office of the Secretary of Defense (OSD), the Joint Staff, the Geographic Combatant Commands, the Department of State, the Department of Homeland Security, and other activities. The incumbent is responsible for assisting with experiential education and training programs for the academic, non-governmental, and interagency policy communities at the strategic and operational levels. The incumbent is specifically responsible for all supervision, contracting, budgeting, branding, and other outreach activities of the Strategic Policy Integration Division and its associated programs.

The second is for an Instructor:

Serves as an Instructor for a Division Director at the Center for Applied Strategic Learning (CASL),National Defense University (NDU). Under the direction of a CASL Division Director, the incumbent will assist with designing, developing, and executing scenario creation and exercise initiatives in support of educational, training, and outreach programs of the Office of the Vice President for Research and Applied Learning and the colleges of the National Defense University (NDU). Incumbent may also assist in designing, developing, and facilitating national security exercises for Members of Congress, State Governors, the Office of the Secretary of Defense (OSD), the Joint Staff, the Geographic Combatant Commands, the Department of State, the Department of Homeland Security, and other activities. The incumbent is also responsible for assisting with experiential learning programs for the academic, non-governmental, and interagency policy communities at the strategic and operational levels.

Those interested in applying will find full details on the USAJOBS website at the links above. The deadlines for application are 13 and 18 October.

%d bloggers like this: