Mr. Punch’s Victorian Era: An illustrated chronicle of the reign of Her Majesty the Queen, Volume 2 (1888), via Google eBooks.
Two months ago a discussion began via PAXsims on simulation design when Natasha Gill (TRACK4) posted a guest contribution on “Happy endings, doomsday prophesies, and the perils of think tank simulations,” responding in part to earlier comments I had made on a Syria simulation at the Brookings Institution. Devin Ellis (ICONS Project) then offered some reflections on Natash’a’s points and the “Dance of the Simulation Designer” (to which Natasha, in return responded).
Today we offer round three of this valuable discussion, as Devin addresses some issues and questions raised by Natasha and myself, and offers more thoughts on policy simulations and high-level participants.
* * *
Policy Simulations and High-Level Participants – Continued
Devin Ellis, ICONS Project
Natasha Gill wrote a very thoughtful response to my spur-of-the-moment comments on her initial post, and I promised Rex that I would continue the dialogue on high-level participants (I think I’m going to give up and go for an acronym – HLP from here on in). I have been distracted by my day job for the past month, but I am hoping to answer Gill’s question(s) and further clarify some of the points in our discussion. I am doing a lot of think tank work right now, watching much more of it as an invited guest, and I have kept this discussion firmly at the forefront of my mind. Assessing what I thought was valuable and not valuable about what I’ve witnessed the past few weeks sharpened my thinking on this subject.
My question to Ellis is this: he writes that the purpose of the exercise is to “explore possible policy reactions to a crisis” but in the same sentence admits that “no one…believes the purpose of a well designed and run wargame is to predict the ‘real future’ in a complex policy environment.
I think this means that although facilitators and participants are well aware that the details of the future can’t be known, the responses of various players to a crisis might be generally predictable in a simulation, in such a way as to be informative or useful for policy makers.
But upon reflection, if this is what Ellis meant, I’m not sure it makes sense to me. If a simulation can’t predict the future, then how much is it really telling you about possible ‘policy reactions’? And how much of what it does tell you about these actually useful (rather than merely interesting) to real policy makers?”
Gill is almost right about what I meant. I did not originally frame it this way, but I stand by the idea that a think tank simulation can tell you a great deal about possible policy reactions without having to be predictive of the one specific future – indeed the well run policy planning simulation should do exactly that.
She also, however, throws in the existential dilemma for all of us in this business: how much of what you get out of this is actually useful to policy makers vs. merely interesting? My answer to this two part question is that I believe there are a lot of circumstances under which think tanks should not do policy simulations. But if they do them it should be with HLPs, and they should not attempt to get them out of their own background and experiences. I argue that this very formula can be what gets your desired outcome: a set of possible courses of action which are useful for planning.
My answer raises two questions: 1) What does a ‘good’ think tank simulation need to deliver? 2) Why do HLPs help get you there? Here we go…
It’s a Bird, It’s a Plane, It’s the Saban Center!
The think tank simulation is a different beast from an in-house policy planning exercise for an agency or office. It is also very different from a simulation used for training. Usually the objective is not to give the participants in the exercise new insight into the issue – if they get it, bully for them, but that’s secondary. The objective is to influence current and possibly future policy makers who are almost never, themselves, participants.
In Stephen Downes-Martin’s talk this year at Connections, he emphasized one of his key questions to the client: “When do you rotate out of here?” He asks that (and we all should!) because with an in-house program you want to make sure that client is served by the time frame of your exercise and the recommendations that emerge from it. For a think tank, your recommendations often go public, and even when they don’t your client is usually an SME, not an SES/O-8-O-9er. This has implications.
As a think tank, you need to be able to defend your conclusions to:
- The current decider
- The next decider
- Their numerous opponents and critics.
What does this mean? The content of the simulation needs to be either non-partisan or bi-partisan in significance; it’s best not to assume any specific political agenda or personality for the actors in major democratic powers (the U.S.); the more your exercise focuses on core thematic and structural issues, the longer its shelf life and the wider its acceptance. In short, you benefit from less – not more – situational and personality specificity.
My first question to the think tank interested in running a crisis simulation is: “what do you want this exercise to tell you about policy planning options?” If the answer is “I want one exercise to reveal the hidden truth of how we are going to untangle the mess in Syria” then I will try and get them to re-think their agenda. If the question is: “I want to better understand what courses of development might face us in the event that US decision makers are confronted by scenario X,” then a good simulation might just be for you!
Admiral, if you could put down the putter for just one moment please…
On doing what we do, I could not say it better than Peter Perla:
Wargames, on the other hand, focus precisely on human behavior, particularly on human decision making. The learning that comes from wargames comes both from the experience of making decisions (playing) and from the process of understanding why those decisions are made (game analysis)… Wargames do not do very well at producing quantitative measures because they are often little more than a single realization of a complex stochastic process. Instead, the value of a wargame lies in qualitative assessments of why decisions are made.
The think tank’s sim has to work for multiple real world audiences, so the specificity of things like political leanings and psychology that would make role sheets for a simulation examining a very specific set of parameters so vital are instead counterproductive. The think tank’s sim is looking for a very particular type of outcome – it’s an exercise in identifying problems and options which can be generalized.
Rather than trying to break the HLPs out of their shells, I am in fact looking for those instinctive, experiential elements they bring. Their professional background – if not their political party or personal feelings – is usually representative of their near peers. When you analyze the thought processes they went through, how they decided what to do, and what they did, you can draw inferences about how a real world administration would likely be advised of its options. Doing so can show you both areas where they are presenting good ideas, and also areas where you can point to problems with myopic thinking, lack of creativity, and constraints on policy choices which should be addressed BEFORE the crisis happens. If a good think tank sim can do these things, it is well worth the time spent on it.
Go Ahead, Kill the Messenger
It will come as no surprise, given the proceeding, that I don’t do role sheets for HLPs in a think tank simulation. As I said, what I’m looking for are their real life assumptions, reactions, and ingrained knowledge. It would be total hubris for me to assume I know more about the gritty details of each of their specialized realms than they do, and the last thing I want is to impose a filter. Also, from a purely practical standpoint, most HLPs are not going to read all their materials if you give them more than half a dozen pages. I love the ones who will, but I plan for those who won’t.
What does this mean? Your scenario has to be superb – it has to withstand the critical glare of your participants, and it has to drive the key elements you want them to focus on in the simulation. Part of my problem with many think tank simulations and wargames is that scenarios are often not rigorously enough vetted for both realism AND their ability to elicit the responses you are most interested in examining in your exercise. Additionally, I am in 100% agreement with Gill on the deleteriousness of throwing endless, seemingly unconnected injects at your participants to stir things up (“let’s see how they’ll react to this! And now this! Etc”). Stick to your story.
Rex raised a good point: for the think-tankers to sell their conclusions, it helps to have those big names attached. Many good ideas only make the leap from academia to policy after a big name policy shop has written them up with the implicit endorsement of beltway heavy hitters. Of course it cuts both ways, and plenty of trash has been given marketing sparkle by former cabinet members. Garbage in, garbage out – at all levels. I think it’s up to the savvy consumer to look at the simulation product much the same as he or she would any other think tank product.