PAXsims

Conflict simulation, peacebuilding, and development

Ethics compliance and wargaming

20 Comments Posted by on 24/10/2023

The following item was written for PAXsims by Ivanka Barzashka. Dr. Barzashka is the CEO and co-founder of Strand Analytica, a British-American tech startup dedicated to powering the emerging science of wargaming through technology for national security and defence applications. She was a founding director of the Wargaming Network at King’s College London, serving as its co-director (2018-2019) with Professor Philip Sabin, director (2019-2020) and managing director (2020-2022) working with Dr David Banks as academic director. 

A recent Freedom of Information Act (FOIA) request revealed a surprising lack of scientific and ethical oversight over analytical wargames, which inform UK defence decisions. According to the MOD, only a single study was submitted for Ministry of Defence Research Ethics Committee (MODREC) review over the past five years.  

This is not a uniquely British problem. The 2023 King’s College London survey of over 140 wargame designers across 19 countries suggests that 80 percent bypassed ethics reviews of their analytical wargames, disregarding norms for research studies involving human participants. 

MOD Ethics Requirements 

The MOD mandates MODREC ethics review for all studies that meet three criteria: (1) are MOD-funded or involve MOD-funded staff or participants, (2) are research defined as “the attempt to derive generalisable or transferable new knowledge to answer or refine relevant questions with scientifically sound methods”, and (3) gather data from human participants. 

According to these criteria, all analytical wargames that are sponsored by the MOD or involve MOD participants should undergo ethics review. Low risk wargaming studies still require the MOD sponsor to submit a MODREC application, but this will trigger a “proportionate review process allows for a rapid (10 working day) turn-around,” as opposed to a full committee review.  

All applications go through a two-step process. First, they undergo a scientific review “to ensure that the research is properly designed and planned” and ensure the “quality research will also be safe, ethical, legal and feasible.” This assessment is done by a Scientific Assessment Committee. If this review is successful, the application proceeds to an ethics review during a full meeting of MODREC, except where there is “minimal risk, minimal burden and minimal intrusion for research participants” and “minimal risk to researchers”. In these circumstances, applications are reviewed “by a sub-committee rather than at a full meeting of MODREC”. 

However, the FOIA response suggests most MOD analytical wargaming studies fail to comply with these guidelines. 

The Need for Ethics Compliance 

Analytical wargaming should adhere to established academic integrity and ethics standards. These principles include mitigating risks, ensuring informed consent and privacy, and promoting participant welfare. Independent oversight provides accountability for these protections. Institutional research ethics committees help assess and address these risks. 

The single 2021 application released under our FOIA request provides a valuable example of MODREC’s rigorous scrutiny of an observational wargaming study. Before granting an approval, the committee asked the researcher to improve their protocols around participant exclusion criteria, the consent process, data usage policies, and risk mitigations. This enhanced protections for the study participants. 

Why Has Wargaming Lacked Oversight?  

For one, authoritative guidelines, like NATO’s wargaming handbook released in September 2023, and the UK’s 2017 variant, do not mention ethics requirements. Additionally, government institutions that sponsor these games often do not mandate formal compliance with research ethics standards. Furthermore, ethical reviews can be time-consuming because they require extensive documentation and lengthy approval processes, which can conflict with the pressing timelines of policy decisions. 

Conclusion 

Strengthening protections for human participants must remain a top priority. As universities and militaries conduct more wargaming research, robust, independent ethics review will be essential to uphold integrity and minimize risks. The current oversight gap highlights the pressing need to reinforce MODREC and expand review of defence wargaming studies.

20231023-foi2023_11650_mod_response_letter-v3_redactedDownload
methodology, simulation and game reviews, simulation and gaming news

20 responses to “Ethics compliance and wargaming”

  1. Timothy Smith 26/10/2023 at 9:32 am

    Ethical professional practice is built into the conduct of reaserch, and specifically wargaming, within and sponsored by DoD and, I presume, MoDUK and Canada’s DoND. Certainly in the sense of human rights, given that (a) the participants are not/not the subjects, and (b) the nature of the activity is essentially egalitarian as well as being inherently uplifting and well-nigh universally popular among participants. Human-factors research is different but there military personnel practices prevail and, in some cases anyway, the subjects are volunteers.

    The real issue is analytic validity, especially in wargames, where, as Stephen, Peter, Ed McGrady, and others have observed, the learning is intense and memorable but is subject to both (a) random bias due to the limited number of trials (n=few) and the nonexistence of any instances (N=0) of the future wars we prepare for, and (b) systematic bias should high-powered players seek to skew results, support desired outcomes, protect programs, etc.

    We do need scientists/methodologists to help ensure analytic validity. We would do better were we to institutionalize/formalize M&S/V&V practices in our wargame management, include some premier philosophers of science, logicians, statisticians/data scientists and military historians in our M&S/wargaming management teams, and task them to evaluate each game for substantive validity. What we don’t need is officious, legalistic bureaucrats intruding from academia loading us down with rules/regs to ensure the rights of serious, professional, adult volunteer participants, many of whom are serving or former military officers.

    ‘If it ain’t broke don’t ‘fix’ it.’ Let us band together to resist this invasive species of red tape, at least until we see participants start complaining about percieved violations of their ‘rights’ at the wargames we run.

  2. Scholar_of_Wargamicus 26/10/2023 at 9:30 am

    Hmm. Much to think about.

  3. Stephen Downes-Martin 26/10/2023 at 8:01 am

    @Ivanka. This conversation has involved five people only, so cannot be used to make claims about the wargaming community. The conversation does not demonstrate a “lack of understanding around ethics requirements for wargames”, it does demonstrate a lack of agreement about external oversight. The only community wide agreement is within the MoD that MODREC is not a priority.

  4. Ivanka Barzashka 26/10/2023 at 4:30 am

    Stephen, I appreciate your perspective, but I this exchange does reveal a lack of understanding around ethics requirements for wargames.

    Looking closely at the full conversation, there seem to be substantive gaps in alignment on this issue within the wargaming community. For instance:

    -Timothy staunchly rejects the applicability of ethics reviews beyond academic contexts, implying a narrow view of human research ethics.
    -Multiple commenters question whether ethics oversight is needed at all for professional wargames.
    -There are requests for examples of unethical practices, suggesting uncertainty around ethics risks.

    At the same time, thoughtful counterpoints are raised – like Rex noting ethical considerations are relevant across all social science or you highlighting existing peer review processes.

    But the range of views expressed and knowledge gaps displayed suggest a lack of consensus within the community around if and how human research ethics apply to analytical wargaming.

    This suggests more dialogue and education could benefit the community to align understanding of ethics oversight for wargaming. There is room for agreement even among principled disagreements on specific oversight approaches.

  5. Stephen Downes-Martin 25/10/2023 at 5:34 pm

    @Ivanka, You state that “this exchange reveals, there remains a lack of understanding regarding the ethics requirements for analytical wargames”.

    I completely disagree. What this exchange reveals is that there is little agreement the MORDEC process is useful. Furthermore the MoD itself is uninterested in mandating it at least as far as wargaming is concerned, so they don’t believe it adds sufficient value to mandate it. It might be informative to know what percentage of analytic studies carried out by the MoD were submitted to MORDEC for review.

    In the organizations I have worked the event (game, meeting whatever) design is scrutinized by leadership and peer designers / analysts (not involved in the event itself) for how well it answers the analytic objectives, whether it violates PII requirements, whether the right people (with diversity of experience in the topics under investigation) have been invited etc. All reviews are carried out by professionals who understand the requirements. Basically a peer and leadership review of the design, followed by a peer and leadership review of the analysis and the report. Normal ego and competitiveness make the reviews comprehensive and not in the least indulgent of unprofessionalism.

    The real danger is the deliberate attempt to manipulate and influence the design, analysis and report (sometimes by selecting the participants) by the stakeholders and their chains of command whose decisions the wargame is meant to inform. An external review before the game goes forward does not address this. The best defense against this is the process, informed by knowledge of deception (self or otherwise), described above.

  6. Ivanka Barzashka 25/10/2023 at 5:00 pm

    @Steven Downes-Martin Could you please elaborate on your comment that “we already have processes in place for analytic and ethical integrity, including peer review”? What are these processes in a UK and US context?

    As this exchange reveals, there remains a lack of understanding regarding the ethics requirements for analytical wargames. Sponsors should mandate ethics compliance, and independent reviews can help ensure this.

    The MODREC process is indeed elaborate. At King’s, for example, there is a much more straightforward procedure for studies with minimal risk — essentially comprising of a checklist, which the researcher self-certifies has been met.

  7. Stephen Downes-Martin 25/10/2023 at 2:13 pm

    The issue is not whether one should or should not pay attention to ethical and analytical integrity, one clearly should and in every wargame I have been involved with in decades of professional wargaming it has. The issue is whether one should pay attention to MODREC. Again, if the UK MoD is so uninterested that it does not mandate compliance, then compliance is unlikely. People follow leadership.

  8. Rex Brynen 25/10/2023 at 1:30 pm

    The problem of “analytic and adjudicative bias conveying false perceptions of the substantive external world” is hardly unique to wargaming—it’s pretty much at the centre of all good social science research, and it would be hard to argue that wargamers pay more attention to it than other researchers working on human decisions-making. Clearly you can do that and conduct research that is informed by ethical considerations at the same time. It’s not that difficult.

    Also, defence departments spend considerable time worrying about the “well-being of the tender souls” in a variety of other contexts, ranging from psychological counselling to range safety. Again, it’s really not hard to do in a wargame design nor does it threaten the validity or timeliness of findings when done properly.

  9. Timothy Smith 25/10/2023 at 12:48 pm

    As discussed in our last go-round on this topic, academic, psychological and medical simulation activities involving students/patients who are the subjects of the research do warrant ethics reviews. Milgram, Asch, etc. — we all got that in Intro to Psych 101 and still get it.

    But that has nothing to do with professional wargaming to support long- or short-term defense/defence planning and operations. There the concern lies not with the well-being of the tender souls simulating the governments/agencies/commands in action but with the potential for analytic and adjudicative bias conveying false perceptions of the substantive external world to the participants, observers and sponsors. Peter Perla and Stephen have written about this at length. ONI’s Sim-Based Analysis & Training (SimBAT) methodology is designed specifically to prevent that kind of systematic bias in findings. It works (quickly enough for wargaming) because it reduces scientific method to a military-style checklist that includes explication of assumptions up front, during the event planning phase.*

    This requires not legalistic bureaucracy but substantive and scientific-methodological experts serving as facilitators empowered to question assumptions and manage event schedules so as to approve movement to the wargame phase only once assumptions have been explicated.

  10. Stephen Downes-Martin 24/10/2023 at 8:05 pm

    Rex pretty much answers the mail. We are adults dealing with serious life and death issues, and we already have processes in place for analytic and ethical integrity, including peer review. Strengthen those processes if need be. The lack of interest by senior leadership in MODREC and MODREC’s failure to make it’s case means it is worthless.

  11. Stephen Downes-Martin 24/10/2023 at 7:59 pm

    So, wargames that do not submit to MODREC for review have to prove they have ethical and analytical integrity, but the claim that MODREC improves the situation more than it costs does not have to justify itself?

  12. Rex Brynen 24/10/2023 at 7:57 pm

    Tim: I understand the point you’re trying to make, but in many fields adult professionals undertaking life-and-death research do have an ethics review process in place—medical research being the most obvious example. Note that Ivanka isn’t discussing wargames to support current operations, but rather longer-term analytical games.

    (Incidentally, we certainly do use systematic psychological profiling of participants in some games, whether as an input to role assignment, or as part of our data collection, or because psychological manipulation is a fundamental part of the research design or objective.)

  13. Timothy Smith 24/10/2023 at 7:39 pm

    Academic research projects, student activities and experiments such as wargames are already covered in academic protections. Adult professionals can be treated as such. No one conducts ‘psychometric profiles’ in wargames. We’re a little busy trying to support national security.

  14. Ivanka Barzashka 24/10/2023 at 6:50 pm

    @Stephen Downes-Martin I appreciate your perspective on this issue. You raise some valid points about the apparent lack of enforcement of MODREC review requirements.

    However, I would argue that just because MODREC oversight has not been sufficiently mandated in the past does not mean it lacks value. Robust ethical review provides an essential layer of accountability that upholds integrity. The fact that most studies have not submitted to MODREC speaks more to lack of compliance than lack of necessity.

    While current professional procedures may uphold some ethical standards, the additional scrutiny of an independent ethics committee can further identify and mitigate risks. Their detailed reviews assess factors like consent processes, protocols, privacy protections, and risk minimization – areas that may be overlooked without formal review.

    The low MODREC submission rate likely owes more to lack of awareness and inconvenient timelines than judgments about the value it provides. The solution should be strengthening enforcement and integrating MODREC earlier into the wargaming process. With proper compliance, MODREC has significant potential to close oversight gaps in analytical wargaming.

    Rather than claim the current system ensures ethical integrity, I would argue expanding MODREC authority and integration could significantly strengthen protections and accountability. Just as academic institutions utilize rigorous ethics review, so too should defense analytical wargaming. With refined compliance and enforcement, MODREC oversight can become an asset rather than an inconvenience. But it requires leadership commitment – the current gaps are likely issues of negligence rather than well-reasoned judgments about MODREC’s lack of necessity or value.

    What are your thoughts on how compliance could be strengthened?

  15. Stephen Downes-Martin 24/10/2023 at 6:38 pm

    If “only a single study was submitted for Ministry of Defence Research Ethics Committee (MODREC) review over the past five years” without the study leaderships being sanctioned then clearly senior MoD leadership is uninterested in MODREC. In which case why would anyone pay attention to MODREC?

    To claim MODREC is necessary for analytic integrity is to claim that a significant percentage of analyses that do not submit to MODREC for review lack analytic integrity, or that there are insufficient professional procedures already in place to ensure the equivalent level of analytic integrity that MODREC claims following its procedures will provide. In either case MODREC has clearly not made its case that it is necessary for ethical and analytic integrity, or that abiding by its procedures is value added.

  16. Ivanka Barzashka 24/10/2023 at 6:23 pm

    @RockyMountainNavy, Here is another ethically-challenged game. The proposed strategic wargame will simulate a fictional conflict between real-world nations, set 5 years in the future. The game will include senior government officials roleplaying their real-world positions and responsibilities. Personal details from their psychometric profiles will covertly inform AI behavior models without consent. The game aims to evaluate deterrence strategies.

    This could raise ethical issues around:
    -Use of personal data without consent
    -Potential psychological or emotional distress from simulated high-stakes crisis
    -Deception or coercion to encourage participation
    -Real-world impact if sensitive findings are leaked
    -Biases encoded in AI models based on profiles

    An ethics review may find concerns with:
    -Privacy infringement and lack of consent on using personal data
    – Insufficient safeguards around potential psychological risks
    – Transparency around data usage and security protocols
    – Unclear delineation between simulation and real decisions
    – Potential real-world implications if findings are misrepresented

    Revisions could include more rigorous consent processes, transparency measures, and protocols to safeguard privacy and mitigate risks. Ethics oversight provides an essential accountability mechanism for strategic wargames involving real-world leaders.

  17. Ivanka Barzashka 24/10/2023 at 6:20 pm

    @RockyMountainNavy, here is an example of potentially unethical practices in a strategic-level wargame:

    The wargame aims to test new national security strategies by simulating a conflict between real-world nations. The organizers recruit student participants to roleplay as national leaders without disclosing the full scope of the simulation. Students are instructed to make decisions they believe will maximize their national interests. However, the scenarios are designed to intentionally escalate tensions and drive toward nuclear war.

    This could raise ethical issues around:

    – Lack of full transparency and informed consent about the nature of the simulation
    – Manipulating student decisions through engineered escalation scenarios
    – Potential psychological distress for students roleplaying nuclear crises
    – Debriefing and support for students after distressing scenarios

    While hypothetical, an ethics review may find concerns with:

    – Being fully upfront about the likelihood of nuclear war outcomes
    – Unduly influencing student decisions through engineered scenarios
    – Protecting student psychological health and welfare
    – Properly debriefing students about the intended goals of the simulation
    – Ensuring fully informed consent, avoiding psychological manipulation, and providing adequate support would help make this kind of strategic simulation more ethically acceptable

  18. RockyMountainNavy 24/10/2023 at 6:03 pm

    Can anyone provide an example of an unethical wargame? As in what would a proposal look like for it to be deemed unethical and not recommended for execution?

  19. Ivanka Barzashka 24/10/2023 at 5:57 pm

    Thank you for your feedback. I respect your perspective and understand the concern about adding another layer of bureaucracy. However, the primary goal here is to ensure the protection of human participants, uphold research integrity, and foster trust in the wargaming process. It’s essential to strike a balance between effective research practices and ensuring ethical standards. While the process might seem cumbersome, its intent is to safeguard participants and the validity of the research. Let’s continue this dialogue to find common ground and improve the wargaming community for everyone. #OpenDialogue #WargamingEthics

  20. Timothy Smith 24/10/2023 at 5:32 pm

    This again. Ludicrous as before. Legalistic bureaucratism run amok. Very important for all wargamers and all wargame managers to band together to resist envelopment by this boa constrictor of useless red tape.

Leave a comment