The primary purpose of any AAR or SimEx is to identify and capture strengths and challenges in a structured manner and to propose concrete recommendations to improve plans, procedures and systems for emergency preparedness and response [5, 7, 8]. From 2016 to 2019, WHO supported AAR and SimEx in the six WHO regions that covered emerging and re-emerging infectious disease outbreaks, environmental and natural disasters, and societal crises . Analyzing the extracted data from available AAR and SimEx reports produced some notable and clear trends. In this study, we showed AFR being the most common geographic region reporting, the vast majority of AAR format used being the working group format, and the most common type of SimEx used being the table top exercise. This study indicated that the predominant type of PHE reviewed or scenario used in AAR and SimEx were epidemics, with one SimEx testing a pandemic scenario involving all six regions, similar to the current COVID-19 pandemic. We also saw that in general, AAR were conducted later than the timeframe (from as soon as possible to 3 months from the end of the event) recommended by WHO to minimize recall bias . In addition, in this study, we were able to link the events and functions reviewed/tested in AARs and SimEx to the 13 IHR core capacities, which can provide complementary information to other components in the IHR MEF and capacity building efforts. This, in turn, can offer a more comprehensive picture of the state of the public health preparedness and response in a given country, region and globally.
The majority of the AAR/SimEx were conducted in the Africa region
In this study, we showed that more than half of the AAR and SimEx activities were conducted in AFR while other regions had far fewer reports available or even none. This may be due to a multitude of reasons. Firstly, the recent West Africa EVD outbreak in 2014–2016 and the EVD outbreak in North Kivu in the Democratic Republic of Congo, which were both declared as PHEIC [34, 35] have resulted in an enhanced priority for emergency preparedness and response in this region. In 2019, WHO AFR office (AFRO) supported four SimEx for EVD preparedness . Secondly, given the WHO AFR encompasses many low-resource settings coupled with more infectious disease outbreaks and health emergencies, there has been a heavy emphasis by both WHO AFRO and countries in this region to strengthen their emergency preparedness and response capacity in light of the risks and vulnerabilities that are well recognized. Given the high frequency of acute public health event reported from AFR , there are more funding from the international community to conduct AAR and SimEx, with WHO AFRO more frequently been requested to support AAR and SimEx by their Member States compared to other WHO regions. Finally, other WHO regions such as AMR may have more experience and capacity to conduct these types of activities on their own without WHO support, and may not necessarily report their activities to WHO. In this study, we analyzed reports that were sent to WHO from WHO-supported AAR and SimEx. Given the increasing uptake of AAR and SimEx activities in the public health domain in recent years, we acknowledge that other SimEx or AAR were conducted by Member States that had the capacity and resources to undertake them on their own. In addition, it is also likely that some SimEx or AAR were done without WHO support but with support from other partner organizations, or simply not reported to WHO. Therefore, it is important to note that the data we presented may not represent all the AARs and SimEx conducted globally during the study period.
Infectious disease outbreak as the predominant PHE reviewed and tested
This study identified that the majority of PHE that were reviewed in an AAR or used as a scenario in a SimEx were infectious disease outbreaks. WHO provides guidance and support to countries to strengthen all-hazards approaches to emergency preparedness which requires AAR and SimEx to be implemented for a range of threats beyond only infectious disease outbreaks, including health consequences from conflict, natural disaster, chemical or radio-nuclear spill and food contamination [1, 38]. As AAR and SimEx under the IHR MEF are mainly targeting the response capacity and capability of the Ministry of Health (MoH), it is reasonable to expect that this category is the most common PHE reviewed and tested. Although this may seem timely given the current COVID-19 pandemic, it is important to recognize that public health consequences are broad and do not exclusively come from infectious disease outbreaks alone. Countries are facing an increasing number of emergencies with health consequences from a broad range of hazards, including natural and human-made. In addition, many emergencies are complex, and can have significant public health, social, economic and political impacts. In practice, AAR and SimEx usually include health and non-health sectors as well as other partners and stakeholders. It is also important to emphasize that emergency management is not an exclusive responsibility of one sector or ministry alone.
The difference in the focus of SimEx and AAR
In our study, the category “human-induced/societal” such as nuclear, chemical and mass gathering events was more likely to be used in SimEx, as compared to AAR. Our explanation for this difference can be found in the fact that these types of events are less common in real-world situations. As a general rule, each AAR and SimEx is tailored to be applicable to the national context or setting it is being implemented in as per their specific purpose and objectives. For AAR, the PHE review depends on the actual situation and what events are occurring in the country. In contrast, for a SimEx any event can be used as the scenario and the selection process is often guided by a multi-sectoral risk assessment to help identify the risks a country is most likely to face. Furthermore, these types of events may also be more sensitive from a security point of view, therefore, countries may wish to keep the reports classified or internal. It is therefore likely that there are even more exercises conducted for such events than reported to WHO.
We saw that in our study, the majority of the SimEx conducted were table top exercises. We hypothesize it is due to table top exercise being a discussion-based exercise that requires the least amount of resources and is the least complex to plan, implement and evaluate. Operational-based exercises such as drills, functional exercises, and field/full-scale exercises require more resources, including time needed, financial costs involved, and organizational experience necessary . These operational exercises may be complex to plan and implement, and often require external support, especially in low-resource settings. WHO and partners have been promoting countries to increase the use of SimEx and to incorporate SimEx as a part of a comprehensive programme made up of progressively complex exercises, with each exercise building on the previous one. This `building block approach` is particularly important for organizations with less experience in conducting these activities, where it should start with basic table top exercises first, followed by progressively complex exercises requiring additional time and resources. Although our findings show the majority of SimEx being table top, it is unclear whether this is due to the fact that these are the least complex and require the least amount of resources, or because countries are adopting WHO’s advice of implementing comprehensive exercise programme’s building block approach. However, as part of a comprehensive exercise programme we hope to see a more equal distribution between discussion- based and more complex operational-based exercises being conducted in the coming years as countries familiarize themselves with SimEx.
Cross-cutting IHR core capacities are more often reviewed
As seen in our study, since the scope, purpose and objectives of an AAR and SimEx can vary substantially, the number of IHR core capacities reviewed or tested can also differ significantly anywhere from one to multiple IHR core capacities reviewed or tested at the same time. Although it is up to the countries how many IHR core capacities they would like to review or test, for an AAR or SimEx to be most effective and useful, it is usually better to simplify and limit the scope by having only a few concrete objectives and a limited number of IHR core capacities to be reviewed or tested. This can help the AAR or SimEx to be more focused, resulting in more concrete outcomes that are more likely to be achieved.
For both AAR and SimEx, the most commonly reviewed IHR core capacities were similar, namely risk communication, IHR coordination, health service provision and national health emergency framework, such as the existence of a Public Health Emergency Operation Centre (PHEOC) or emergency preparedness and response plans. Consequently, the most common strengths and challenges were also linked to these main IHR core capacities reviewed or tested as anticipated. We hypothesize these IHR core capacities are most often reviewed or tested given their broad and cross-cutting nature, which often form the key elements in any emergency regardless of the type of PHE [39,40,41].
Moving forward with proposed recommendations from AAR and SimEx
The proposed recommendations in AAR and SimEx were aligned with identified gaps in countries. However, the specificity of the priority recommendations vary per IHR core capacity, ranging from recommendations such as establishing or strengthening coordination mechanisms, ensuring or improving isolation for cases, to broader recommendations that can apply to different IHR core capacities, but if not concrete may not be as actionable, such as training and plan development. A major longstanding challenge after AAR and SimEx is accountability and the implementation of proposed recommendations. Although the involved stakeholders hold great expertise in identifying gaps, implementing measures on the local level, and genuinely learning lessons by implementing the needed steps, remains a persistent challenge. Therefore, post-AAR and post-SimEx follow up is vital to ensure expected or assumed improvements were made based on proposed recommendations. We believe the ideal way for recommendations to be implemented is to incorporate them into existing national plans such as the national action plan for health security (NAPHS) , and integrate them into national operational planning and budget cycles. WHO has also been developing additional guidance to better ensure that Simex and AAR include actionable lessons learned with identified lead implementers (including WHO and partner organizations), so that the recommended actions can be successfully implemented. This document titled, “Roadmap for the implementation of recommendations from conducted IAR/AAR and SimEx” is expected to be published on WHO website in quarter four of 2020 (WHO, unpublished).
Benefits of AAR and SimEx
AAR and SimEx are recognized as key system improvement and learning tools in emergency management that help countries to assess and enhance their operational capability for public health preparedness and response . Used by many organizations and across sectors, AARs and SimEx not only provide functional assessments but also play a key role in identifying strengths and gaps in the implementation of IHR core capacities. They can be used to review, validate or “stress test” the IHR core capacities reviewed by other IHR MEF instruments , for example, by looking at how effective a policy, plan or guideline is implemented, versus the existence of relevant policies, plans and guidelines. In this regard, AARs and SimEx are complementary to the other two IHR MEF components: SPAR  and JEE  as they provide a different perspective on how the IHR core capacities or response system functions in a “real” or simulated event.
Another critical benefit of AAR and SimEx is through the process of planning and conducting these activities, they can also build awareness of roles and responsibilities in different sectors involved in the PHE. These activities serve more than just identifying gaps and lessons learned. It can also start a cross-cutting dialogue across sectors and between individuals needed to strengthen preparedness and response to PHE. The AAR is an important learning tool and effective method for informing stakeholders of best practices, challenges and the root causes of preparedness gaps, and is used by many organizations and across sectors [6, 15, 16, 44,45,46,47]. Similar to the AAR, a high-fidelity simulation enables multiple learning objectives to be achieved in a realistic and secure context . The use of simulation exercises involving the health community has also shown clear benefits on the individual level as well as on the organizational level, and are valuable and effective in the immediate, and to a lesser extend to the longer term . The recommendations formulated during AAR and SimEx shed light for stakeholders and illuminate the way forward.
Standardization of data & common principles for successful AAR & SimEx
The submitted reports were often not standardized, and there were inconsistencies in the structure, format, methodology and availability of key information. These discrepancies made it difficult to code and analyze findings, with some variables unable to be analyzed given only a minority of countries reported the information.
Nevertheless, various common principles were found to be essential for the successful planning, implementation and evaluation of AARs and SimEx. This includes clearly defined purpose, scope and objectives, having the right participants/organizations participate and having a structured evaluation and reporting process that ensures (national) ownership of the findings.
The purpose, scope and objectives are the foundation of any AAR and SimEx and should be carefully chosen to ensure the success of the activity in line with national priorities. Identifying and selecting the right participants is another crucial element for an AAR or SimEx and should be based on the purpose, scope and objectives, and thus on the functional areas or pillars that are reviewed or tested, respectively. If specific participants or agencies do not participate, inaccurate assumptions about the response functionality are likely to be made. WHO recommends countries to use a whole-of-society approach to ensure a broad commitment and mutual accountability in the support of and follow-up in implementations of recommendations emerging from AAR and SimEx. Furthermore, AAR and SimEx objectives should be linked with the capacities to be reviewed or tested, to ensure a well-structured evaluation process and report. Improvements in the design and implementation of SimEx and AARs could facilitate better reporting and measurement of preparedness outcomes [50, 51].
In line with above, WHO published the Country Implementation Guidance for After Action Reviews and Simulation Exercises under the IHR MEF . This provides strategic guidance and criteria for inclusion of AAR and SimEx under the IHR MEF and introduces a structured evaluation method as well as a standardized minimum reporting template with timeline indicators for AARs. Furthermore, the Guidance for AAR and SimEx has also been published, which offers additional detail into the planning, execution and follow-up as of both activities [5, 8]. Using simple, standardized reporting format such as those provided by WHO and other partners [7, 14] will help consistent and standardized information collection for data analysis, which can, in turn, facilitate a comprehensive understanding about the Member States’ emergency preparedness and response capabilities under the IHR (2005).
Limitations of study
There were several limitations to this study. Firstly, this study was limited by the fact that only a limited number of reports were available due to voluntary nature of AAR and SimEx under IHR MEF and therefore may not necessarily reflect a representative overview of all public health AAR and SimEx conducted globally. It is highly possible some countries with higher capacity conducted their own AAR and SimEx without informing WHO, therefore, were not included in this study. Secondly, as the study only used information provided in reports of activities. We did not have further secondary data to corroborate our findings, and that could be used as evidence to examine the impacts and benefits of AAR and SimEx. Thirdly, as WHO Member States may not have reviewed and tested the public health response pillar with explicit reference to the IHR core capacities, at times, it was challenging to code the response pillar reviewed and tested to the 13 IHR core capacities. Coding inconsistencies were addressed by having two independent coders and obtaining consensus with a team of WHO experts in public health emergency preparedness and response. Fourthly, overall, the timeframe from the end of an event to the AAR was longer than the 3 months WHO recommends , which may have resulted in some level of recall bias. Finally, in this study, it was evident the complexity of analyzing real-world public health practice data. In practice, one SimEx can involve one country, several countries in one region, or multiple countries in all WHO regions. For AAR, sometimes countries may request to conduct AAR for multiple public health events during the same AAR. These considerations may have been practical and useful for the countries and regions, but made the analysis more challenging when describing the data.
Conclusions and recommendations
Every country faces a broad range of emergencies resulting from a variety of hazards that differ in scale, complexity and international consequences. In developed and developing countries alike, these emergencies can have extensive political, economic, social and public health impacts, with potential long-term consequences sometimes persisting for years after the emergency. AAR and SimEx are useful tools that can review PHE experienced by the country or simulate a rare PHE to facilitate individual and collective learning on the coordination and response of a future PHE should it arise.
From the analysis, it is fair to conclude that the strengths, challenges and recommendations all aligned with the functional areas or IHR capacities tested or reviewed. However, it is not possible to conclude whether these areas or IHR core capacities are indeed the key priority to invest in for enhancing public health preparedness and response. Future analysis may be useful, including the cross-analysis with other IHR MEF assessments available such as the SPAR and JEE results, and using proxy outcome measures such as reduction of morbidity and mortality, and timeliness of outbreak metrics to benchmark achievements.
Moving forward, it is vital to reiterate the importance of 1) scaling-up the implementation of SimEx and AAR as a means of enhancing preparedness; 2) reducing the timeframe from the end of an event and its AAR; 3) standardizing the critical information to be captured in AAR and SimEx and improving the ease and importance of information sharing with WHO; 4) clearly linking the public health response pillars tested and reviewed to the 13 IHR core capacities; 5) better defining the purpose, scope and objectives of the activity so results and the impact can be better measured; 6) Besides reviewing or testing health capacities in infectious disease outbreaks, encourage countries to review and test other hazards by adopting and promoting the all-hazard and multi-sectorial approach. With the recent guidance published by WHO [5, 7, 8], as well as from partner organizations [9,10,11,12,13,14,15,16,17,18]. we hope to see standardization in the AAR and SimEx methodology, practice and reporting.
Current situation: AAR and SimEx in the context of the COVID-19 pandemic
Since December 2019, the COVID-19 pandemic has caused unprecedented global disruptions in all aspects of lives and livelihood of individuals, impacted global economic, trade and tourism, and pushed world leaders to rapidly come up with solutions to resolve this crisis . We are at a critical juncture, as various public health measures are being implemented and their effects monitored. We are faced with a “new normal” in the way we conduct our daily lives for the months ahead until an effective and safe vaccine can be developed, and broadly and equitably distributed.
It is interesting to note that the main challenges we observed in our analysis of the AAR and SimEx reports from 2016 to 2019, such as poor community engagement, challenges with unified command and coordination system, and inadequate infection prevention and control (IPC) practices and supplies, including isolation of confirmed cases, were also some of the key challenges seen in the COVID-19 pandemic. As emphasized in the Global Preparedness Monitoring Board (GPMB) 2020 report , the GPMB calls for urgent actions for “responsible leadership; engaged citizenship; strong and agile national and global systems for global health security; sustained investment in prevention and preparedness, commensurate with the scale of a pandemic threat; and robust global governance of preparedness for health emergencies.”
As the number of COVID-19 cases and deaths decrease in certain countries, we urge affected countries to start preparing for IARs and AARs to ensure critical lessons can be learned, in preparation for future PHE. For the few countries, territories and areas which have none or very few COVID-19 cases and deaths, it could also be helpful to conduct COVID-19 SimEx to prepare for potential epidemics. In addition, COVID-19 SimEx can also be useful for those countries with more cases to scale up emergency operations and enhance preparedness capacities for possible next waves. Besides the existing generic guidance and tools for AAR and SimEx, WHO has published specific guidance on country COVID-19 IAR  as well as four COVID-19 specific SimEx packages. The COVID-19 IAR guidance and its ten accompanying tools were developed to support countries to conduct periodic review(s) of their national and subnational COVID-19 response efforts. The four COVID-19 SimEx packages include a generic SimEx that can be used at the national level, a health facility and IPC specific SimEx, a points of entry (PoE) specific SimEx, and a SimEx for the urban environments . As of 2 October 2020, nine IARs have been conducted in AFR, SEAR and EUR and eight more are currently in the pipeline . As of 14 October 2020, the WHO COVID-19 SimEx website has reached over 219,900 visitors since its first publication on 4 February 2020 (Copper, F.A. unpublished data). The GPMB has also highlighted in their 2020 report calls for urgent actions to countries to “routinely conduct multisectoral simulation exercises to establish and maintain effective preparedness” . Both the COVID-19 IAR and SimEx packages have been added to the countries action checklist of the COVID-19 Partners Platform  to monitor and report which countries are conducting these activities and using them to update national COVID-19 preparedness and response plans.
The COVID-19 pandemic has affected both developed and developing countries irrespective of income level. This is a wake-up call to all countries that no country is immune to an emerging or re-emerging public health threat in our inter-dependent and inter-connected world. We urge all countries to invest in preparedness and incorporate the lessons from this pandemic to further advance national, regional and global health security.