(1st Author, year, country, titel)
|Ablah E., 2007, USA, ||Online questionnaire with 5-point-Likert-scale answer options||
RR 56/65: 86%|
- to identify the need for and implement surge capacity: 43% good / excellent
- Participate in a coordinated response: 32% good / excellent
- To implement your risk communication skill set: 12%
- To identify and locate your agency infectious disease resrouces: 49%
RR 48/65: 74%|
- ability to identify the need for and implement surge capacity: 74% good / excellent (p = 0.003)
- Participation in a coordinated response: 60% good / excellent (p = 0.017)
- To implement your risk communication skill set: 25% (not sign)
- To identify and locate your agency infectious disease resrouces: 60% (not sign)
|Ablah E., 2008, USA, ||< 4 weeks after the exercise, focus groups were held, transcribed and main themes identified||-||
Self-reported skills: improvements in surge capacity, coordination between counties, risk communication, and awareness of protocols and procedures; better able to (be) support (ed), they see necessity of a coordinator, effective relation building.
Satisfaction/ training content/methodology: The exercise format was liked.
|Aiello A., 2011, Canada, ||Retrospective pre-test & post-test: questionnaire with 5-point-Likert-scale answer options||
RR 1250/1020 (82)|
35% felt prepared to deal confidently with the situation during a pandemic
RR 1250/1020 (82)|
76% felt prepared to deal confidently with the situation during a pandemic. (p = 0.0020).
“a high proportion” thought the training relevant for work- & personal life, useful, helpful and informative.
|Alexander L.K., 2005, USA, ||Pre-test and post-test on knowledge with MCP questions after each online module; course evaluation after 5th module with 4-point-likert-scale and open answer options.||
Knowledge results not available
Knowledge results not available
100% recommended course to others; 80% strongly intended to use the information gained; 100% agreed that lectures (100%), readings (94%), activities (98%), pre-and post-test(94%), and face-to-face module (95%) helped their learning.
33% would like online interaction.
|Alexander L.K., 2008, USA, ||
Pre-test: retrospectively in post-test; questionnaire with 5-point Likert-scale answer options.|
Post-test: questionnaire with 5-point-Likert-scale answer options
RR 177/156 (88)|
- Use reports to identify health issues: 2.67
- Communicate with other agency to identify new cases: 3.04
- Maintain security and confidentiality of personal and PH information: 3.96
- Stay informed on PH laws and regulations: 2.77
- Use regulations to promote health: 2.69
- Recognize a disease outbreak: 2.79
- Collect biological or environmental samples: 2.30
- Be aware of amount of each important health problem: 2.76
- Work as part of a team during outbreak investigation: 2.77
- Write a press release: 2.06
- Create a line listing: 2.04
- Create an epidemic curve: 1.81
RR 177/156 (88)|
- Use reports to identify health issues: 3.62 (p = < 0.0001)
- Communicate with other agency to identify new cases: 3.87 (p = < 0.0001)
- Maintain security and confidentiality of personal and PH information: 4.31 (p = < 0.0001)
- Stay informed on PH laws and regulations: 3.63 (p = < 0.0001)
- Use regulations to promote health: 3.55 (p = < 0.0001)
- Recognize a disease outbreak: 3.89 (p = < 0.0001)
- Collect biological or environmental samples: 3.17 (p = < 0.0001)
- Be aware of amount of each important health problem: 3.49 (p = < 0.0001)
- Work as part of a team during outbreak investigation: 3.89 (p = < 0.0001)
- Write a press release: 3.28 (p = < 0.0001)
- Create a line listing: 3.76 (p = < 0.0001)
- Create an epidemic curve: 3.69 (p = < 0.0001)
Satisfaction: 99% would recommend it; 96% thought the course was excellent; 100% inteded to use the information in their jobs; 100% said the material met the objectives; 98% thought that both internet, activities and case-studies helped their learning; 91% thought the face-to-face module helped. 4% thought the material did not fit together.
|Araz O.M., 2012, USA, ||Post-test survey with 5-point Likert-scale answer options and open ended questions||-||
Satisfaction/ training methodology
66% found the multimedia presentations very helpful in terms of understanding the state of the outbreak; 69% agreed that the displayed scenarios were realistic and plausible; 84% agreed that the information exchange during the exercise was in high quality, which means that the decision-making environment facilitated communication.; “most particpants” agreed that the presented method increased readiness for a pandemic. “participants” stated that the video scenarios and the simulation model made it very real.
|Araz O.M. & Jehn M., 2013, USA, ||Pre- and post-test: 3-point-Likert scale answer options||
- Leadership and management: 93.5%
- Mass care: 84.1%
- Communication: 84.7%
- Disease control and prevention: 91.5%
- Surveillance: 82.0%
- Leadership and management: 99.0% (p = 0.010)
- Mass care: 85.6%
- Communication: 94.1% (p = 0.001)
- Disease control and prevention: 91.0% (p = 0.010)
- Surveillance: 90.0% (p = 0.010)
|Atack L., 2008, Canada, ||
Pre- and post-test (< 2 weeks after): questionnaire with Likert-scale answer options and open-ended questions, adapted from validated questionnaire.|
Four validated questionnaires on organizational climate, receptivity, and course satisfaction:
- Social System and the Organization (SSO) (climate)
- New Ideas and You (NIY) (receptivity)
- Education Program and Change (EPC) (feasibility)
- Continuing Professional Education (CPE) (course satisfaction)
RR 67/76 (88)|
Knowledge test: Mean score of 38.4/56
RR 67/76 (88)|
Knowledge test: mean score of 46.4/56 (p = < 0.001)
Knowledge & the system
Relation between test and postcourse competency: SSO, NIY and CPE not sign.
EPC: r = 0.431 p = 0.001.
100% agreed that the course was extremely helpful, 95% creative. 100% would recommend online learning as a way to learn about infection control; 27% thought video demonstration was in important to learning. Suggestions for approval: function to skip video’s (11%); not to re-do the entire module after failed for postquiz (7,5%). Open ended questions how the course had been useful: three major themes: improved hand hygiene practice, improved teaching to patients/visitors/staff on how to use PPE, improved their own techniques
RR 55/76 (72)
Behavioral change: reported an example of behavioral change after the course.
|Atlas R.M., 2005, USA, ||Scored evaluation and anecdotal comments||-||
Participants regularly identify the moulaged patients as the most effective element of training.
Self-assessed attitude: 94% say use of patient simulator changed their awareness of respiratory disease transmission,
Self-assessed Behavior: 98% would alter respiratory protection when confronted with a patient demonstrating signs and symptoms of a respiratory tract infection.
|Baldwin K., 2005, USA, ||
Post-test: 5–10 MCP knowledge questions after each module.|
+ unknown qualitative evaluation method for satisfaction
Results not reported
Overly complex system of accessing different parts of the program; impossible to predict large number of updates required; internal shift of focus could not be implemented; funding to sustain the program was hard; lack of designated ownership of the program.
Use of an readily available system, accessible at different sites; completed at own pace and role; evaluation of compliance could be monitored; individuals could be identified for additional training.
|CBS||Bazeyo K.M., 2015, Uganda, ||Pre- and post- knowledge test. Format unknown.||
Mean score 5–20% lower than in post-test for all districts. No numerical results available.
Mean score 5–20% higher than in the pre-test for all districts. No statistical results available.
|Becker K.M., 2015, Ghana, Uganda, Nigeria, USA, ||Post-test: summing up achievements||-||
RR 43/43(100), two years.|
Built in-country cadres of midlevel public health; cross-border rabies vaccination program; evaluation of 66 disease surveillance systems; integrated vaccination campaigns & improved coverage rates; increased capacity for epidemiological studies, provide an evidentiary basis for PH action/strategy; influence PH policy; improved diagnostic and laboratory management skills; participants trained district level laboratory personnel in integrated disease surveillance.
|TOT||Berrian A.M., 2018, USA, ||
Pre- and post-test: Written MCP knowledge test on content;|
Post-test: written self-report (4-point Likert-scale) of knowledge, skills, pedagogical skills, leadership skills
Follow up: -
Knowledge: Mean increase of 17% p = 0.0078;
Self-assessed knowledge: Improved on pathogen transmission (p = 0.0156), risk mitigation (p = 0.0020) and effective pedagogy (p = 0.0020).
Self-assessed skills: Improved for risk assessment p = 0.0078; facilitate a workshop (p = 0.0020).
|Results from the second wave were part of the train-the-trainer session, and are not presented because they were non-professionals.|
|Biddinger P.D., 2010, USA, ||
Pre- and post test of knowledge and confidence;|
Post-exercise survey to assess awareness of agencies’ roles and responsibilities; AAR
RR 1145/5892 (participants of table-top exercises)|
Knowledge: no data available;
Attitude: no data available.
RR 1145/5892 (participants of table-top exercises)|
Knowledge: increased with 25% on average. 56% increase knowledge of what other agencies may have to offer (resources/assets); 69% clarify role&responsbility of perosn & organizaiton;
Attitude: On average 12% better confidence.
Satisfaction/ training methodology:77% scored it effective in practicing working together; 73% providing an opportunity to evaluate plans/procedures;
Satisfaction for target groups: Regional response exercise respondents reported higher satisfaction with the effectiveness of understanding roles (p < 0.001), providing right environment (p < 0.001), promoting regionwide cooperation and mutual aid (p < 0.001); more engaged in the exercise (p < 0.006). No relation in knowledge and satisfaction with the program. Small and big towns had higher satisfaction.
Network building: 70% assessing connectivity within/across agencies; 53% promoting region-wide cooperation and mutual aid.
|Cathcart L.A., 2018, USA, ||Pre-, post- and follow-up (timing unknown): survey with 5–8 statements rated with a 5-point-Likert-scale answer options.||
Mean score 2.2
Mean score 4.0
Mean score 4.3
Used what I learned: mean score 4.2 +/−1.0
Different way of conducting my job: 3.9 +/− 1.0;
Prepared me for my job: mean score 4.0 +/− 1.0
|Chandler T., 2008, USA, ||
Knowledge test of 15 questions (unknown methodology)|
Self-assessed knowledge on a 4-point-Likert-scale
Skills and behavior assessed by local supervisors (unknown methodology)
Knowledge: mean score 72.17 (SD 16.31)
RR 817/> 817|
Knowledge: Mean score 94.25 (SD 8.07) (p < 0.001).
RR 764/> 817
86% felt more knowledgeable about the basic emergency preparedness core competencies; 82% about their agency’s chain of command during an emergency response; 80% about their own functional roles during emergency reponse.
RR 511/> 817
No data available
RR 511/> 817 Behavior
Supervisors: usage of the course was an effective means for improving work performance.
|Chiu M., 2011, USA, ||Survey in which self-assessed confidence in preparedness, response and recovery skills and needs were rated using a 5-point-Likert-scale||
10 preparedness competences: 30.2 SD6.7
8 response competences: 26.0 SD5.5
7 recovery competences: 23.1 SD5.5
Preparedness: 32.7 SD7.3
Response: 26.0 SD6.9
Recovery: 22.6 SD6.5
10 preparedness competences: 36.2 SD4.9 (p < 0.001)
8 response competences: 30.9 SD4.5 (p < 0.001)
7 recovery competences: 28.8 SD4.18 (p < 0.001)
Preparedness: 22.0 SD9.5 (p < 0.001)
Response: 17.1 SD7.3 (p < 0.001)
Recovery: 15.7 SD6.6 (p < 0.001)
|Craig A.T., 2007, Australia, ||Post-test: feedback from stakeholders in debriefing sessions; state-wide participant questionnaire with 5-point-Likert-scale answer options; both retrospective pre-test and post-test.||
Self-assessed system performance:
Prepared to respond to a case of pandemic influenza: mean score 3; > 55% agreed or strongly agreed.
Self-assessed system performance:
Prepared to respond to a case of pandemic influenza: mean score 4; > 90% agreed or strongly agreed
Satisfaction: 84% agreed that exercise Paton prompted their influenza pandemic planning.
|Dausey D.J., 2007, USA, ||Review of after action reports, participants post-exercise evaluations, and internal team discussions and consensus following exercise debriefings||-||
Exercise methodology: Exercises should a) be designed to achieve a specific objective. B) be as realistic as possible & logistically feasible; C) be designed around “issue areas” rather than scenarios. E) have forced, targeted and time delineated decision making. F) have limited number of participants. G) be designed and executed from collaborative engagement of representatives from participating agencies and external developers and facilitators.
|CBS||Dausey D.J., 2014, USA, ||
Post-test: evaluation form with Likert-scale answer options, some with open-ended questions|
Follow-up: semi-structured interviews with health leaders at ministry level
Satisfaction/methodology: 88–100% rating the quality of the exercise as high; 92–94% exercise helps to understand roles and responsibilities of organizations responding to influenza pandemic; 50–73% ability to identify key gaps in performance.; more sectors should be included, including more private partners and NGOs; better grounding of theoretical responses with practical responses are required.
Self-assessed knowledge: 82–100% gain knowledge that they plan to improve the preparedness of their organization;
Self-assessed attitude: the exercise raised awareness and understanding about public health threats;
unknown 137 observers|
Participation in the exercise helped to motivate to develop an exercise program and regularly assess different aspects of their public health preparedness. Barriers are a lack of financial resources, limited support among leadership to develop and sustain the program.
Most countries reported modifying and using exercise templates.
|Dickmann P., 2016, Hong Kong, Poland, Sweden, Switzerland, UK, ||Pre-and post-questionnaires; day assessments at the end of each day||RR 15/15||
Satisfaction: 14/15 expectations have been fully met; they appreciated that the training was based on a reflective and reframing approach
Self-assessed knowledge: 14/16 stated their understanding had increased considerably
|Self-assessed knowledge: 50% good knowledge of risk communication theory.||Self-assessed attitude: the part that had good knowledge, felt better prepared to advocate for this change|
|CBS||El-Bahnasawy M.M., 2014, Egypt, ||Pre- and post- knowledge tests (methodology and scales unknown)||
Knowledge: median scores: Anthrax 3a; Tick-borne relapsing fever: 2a; Lice-borne relapsing fever: 2a; Clonorchis sinensis:4a
Knowledge: median scores: Anthrax 8b; Tick-borne relapsing fever: 4b; Lice-borne relapsing fever: 4b; Clonorchis sinensis: 9b
Satisfaction/methodology: results useless because no information about training methods is provided.
Knowledge: median scores: Anthrax 9c; Tick-borne relapsing fever: 5c; Lice-borne relapsing fever: 5c; Clonorchis sinensis: 9b
(p < 0.001 for all subjects)
|TOT||Faass J., 2013, USA, ||Post-test: Partly hardcopy, partly online survey with 29 open-ended and 5-point-Likert-scale questions||–||
RR 120/231(52): Satisfaction: 90% of attendees rated the overall impression as “good” or higher. 94% answered “agree”/ “strongly agree” on relevance of course materials to their organizations and employees.|
Attitude: 84% agreed that pandemic preparedness posed a challenge for the transit industry;
|Results from the second wave of training: -|
|Fowkes V., 2007, ||Post-test evaluation form with 5-point-Lickert-scale answer options and open-ended question: How would you incorporate what you have learned?||-||
RR > 6000/9537|
Satisfaction/methodology: 95% rated modules as good/excellent
Self-assessed attitudes: modules reinforced needs for emergency plans; enhanced awareness of possible unusual clinical presentations; enhanced consciousness about infection control.
Self-assessed knowledge: modules provided useful approaches to decontamination;
|Fowkes V., 2010, ||Evaluation forms; assessment of the departments’ emergency plans; AARs completed by the exercise groups; debriefing of coordinators; interviews with trainees; follow-up interviews; quarterly reports by the organizing party||
System results: 37% of emergency operation plans complete
System results: 46% of emergency operation plans complete
Satisfaction/methodology: 92–98% rating the training experiences as good to excellent.;
Self-assessed knowledge/skills: 92–98% rates knowledge and skills gained form the exercises as good to excellent.;
Skills: 72–90% were able to describe events and steps necessary to activate their plan, actions that should be taken when it is activated, roles of individuals, internal and external communications needed, how to participate in a coordinated response, and who is responsible for the oversight of the plan. 69% how to correct the plan when needed; 62% to lessen the spread of disease to staff, patients, and families. 42% to plan for a surge of infectious patients.
System results: 74% of emergency operation plans complete; 91% of clinics made improvements in their emergency plans compared to post-test with statistically significant changes in 2/3 of the 15 criteria (p = 0.001–0.46).
|Gershon R.R., 2010, ||Pre- and post-test of 7 questions on knowledge and 9 items course evaluation.||
Knowledge: mean 6.3 +/− 1.1
Knowledge: mean 6.6 +/−0.8 (p < 0.001)
Satisfaction: 98% thought the program was valuable; 97% risk was addressed; 95% length of training is acceptable.
Self-assessed knowledge: 95% their PPE understanding was reinforced; > 92% knowledge of respiratory illnesses improved;
|TOT||Grillo M., 2017, ||1 h pre- and post test with 40 MPC and true/false questions testing knowledge.||
Knowledge: 50.5% mean score
Knowledge: 67.9% mean score (p < 0.001)
|Results from the second wave of training: -|
|Hegle J., 2011, ||Observation by at least 2 researchers using an observation guide; semi-structured interviews with exercise leaders; review of planning and exercise materials||
System performance: Direct result of the exercise: 1) building relationships among response partners across counties and agencies; enhancing social capital. 2) promoting visibility of public health and assets in an emergency response; 3) testing multiple communication systems. The use of these systems was problematic in all exercises observed; 4) training public health practitioners in exercise evaluation.
|Hoeppner M.M., 2010, USA, ||At 6 and 12 months after participation in one of the courses, a survey with eight questions. Skills/knowledge gained, changed attitudes, changed behavior, affected others, face barriers. Based on the Kirkpatrick model||-||-||
RR 244/387 (63)|
64% agreed that courses helped them develop the competencies; 12 months: 63.5%.
Self-assessed knowledge/skills: Development of critical thinking and systems-thinking. Deeper appreciation for the complexity of planning and response and the need for collaboration across all agencies and levels of government. Recognised the national incident management system as a organizing mechanism. Better able to gauge the resources needed and better advocate for these. Better able data collection; better able to inform and educate others.
Self-assessed attitude: Recognized the need for improved surveillance, use of technology to organize data, shared data management across agencies.
Behavioral change: Performed better data collection; inform and educate others; developed wider range of exercises, revised emergency plans, enhanced and eexpanded cross-agency collaboration.
Barriers: financial and human resources, lack of time, unavailable software, how to convince others?, underutilization of expertise due to changing jobs.
|Horney J.A., 2005, USA, ||Knowledge post-test after each module; survey;||-||
Knowledge: no results available
98% indicated that the training module provided the information they were looking for (100% for PH nurses); 62% had recommended the course to collegues.
Self-assessed knowledge: 2/3 module introduced them to terms and concept they were previously unfamiliar with; 87% module clarified terms or concepts they had not enough information about; 97% module reinforced familiar terms and concepts; 81% module specifically adressed prof roles and responsibilities (80% for PH nurses);
Self-assessed attitudes: 92% module made them feel better equipped to do their job (94% for PH nurses);
|Hueston W.D., 2008, USA, ||Representatives of partner universities shared their lessons learned.||-||-||
Several years after the start of the program:
Methodology: Strong collaborations and good communication are essential; differences in DVM and MPH professional cultures must be understood; differences in CVM and SPH organizational cultures must be bridged; human and financial resources provide significant challenges; faculty efforts systems may differ; different curriculum design and delivery call for innovative approaches; CVM/SPH partnerships are mutually beneficial; Technology is not a panacea - developing good distance learning courses takes a lot of time and money; no single standard for distance learning exists to unify the competing software platforms and various computer operating systems.; be prepared for succes.
|Johnson Y.J., 2009, USA, ||Post-exercise evaluation form with 11 five-point-Likert-scale questions||-||
Satisfaction/ Training methodology: Exercise was well structured and organized (4.0); scenario was plausible & realistic (4.14); knowledgeable facilitators about (4.14); facilitators kept exercise on target (3.86); technologies were useful (4.14); participation was appropriate for my position (4.00); correct mix of people (3.79); adequate facilities utilized (4.36); meals and breaks (4.71);
Self-reported skill: could practice and improve responsiveness (3.86);
after the exercise my agency/jurisdiction is better prepared to successfully deal with the scenario (4.00).
|Kohn S., 2010, USA, ||
Post-test: evaluation sheet with Likert-scale answer options; informal conversations with participants and trainers.
80% agreed enough interactive exercises; 89% that session provided useful information now/ in near future; 90% that overall session was valuable.; conversations identified: participants saw the course as effective and helpful in their professional setting; overwhelming volume of information; wish for more examples/ scenarios to illustrate practical application. ‘Many’ thought the material as not relevant to them. INFLUENZA TRAINING:
Positive ratings of course content, facilitator, and presentation; information provided helped frame their emergency response roles & responsibilities within a PH context; training is essential in clarifying some response activities; core concepts taught help operate more effectively during a variety of future emergencies; still the course material could have been better tailored.
|TOT||Livet M. 2005, USA, ||Participants surveys using 6-point-likert scales; observation forms; and evaluation instrument for a TTX||
1.60 – 4.10 on a 6-point scale
Describe PH role during emergency response: 4.09
Describe chain of command: 4.58
Identify response plan: 5.06
Describe functional roles: 5.07
Describe communication roles: 4.61
Identify limits to own competence: 4.48
Recognize unusual events: 4.66
Apply creative problem solving and evaluate effectiveness: 4.62
Plan a TTX: 3.84
Implement a TTX: 3.80
Evaluate a TTX: 3.76
Write an AAR: 3.72
Describe the ICS: 4.62
Describe risk communication strategies: 4.04
Describe criminal and epidemiological investigative methods: 3.84
Satisfaction: Organization of the sessions and TTX 60–90% thought it good-excellent. Usefulness and applicability 60–84% agreed or strongly agreed.
4.91–5.67 on a 6-point scale (p < 0.001)
Describe PH role during emergency response: 5.30 (p < 0.001)
Describe chain of command: 5.35 (p < 0.001)
Identify response plan: 5.57 (p = 0.001)
Describe functional roles: 5.49 (p < 0.001)
Describe communication roles: 5.31(p < 0.001)
Identify limits to own competence: 5.30
Recognize unusual events: 5.40 (p < 0.001)
Apply creative problem solving and evaluate effectiveness: 5.26 (p < 0.001)
Plan a TTX: 5.44 (p < 0.001)
Implement a TTX: 5.42 (p < 0.001)
Evaluate a TTX: 5.36 (p < 0.001)
Write an AAR: 5.23 (p < 0.001)
Describe the ICS: 5.46 (p < 0.001)
Describe risk communication strategies: 5.29 (p < 0.001)
Describe criminal and epidemiological investigative methods: 5.20 (p < 0.001)
|Results of the second wave of training: -|
|Macario E., 2007, USA, ||
Broadcast evaluation through a survey and knowledge post-test for those who wished further education;|
Tabletop evaluation through participant & facilitator surveys;
RR: broadcast 821/25000|
RR: tabletop 164/unknown
RR knowledge test: 735/25000
RR teleconference: 21/unknown
Satisfaction: 75–98% agree on satisfaction statements for the broadcast; 4.1/5 tabletop is helpful in feeling more prepared; 4.1 for tabletop overall. 2 h was too short for real effectiveness. Tabletop helped to learn the language of such preparedness activities and identified major gaps; 4.1 on how deliverables helped them in familiarizing themselves with pp. plans; 3.7 for broadcast info; Survey tabletop facilitators: 4.7 program was appropriate for their health departments; 4.3 training helped increasing capacity to respond; program helped to identify key deficiencies in preparedness, as well as to learn what is and is not in their control. Future tabletops address more localized concerns and have materials earlier available;
Knowledge: 90% had 100% score;
survey tabletop particpant: teleconference satisfaction, skill, confidence: pairing of broadcast with tabletop was effective complementary training; training helped them better understand their plans, roles of decion makers in the chain of command;
|CBS||Martin G., 2018, Ireland, ||Lessons learned in hot debrief + observers/evaluators during the exercise||
Methodology: Exercise planning should not be overly ambitious. If testing too many facets of the emergency response protocols, the public health response can be deprioritised; the practical implementation of communication protocols in a real time exercise of this scope proved challenging; the roles and responsibilities of the various agencies involved needs to be clear. In the chaos of an incident it is easy for role confusion to set in; equipment and infrastructure must be in place and must have been thought about before an actual incident (wheter ore not cell phone signals are available on site or requiring boosting e.g.).
|Mitka M., 2003, USA, ||Comments of participants||-||Reaction: exercise helped to improve the nation’s response by testing emergency plans and finetune them.||-|
|Morris J.G., 2012, USA, ||Post-test: survey 6 open-ended & 10 questions with Likert-scale answer options||-||
22/22 scenario plausible, 20/21 scenario comprehensive, 22/22 scenario generated productive discussion, 22/22 scenario helped identify strength/weaknesses, 17/22 scenario helped identify gaps in current planning, 19/21 helped build relationships with participant from key agencies, 16/20 relationships with participants from other states
22/22 knowledge of this type of emergency event has increased, 17/22 understanding of my role has increased, 22/22 my understanding of others’ role has increased.
|Olson D., 2008, USA, ||
Post-test: Course evaluations (no results)|
Follow-up: 6 & 12 months after end of program (no results)
Testimonials (unknown method)
Based on the Kirkpatrick Model
“I would not be in the position I am today if I had not had the opportunity to advance my knowledge in their uniquely structured program” “It has helped me in everything I do related to emergency preparedness!”
Behavior: “I have reviewed and updated our plans. I have trained staff on the topics I studied. We have addressed mental health needs more in our planning and training.”
Self-assessed knowledge: These courses have made me understand what other training I need as well as training needed by other staff and partners in preparedness. “Gave me more knowledge about writing plans, designing plan exercises, working with media and developing media kit, discussing mental health issues with local providers, planning for emergencies with community leaders.
|TOT||Orfaly R.A., Frances J.C., 2005, USA, ||Evaluation of trainings performed by the TRAINED-TRAINERs 6 months after the TTT: Survey; Open-ended questions; registered number of trainings and people trained in second wave; follow-up interviews with TRAINED-TRAINERs||-||-||
TRAINED-TRAINERs RR: 21/21 (100)|
Participants RR unknown
118 participants trained;
20% of TRAINED-TRAINERs had conducted a training themselves after 6 months. Barriers were: lack of time&resources; not enough confidence; trainings were too general and did not provide enough information specific to particpants’ funcitonal roles
Satisfaction: high rankings for the instructors (4.5/5); similar high scores for knowledge, responsiveness and organization; content and materials 4.0; self reported findings on training objectives:> 4.0 for four of the trainers and 3.5 for the fifth; qualitative evaluation of participants were favorable;
|Orfaly R.A., Biddinger P.D., 2005a, USA, ||Evaluation survey (unknown method)||-||
96% would recommend the course to classmates; 92% the course is superior to other courses they had taken at HSPH; 100% the course is useful to their profession; 83% there was appropriate emphasis on practical skills.
|TOT||Otto J.L., 2010, USA, ||Post-test: survey with MCP/Lickert scale questions; Follow up in six months with a survey with MCP and open questions on implemented trainings, revised plans, and barriers||-||
RR 65/85 (76)|
95% expectations & instructions clearly presented, 92% scenario is realistic and credible, 87% TTX better equiped them to plan and execute an installation-level pandemic influenza exercise. 89% TTX identified strengths and gaps in response, 77% identified consequences of interventions and described strategies for dealing with them. 94% TTX identified opportunities for enhanced military and civilian coordination.
68% had a new/revised pandemic influenza plan, 44% my installation conducted a pandemic influenza exercise.
Barriers: 67% competing priorities, 37% time limitations, 19% personnel limitations, 11% budget limitations, 37% other.
|Peddecord K.M., 2007, USA, ||
Pre-post- and 6-week follow-up: questionnaire with MCP, likert-scale and open-ended questions.|
Pre-and post on knowledge
Follow-up: attitudes + behavior
520/> 1658 (< 31)|
Knowledge: mean score 81%
76% thinks mass vaccination clinics are necessary and beneficial
8% thinks it is a good idea but not that important in the overall PH system
10% feels intimidated by the scope& responsibilities
3% no opinion
520/> 1658 (< 31)|
Knowledge: mean score 85% (p < 0.001)
92% thinks mass vaccination clinics are necessary and beneficial
3% thinks it is a good idea but not that important in the overall PH system
4% feels intimidated by the scope& responsibilities
< 1% ambivalent
< 1% no opinion
(p < 0,001 compared with pre-test)
291/> 1658 (< 18)|
84% thinks mass vaccination clinics are necessary and beneficial
4% thinks it is a good idea but not that important in the overall PH system
8% feels intimidated by the scope& responsibilities
2% no opinion
(p = 0.002 compared to pre-test, p = 0.005 compared with post-test)
Behavior: 44% planned or taken action to change their health departments’ emergency preparedness plan
|Potter M.A., 2005, USA, ||Post-test: survey with Likert-scale answer options, results of own project||
Satisfaction/methodology: high scores, see study in Table 1.
Knowledge & skills:
Mean scores between 4.00 and 4.93 for assessing change potential; system thinking; improving negotiation skills; solution mapping; legal mandates; implication wheel, interest-based dispute resolution.
|Quiram B.J., 2005, USA, ||Pre- and posttest: knowledge test;||
Knowledge: see data post-test
Knowledge: increases on all topics. For different sessions: overall: between + 9.7 and + 32.2%
Preparedness planning: between + 5.8 and + 24,5%
Biological/chemical agents: between + 3.9 and 34.3%
Laboratory capacity: − 2.4 and + 49.5%
Risk communication: + 4.6 and 30.2%
Communication and IT: + 12.9 and 47.1%
Surveillance & epidemiology:+ 12.7 and + 45.6%
|Qureshi K.A., 2004, USA, ||
Pre-post- and follow-up (1–6 months after) had 10 MCP questions on knowledge, 8 on behavioral intentions and attitudes using 3-point-likert scales|
General evaluation to a random sample of 100 people
Knowledge: means score 8.24
84.3% important to respond during emergency
91.0% responding will help the community
72.9% other PH nurses will respond during an emergency
78.9% they are themselves responsible to assist during emergency
73.2% intend to respond when needed
70.1% coworkers approve my role in emergency response
59.2% believes significant other approves role in emergency response
Knowledge: mean score 8.38 (p < 0.05)
85.3% important to respond during emergency (NS)
91.6% responding will help the community (NS)
76.9% other PH nurses will respond during an emergency (p < 0.05)
84.6% they are themselves responsible to assist during emergency (p < 0.01)
76.5% intends to respond when needed (p < 0.05)
73.0% coworkers approve my role in emergency response (NS)
64.1% believes significant other approves role in emergency response (p < 0.05)
RR 94/100 (94) Satisfaction: 92% program was clear; 88% well-organized, 72% reinforced knowledge regarding emergency response
Knowledge mean score unknown
93.5% important to respond during emergency (p < 0.05 with post)
94.0% responding will help the community (NS with post)
83.6% other PH nurses will respond during an emergency (NS with post)
91.8% they are themselves responsible to assist during emergency (NS with post)
26.2% intends to respond when needed (NS with post)
81.5% coworkers approve my role in emergency response (NS with post)
73.4% believes significant other approves role in emergency response (p < 0.05 with post)
|Rega P.P., 2013, USA, ||Pre- and post-exercise survey with 5-point-Likert-scale answer options about knowledge, satisfaction||Self-assessed knowledge: few participants reported ‘excellent’ knowledge||
Self-assessed knowledge: 96–100% improved knowledge
Satisfaction: ‘most participants’ valued the exercises for their usefulness and content; exercises are innovative, entertaining, educational; was recommended to be delivered to fellows.
|CBS||Richter J., 2005, USA, ||Post-test: survey collecting qualitative and quantitative data||-||
majority like small groups; all agreed the workshop would benefit their job. Majority information was useful and met expectations.
All agreed increased understanding of PH response protocols during outbreaks. 91% agreed better understood protocol during BT attack. Majority had gained greater understanding in public safety, law enforcement and security response protocols (97%) and a BT attack (94). 100% learned about others’ roles during outbreak, 96% during BT attack.
95% workshop enabled make contacts and network with staff with similar positions at different agencies. 97% had opportunity to network with different positions.
|Rottman S.J., 2005, USA, ||Pre- and post-test: survey||
Knowledge: 75,5% mean test score
Self-reported knowledge: no data available
87.4% mean test score (p < 0.001).
Known own and health department’s roles in disaster: increased (p < 0.001).
|Sandstrom B.E., 2014, Sweden, ||Observation of the exercises; evaluation seminars after the exercise||-||
Satisfaction/Methodology: applicability of the model was independent of the type of scenario, i.e. it could be used as a generic tool for exercises.; First test (3p): Mutual recognition of the scenario site benefitted participants; Discussion concerning preparedness took 1/3 of the exercise.; important having a highly professional exercise director; high flexibility of the concept; possibility to concentrate the exercise on a few cards without losing the strength of the exercise; participants considered it a cost-effective way of performing table-top exercises; suggested time-frames were removed. Second test (> 40 p): used as a tool to raise awareness; in a heterogeneous population overall understanding of different organizations needs and limitations is possible; not efficient for going into deeper detail in specific procedures and plans.; With a large audience, merely spectators, ineffective for experienced personnel; high flexibility of the concept supported conducting a multi-agency exercise on a more overarching level; allowed for giving different timeframes for various categories of participants; a few more topics were included.;|
Third test (35p in groups 5–7): the scenario was easily adapted to serve in an international emergency response exercise.; different phases of the response could be emphasized in depth by applying the Director card issues and the corresponding topics.; the tool supported emphasis on different areas of operations in specific phases.; participants confirmed the tests before and appreciated the simplicity of the exercise card concept; also usable for international cross-border exercises.
|Sarpy S.A., 2005, USA, ||
Pre-and post-test: survey with Likert-scale answer options on behavior, attitude)|
Post-test: additional Likert-scale questions on self-assessed objectives, satisfaction and open questions on most valuable and useable parts
Prepared to effectively respond to a SARS event: 3.98(1.52)
Community is prepared: 3.10(1.39)
Recognize a SARS outbreak: 4.53(1.40)
Establish contact and coordinate others: 5.02(1.18)
Maintain effective protocols for roles& responsibilities: 4.65(1.33)
Use the chain of command: 5.24(1.22)
Communicate relevant information in/externally: 5.37(1.20)
Determine communication to public: 4.76(1.51)
Determine communication to media: 4.63(1.51)
Monitor progress and action: 4.71(1.30)
Use investigation& management strategies: 4.88(1.31)
SARS is threat to my community: 5.65(1.15)
Prepared to effectively respond to a SARS event: 5.20(1.13) p < 0.001
Community is prepared: 3.93(1.26) NS
Recognize a SARS outbreak: 5.75(0.92) p < 0.001
Establish contact and coordinate others: 5.73(0.90) p < 0.001
Maintain effective protocols for roles& responsibilities: 5.32(1.05) NS
Use the chain of command: 5.80(0.93) p = 0.002
Communicate relevant information in/externally: 5.86(0.88) NS
Determine communication to public: 5.41(1.15) NS
Determine communication to media: 5.45(1.13) p = 0.004
Monitor progress and action: 5.41(1.00) p = 0.003
Use investigation& management strategies: 5.36(0.99) NS
SARS is threat to my community: 5.82(1.02) NS
Self-assessed knowledge: improved understanding of biological agents: 5.34/7
Improved understanding of functional roles & resp.: 5.57/7
Content: 5.88; format 5.77; mix of participant:5.86; pre-ttx lecture:6.36; ttx: 5.93; overall effectiveness: 6.12
Qualitative answers: most valuable: mix of participants, experts present, discussion sessions, networking possibility. Relevance to job because of: symptom identification, communication, in-& external partners. Improvements on: individual answers, missing perspectives, missing formalized introductions, packaging for greater dissemination.
|Savoia E., 2009, USA, ||Pre- and post-test surveys with Likert-scale answer options and % of substantive answers||
Between 25 and 85% substantive answers for different questions
See results post-test
Between 70 and 100% substantive answers for different questions
(p < 0.05 in all topic areas)
Legal authorities: Between the 32 and 46% difference between pre-and post-test (p < 0.05 for all topic areas)
Policies and procedures:
Between 30 and 43% difference between pre- and post-test (p < 0.05 for all but one topic (declaration of emergencies).
|Savoia E., 2013, USA, ||Guided group discussions where lists of recommendations were developed||-||
Training methodology: Include practitioners playing key leadership roles in the real world; representatives from agencies and disciplines across the range of jurisdictions that would respond to a specific public health threat; senior level players and specific agencies; a plausible scenario and timeline; clear an measurable exercise objectives; link exercises to prior years’ efforts and prior-tested capabilities; Expertise in exercise planning is a limited or unavailable resource in local health departments; heterogeneity in scopes has implications for the exercise design, required level of participation, and the approach to evaluation.
|TOT||Soeters H.M., 2018, Guinea & USA, ||Pre-and post-test with 30 MCP questions on knowledge. Post-test demonstration of skills||
Median score 17
Knowledge: median score 25
Skills: donning/doffing: 70% ‘acceptable’ score for HCW, 97% for supervisors, 83% for trainers
Chlorinated water preparation: 80% for HCW, 79% for supervisors, 81% for trainers.
|Taylor J.L., 2005, USA, ||Comments of participants during exercise; post-test: written evaluations (unknown methods)||-||
particpants realized needs to continue to build surge capacity specific to the challenges of an influenza pandemic; elected officials and decision makers must have clear understanding of the potential implications of an influenza pandemic and the additional efforts are needed to assure that such officials are adequately informed; pandemic influenza planning needs to be further coordinated with the existing emergency response infrastructure and additional training in incident command is needed; More detailed operational planning is required to achieve an effective overall response; additional support is needed at the federal level
|Umble K.E., 2000, USA, ||
Pre- and post-tests and follow-up (3 months):|
Knowledge 5 MCP questions; Agreement, Self-efficacy, Adherence, Setting factor, awareness using Likert-scales.
The model used to examine the program’s
effects was rooted in several health behavior
theories, including the health belief model,15,16
social cognitive theory,16–18 the transtheoretical
model,19,20 and the theory of reasoned action.
RR 196/470 for classroom|
Polio schedule 3.82/5 (1.40)
Attitude: Agreement with the polio schedule 33.80/40 (5.55)
Adherence to general recommendations: 11.50/15 (4.22)
Adherence to the polio schedule 9.30 (5.71)
Self-assessed skill: self-efficacy of the polio schedule 4.68/8 (2.09)
RR 116/251 for broadcast
Polio schedule 2.68/5 (1.61)
Attitude: Agreement with the polio schedule 32.37/40 (5.91)
Adherence to general recommendations: 9.50/15 (4.49)
Adherence to the polio schedule 11.84 (4.31)
Self-assessed skill: self-efficacy of the polio schedule 3.53/8 (1.99)
RR 196/470 for classroom|
Polio schedule 4.48/5 (0.89) (p < 0.001)
Attitude: Agreement with the polio schedule 38.38/40 (2.27) (p < 0.001)
Adherence to general recommendations:
Adherence to the polio schedule: unknown
Self-assessed skill: self-efficacy of the polio schedule 6.93/8 (1.84) (p < 0.001)
RR 116/251 for broadcast
Polio schedule 4.19/5 (1.16) (p < 0.001)
Attitude: Agreement with the polio schedule 37.37/40 (3.33) (p < 0.001)
Adherence to general recommendations:
Adherence to the polio schedule: unknown
Self-assessed skill: self-efficacy of the polio schedule 6.55/8 (1.96) (p < 0.001)
Significant (p = 0.006) higher increase in knowledge for classroom education compared to broadcasts.
RR 196/470 for classroom|
Polio schedule 4.52/5 (0.85) (p < 0.001 with pre)
Attitude: Agreement with the polio schedule 37.36/40 (2.71) (p < 0.001 with pre)
Adherence to general recommendations: 12.38/15 (3.36) (p < 0.05 with pre)
Adherence to the polio schedule 16.25 (4.90) (p < 0.001 with pre)
Self-assessed skill: self-efficacy of the polio schedule 6.88/8 (1.64) (p < 0.001 with pre)
RR 116/251 for broadcast
Polio schedule 4.24/5 (1.07) (p < 0.001 with pre)
Attitude: Agreement with the polio schedule 37.01/40 (3.60) (p < 0.001 with pre)
Adherence to general recommendations: 11.84/15 (4.31) (p < 0.05 with pre)
Adherence to the polio schedule 14.42 (5.44) (p = 0.084 with pre)
Self-assessed skill: self-efficacy of the polio schedule 6.55/8 (1.96) (p = 0.001 with pre)
No significant difference between classroom and broadcast
|Waltz E.C., 2010, USA, ||
ARS: post-test evaluation form|
Broadcasts: # views
Web-based lecture: post-test evaluation form
ARS RR 93/93 (100)|
Satellite broadcast RR none
Web-based education RR 20.000/44.000 (48)
ARS: 95% agreed that the technology was beneficial to the training;
Broadcast: 3871 views on average where in three years for some courses the number of later views exceeded the live views.;
Web-based education: 96% rated course quality as good, 99% would recommend the course, 88% the course would help perform their job more effectively
|Wang C., Wei S., Xiang H., Wu J., 2008, China, ||
Pre-, post- and follow-up (12 months) tests of 40 MCP questions (knowledge) and 5-point-Likert scales for skills.|
Observation by colleagues, and evaluation of a subsequent real outbreak.
RR 41/43 (95)|
Knowledge: mean score 21.62 +/− 6.37
Self-assessed skills: mean scores between 2.54–3.05
RR 41/43 (95)|
Knowledge: mean score 31.59 +/− 5.85 (p < 0.01 with pre-test)
Self-assessed skills: mean scores between 3.49–4.12. Increase in mean scores for all individual skills (p < 0.01)
Satisfaction/ Training methodology:
90% thought the training to be excellent. 98% very satisfied with venue, logistics, communication.
RR 41/43 (95)|
Knowledge: mean score 32.39 +/− 5.15 (p < 0.01 with pre-test)
Increase p < 0.05 for assessment knowledge post vs. follow-up test
Self-assessed skills: mean scores between 3.68–4.07. Increase in mean scores for all individual skills (p < 0.01) compared with pre-test
|Wang C., Wei S., Xiang H., 2008a, China, ||pre- and post and follow-up (12 months) test of 30 MCP questions on knowledge (objective)& 8 questions on attitude and behaviroal intentions (subjective); during training semistructured interveiws on training method||
Knowledge: mean score 19.79 +/− 2.41
Assessment: 2.77 (0.81);
Policy development: 2.11 (0.69);
Communication: 2.68 (0.78);
Cultural competency: 2.55 (0.96);
Community dimensions of practice: 2.82 (0.73);
Basic PH sciences: 2.68 (0.72);
Financial planning & management:2.32 (0.89);
Leadership & system thinking: 2.86 (0.99).
Knowledge: mean score 24.49 +/− 0.86 (p < 0.001 with pre-test)
Assessment: 3.69 (0.61);
Policy development: 3.95 (0.51);
Communication: 3.95 (0.51);
Cultural competency: 3.95 (0.69);
Community dimensions of practice: 3.84 (0.59);
Basic PH sciences: 4.11 (0.45);
Financial planning & management: 3.47 (0.82);
Leadership & system thinking: 3.89 (0.55);
All p < 0.05 compared with pre-test.
Satisfaction: 96% thought the methods good/excellent; 100% thought the training innovative;
Knowledge: mean score 24.24 +/− 1.58 (p < 0.001 with pre-test)|
Assessment: 4.35 (0.72) p < 0.05 compared with post-test;
Policy development: 2.94 (0.55) p < 0.05 compared with post-test;
Communication: 3.82 (0.61);
Cultural competency: 3.56 (0.49);
Community dimensions of practice: 3.99 (0.51);
Basic PH sciences: 3.74 (0.69);
Financial planning & management: 2.66 (0.74) p < 0.05 compared with post-test;
Leadership & system thinking: 3.82 (0.62).
All p < 0.001 compared with pre-test
|Wang C., Xiang H., 2010, China, ||pre- and post and follow-up (12 months) test of 30 MCP questions on knowledge (objective)& 8 questions on attitude and behaviroal intentions (subjective); during training semistructured interveiws on training method||
Knowledge: mean score 18.50 +/− 3.23
Assessment: 2.54 (0.76);
Policy development: 2.33 (1.06);
Communication: 3.16 (0.84);
Cultural competency: 2.26 (0.76);
Community dimensions of practice: 2.69 (0.81);
Basic PH sciences: 3.12 (0.93);
Financial planning & management: 2.07 (1.03);
Leadership & system thinking: 2.71 (0.99).
Knowledge: mean score 22.78 +/− 1.14 (p < 0.001 with pre-test)
Assessment: 3.91 (0.65) p < 0.05;
Policy development: 3.48 (0.70) p < 0.05;
Communication: 4.13 (0.65) p < 0.05;
Cultural competency: 3.44 (0.61) p < 0.05;
Community dimensions of practice: 3.87 (0.73) p < 0.05;
Basic PH sciences: 4.69 (0.49) p < 0.05;
Financial planning & management: 3.26 (0.74) p < 0.05;
Leadership & system thinking: 3.05 (0.69). NS
Satisfaction: 92% thought training methods excellent, 96% satisfied with trainers’ performance, 89% training approach scientific and feasible, 99% very satisfied with venue, training, logistics.
Knowledge: mean score 22.69 +/− 2.49 (p < 0.001 with pre-test)
Assessment: 4.46 (0.73) p < 0.05 with pre- and post-test;
Policy development: 2.82 (0.82) NS;
Communication: 4.27 (0.61) p < 0.05 with pre-test;
Cultural competency: 3.21 (0.79) p < 0.05 with pre-test;
Community dimensions of practice: 3.79 (0.65) NS;
Basic PH sciences: 4.35 (0.54) p < 0.05 with pre-test;
Financial planning & management: 2.79 (0.92) NS;
Leadership & system thinking: 2.84 (0.77) NS.
|Yamada S., 2007, Hawaii/USA, ||
Post-test: course evaluation questionnaires, interviews with participants
83/85 thought the case appropriate to own setting; 84/85 liked group work; 80/86 thought mixed groups to be helpful for learning; 84/85 thought the case overall worthwhile as a continuing education activity. Suggestions for better case: longer time to analyze the case and prepare for the discussion on the learning issues; provide pictures. Suggestions for the facilitator: more in-depth knowledge, all members included in discussion
|Yellowlees P., 2008, USA, ||Post-test: survey with 4 questions and 5-point-Likert-scale answer options + open ended questions||-||
3.95 practicality of the program
4.5: time well spent
4.3: met the objectives.