Table 2 Results
Study (1st Author, year, country, titel) | Evaluation method | Pre-test | Post-test | Follow-up | |
|---|---|---|---|---|---|
Ablah E., 2007, USA, [21] | Online questionnaire with 5-point-Likert-scale answer options | RR 56/65: 86% Self-assessed skills: - to identify the need for and implement surge capacity: 43% good / excellent - Participate in a coordinated response: 32% good / excellent - To implement your risk communication skill set: 12% - To identify and locate your agency infectious disease resrouces: 49% | RR 48/65: 74% Self-assessed skills: - ability to identify the need for and implement surge capacity: 74% good / excellent (p = 0.003) - Participation in a coordinated response: 60% good / excellent (p = 0.017) - To implement your risk communication skill set: 25% (not sign) - To identify and locate your agency infectious disease resrouces: 60% (not sign) | – | |
Ablah E., 2008, USA, [22] | < 4 weeks after the exercise, focus groups were held, transcribed and main themes identified | - | RR unknown Self-reported skills: improvements in surge capacity, coordination between counties, risk communication, and awareness of protocols and procedures; better able to (be) support (ed), they see necessity of a coordinator, effective relation building. Satisfaction/ training content/methodology: The exercise format was liked. | - | |
Aiello A., 2011, Canada, [23] | Retrospective pre-test & post-test: questionnaire with 5-point-Likert-scale answer options | RR 1250/1020 (82) Self-assessed skills: 35% felt prepared to deal confidently with the situation during a pandemic | RR 1250/1020 (82) Response Rate Self-assessed skills: 76% felt prepared to deal confidently with the situation during a pandemic. (p = 0.0020). Satisfaction “a high proportion” thought the training relevant for work- & personal life, useful, helpful and informative. | - | |
Alexander L.K., 2005, USA, [24] | Pre-test and post-test on knowledge with MCP questions after each online module; course evaluation after 5th module with 4-point-likert-scale and open answer options. | 55/80 (69) Knowledge results not available | 55/80 (69) Knowledge results not available Satisfaction: 100% recommended course to others; 80% strongly intended to use the information gained; 100% agreed that lectures (100%), readings (94%), activities (98%), pre-and post-test(94%), and face-to-face module (95%) helped their learning. 33% would like online interaction. | - | |
Alexander L.K., 2008, USA, [25] | Pre-test: retrospectively in post-test; questionnaire with 5-point Likert-scale answer options. Post-test: questionnaire with 5-point-Likert-scale answer options | RR 177/156 (88) Self-assessed skills: - Use reports to identify health issues: 2.67 - Communicate with other agency to identify new cases: 3.04 - Maintain security and confidentiality of personal and PH information: 3.96 - Stay informed on PH laws and regulations: 2.77 - Use regulations to promote health: 2.69 - Recognize a disease outbreak: 2.79 - Collect biological or environmental samples: 2.30 - Be aware of amount of each important health problem: 2.76 - Work as part of a team during outbreak investigation: 2.77 - Write a press release: 2.06 - Create a line listing: 2.04 - Create an epidemic curve: 1.81 | RR 177/156 (88) Self-assessed skills: - Use reports to identify health issues: 3.62 (p = < 0.0001) - Communicate with other agency to identify new cases: 3.87 (p = < 0.0001) - Maintain security and confidentiality of personal and PH information: 4.31 (p = < 0.0001) - Stay informed on PH laws and regulations: 3.63 (p = < 0.0001) - Use regulations to promote health: 3.55 (p = < 0.0001) - Recognize a disease outbreak: 3.89 (p = < 0.0001) - Collect biological or environmental samples: 3.17 (p = < 0.0001) - Be aware of amount of each important health problem: 3.49 (p = < 0.0001) - Work as part of a team during outbreak investigation: 3.89 (p = < 0.0001) - Write a press release: 3.28 (p = < 0.0001) - Create a line listing: 3.76 (p = < 0.0001) - Create an epidemic curve: 3.69 (p = < 0.0001) Satisfaction: 99% would recommend it; 96% thought the course was excellent; 100% inteded to use the information in their jobs; 100% said the material met the objectives; 98% thought that both internet, activities and case-studies helped their learning; 91% thought the face-to-face module helped. 4% thought the material did not fit together. | - | |
Araz O.M., 2012, USA, [26] | Post-test survey with 5-point Likert-scale answer options and open ended questions | - | RR Unknown Satisfaction/ training methodology 66% found the multimedia presentations very helpful in terms of understanding the state of the outbreak; 69% agreed that the displayed scenarios were realistic and plausible; 84% agreed that the information exchange during the exercise was in high quality, which means that the decision-making environment facilitated communication.; “most particpants” agreed that the presented method increased readiness for a pandemic. “participants” stated that the video scenarios and the simulation model made it very real. | - | |
Araz O.M. & Jehn M., 2013, USA, [27] | Pre- and post-test: 3-point-Likert scale answer options | RR 109/121(88) Self-reported capability: - Leadership and management: 93.5% - Mass care: 84.1% - Communication: 84.7% - Disease control and prevention: 91.5% - Surveillance: 82.0% | RR 109/121(88) Self-reported capability: - Leadership and management: 99.0% (p = 0.010) - Mass care: 85.6% - Communication: 94.1% (p = 0.001) - Disease control and prevention: 91.0% (p = 0.010) - Surveillance: 90.0% (p = 0.010) | .- | |
Atack L., 2008, Canada, [28] | Pre- and post-test (< 2 weeks after): questionnaire with Likert-scale answer options and open-ended questions, adapted from validated questionnaire. Four validated questionnaires on organizational climate, receptivity, and course satisfaction: - Social System and the Organization (SSO) (climate) - New Ideas and You (NIY) (receptivity) - Education Program and Change (EPC) (feasibility) - Continuing Professional Education (CPE) (course satisfaction) | RR 67/76 (88) Knowledge test: Mean score of 38.4/56 | RR 67/76 (88) Knowledge test: mean score of 46.4/56 (p = < 0.001) Knowledge & the system SSO: 75.6%; NIY: 85.6% EPC: 86.3% CPE:90.4% Relation between test and postcourse competency: SSO, NIY and CPE not sign. EPC: r = 0.431 p = 0.001. (80) Satisfaction: 100% agreed that the course was extremely helpful, 95% creative. 100% would recommend online learning as a way to learn about infection control; 27% thought video demonstration was in important to learning. Suggestions for approval: function to skip video’s (11%); not to re-do the entire module after failed for postquiz (7,5%). Open ended questions how the course had been useful: three major themes: improved hand hygiene practice, improved teaching to patients/visitors/staff on how to use PPE, improved their own techniques RR 55/76 (72) Behavioral change: reported an example of behavioral change after the course. | - | |
Atlas R.M., 2005, USA, [29] | Scored evaluation and anecdotal comments | - | RR unknown Participants regularly identify the moulaged patients as the most effective element of training. Self-assessed attitude: 94% say use of patient simulator changed their awareness of respiratory disease transmission, Self-assessed Behavior: 98% would alter respiratory protection when confronted with a patient demonstrating signs and symptoms of a respiratory tract infection. | - | |
Baldwin K., 2005, USA, [30] | Post-test: 5–10 MCP knowledge questions after each module. + unknown qualitative evaluation method for satisfaction | - | RR 15/15(100) Knowledge Results not reported Satisfaction/training methodology: Challenges: Overly complex system of accessing different parts of the program; impossible to predict large number of updates required; internal shift of focus could not be implemented; funding to sustain the program was hard; lack of designated ownership of the program. Benefits: Use of an readily available system, accessible at different sites; completed at own pace and role; evaluation of compliance could be monitored; individuals could be identified for additional training. | - | |
CBS | Bazeyo K.M., 2015, Uganda, [31] | Pre- and post- knowledge test. Format unknown. | RR 330/unknown Knowledge: Mean score 5–20% lower than in post-test for all districts. No numerical results available. | RR 330/unknown Knowledge: Mean score 5–20% higher than in the pre-test for all districts. No statistical results available. | – |
Becker K.M., 2015, Ghana, Uganda, Nigeria, USA, [32] | Post-test: summing up achievements | - | RR 43/43(100), two years. System performance: Built in-country cadres of midlevel public health; cross-border rabies vaccination program; evaluation of 66 disease surveillance systems; integrated vaccination campaigns & improved coverage rates; increased capacity for epidemiological studies, provide an evidentiary basis for PH action/strategy; influence PH policy; improved diagnostic and laboratory management skills; participants trained district level laboratory personnel in integrated disease surveillance. | - | |
TOT | Berrian A.M., 2018, USA, [33] | Pre- and post-test: Written MCP knowledge test on content; Post-test: written self-report (4-point Likert-scale) of knowledge, skills, pedagogical skills, leadership skills Follow up: - | RR 10/10(100) Knowledge: No data Self-assessed knowledge: No data Self-assessed skills: No data | RR 10/10(100) Knowledge: Mean increase of 17% p = 0.0078; Self-assessed knowledge: Improved on pathogen transmission (p = 0.0156), risk mitigation (p = 0.0020) and effective pedagogy (p = 0.0020). Self-assessed skills: Improved for risk assessment p = 0.0078; facilitate a workshop (p = 0.0020). | Results from the second wave were part of the train-the-trainer session, and are not presented because they were non-professionals. |
Biddinger P.D., 2010, USA, [34] | Pre- and post test of knowledge and confidence; Post-exercise survey to assess awareness of agencies’ roles and responsibilities; AAR | RR 1145/5892 (participants of table-top exercises) Knowledge: no data available; Attitude: no data available. | RR 1145/5892 (participants of table-top exercises) Knowledge: increased with 25% on average. 56% increase knowledge of what other agencies may have to offer (resources/assets); 69% clarify role&responsbility of perosn & organizaiton; Attitude: On average 12% better confidence. Satisfaction/ training methodology:77% scored it effective in practicing working together; 73% providing an opportunity to evaluate plans/procedures; Satisfaction for target groups: Regional response exercise respondents reported higher satisfaction with the effectiveness of understanding roles (p < 0.001), providing right environment (p < 0.001), promoting regionwide cooperation and mutual aid (p < 0.001); more engaged in the exercise (p < 0.006). No relation in knowledge and satisfaction with the program. Small and big towns had higher satisfaction. Network building: 70% assessing connectivity within/across agencies; 53% promoting region-wide cooperation and mutual aid. | - | |
Cathcart L.A., 2018, USA, [35] | Pre-, post- and follow-up (timing unknown): survey with 5–8 statements rated with a 5-point-Likert-scale answer options. | 120/120 (100), Self-assessed skills Mean score 2.2 | 120/120 (100), Self-assessed skills Mean score 4.0 | 59/120 (49) Self-assessed skills Mean score 4.3 Self-assessed behavior: Used what I learned: mean score 4.2 +/−1.0 Different way of conducting my job: 3.9 +/− 1.0; Satisfaction: Prepared me for my job: mean score 4.0 +/− 1.0 | |
Chandler T., 2008, USA, [36] | Knowledge test of 15 questions (unknown methodology) Self-assessed knowledge on a 4-point-Likert-scale Skills and behavior assessed by local supervisors (unknown methodology) | 817/> 817 Knowledge: mean score 72.17 (SD 16.31) | RR 817/> 817 Knowledge: Mean score 94.25 (SD 8.07) (p < 0.001). RR 764/> 817 Self-assessed knowledge: 86% felt more knowledgeable about the basic emergency preparedness core competencies; 82% about their agency’s chain of command during an emergency response; 80% about their own functional roles during emergency reponse. RR 511/> 817 Skills No data available RR 511/> 817 Behavior Supervisors: usage of the course was an effective means for improving work performance. | - | |
Chiu M., 2011, USA, [37] | Survey in which self-assessed confidence in preparedness, response and recovery skills and needs were rated using a 5-point-Likert-scale | 41–54/182 (23–30) Self-assessed skills: 10 preparedness competences: 30.2 SD6.7 8 response competences: 26.0 SD5.5 7 recovery competences: 23.1 SD5.5 Training needs Preparedness: 32.7 SD7.3 Response: 26.0 SD6.9 Recovery: 22.6 SD6.5 | 41–54/182 (23–30) Self-assessed skills: 10 preparedness competences: 36.2 SD4.9 (p < 0.001) 8 response competences: 30.9 SD4.5 (p < 0.001) 7 recovery competences: 28.8 SD4.18 (p < 0.001) Training needs Preparedness: 22.0 SD9.5 (p < 0.001) Response: 17.1 SD7.3 (p < 0.001) Recovery: 15.7 SD6.6 (p < 0.001) | - | |
Craig A.T., 2007, Australia, [38] | Post-test: feedback from stakeholders in debriefing sessions; state-wide participant questionnaire with 5-point-Likert-scale answer options; both retrospective pre-test and post-test. | RR unknown Self-assessed system performance: Prepared to respond to a case of pandemic influenza: mean score 3; > 55% agreed or strongly agreed. | RR unknown Self-assessed system performance: Prepared to respond to a case of pandemic influenza: mean score 4; > 90% agreed or strongly agreed Satisfaction: 84% agreed that exercise Paton prompted their influenza pandemic planning. | - | |
Dausey D.J., 2007, USA, [39] | Review of after action reports, participants post-exercise evaluations, and internal team discussions and consensus following exercise debriefings | - | RR 513/unknown Exercise methodology: Exercises should a) be designed to achieve a specific objective. B) be as realistic as possible & logistically feasible; C) be designed around “issue areas” rather than scenarios. E) have forced, targeted and time delineated decision making. F) have limited number of participants. G) be designed and executed from collaborative engagement of representatives from participating agencies and external developers and facilitators. | - | |
CBS | Dausey D.J., 2014, USA, [40] | Post-test: evaluation form with Likert-scale answer options, some with open-ended questions Follow-up: semi-structured interviews with health leaders at ministry level | - | Unknown/558 participants; Satisfaction/methodology: 88–100% rating the quality of the exercise as high; 92–94% exercise helps to understand roles and responsibilities of organizations responding to influenza pandemic; 50–73% ability to identify key gaps in performance.; more sectors should be included, including more private partners and NGOs; better grounding of theoretical responses with practical responses are required. Self-assessed knowledge: 82–100% gain knowledge that they plan to improve the preparedness of their organization; Self-assessed attitude: the exercise raised awareness and understanding about public health threats; | unknown 137 observers Attitude Participation in the exercise helped to motivate to develop an exercise program and regularly assess different aspects of their public health preparedness. Barriers are a lack of financial resources, limited support among leadership to develop and sustain the program. Behavior: Most countries reported modifying and using exercise templates. |
Dickmann P., 2016, Hong Kong, Poland, Sweden, Switzerland, UK, [41] | Pre-and post-questionnaires; day assessments at the end of each day | RR 15/15 | RR 15/15 Satisfaction: 14/15 expectations have been fully met; they appreciated that the training was based on a reflective and reframing approach Self-assessed knowledge: 14/16 stated their understanding had increased considerably | - | |
Self-assessed knowledge: 50% good knowledge of risk communication theory. | Self-assessed attitude: the part that had good knowledge, felt better prepared to advocate for this change | ||||
CBS | El-Bahnasawy M.M., 2014, Egypt, [42] | Pre- and post- knowledge tests (methodology and scales unknown) | RR 125/unknown Knowledge: median scores: Anthrax 3a; Tick-borne relapsing fever: 2a; Lice-borne relapsing fever: 2a; Clonorchis sinensis:4a | RR 125/unknown Knowledge: median scores: Anthrax 8b; Tick-borne relapsing fever: 4b; Lice-borne relapsing fever: 4b; Clonorchis sinensis: 9b Satisfaction/methodology: results useless because no information about training methods is provided. | RR 125/unknown Knowledge: median scores: Anthrax 9c; Tick-borne relapsing fever: 5c; Lice-borne relapsing fever: 5c; Clonorchis sinensis: 9b (p < 0.001 for all subjects) |
TOT | Faass J., 2013, USA, [43] | Post-test: Partly hardcopy, partly online survey with 29 open-ended and 5-point-Likert-scale questions | – | RR 120/231(52): Satisfaction: 90% of attendees rated the overall impression as “good” or higher. 94% answered “agree”/ “strongly agree” on relevance of course materials to their organizations and employees. Attitude: 84% agreed that pandemic preparedness posed a challenge for the transit industry; | Results from the second wave of training: - |
Fowkes V., 2007, [44] | Post-test evaluation form with 5-point-Lickert-scale answer options and open-ended question: How would you incorporate what you have learned? | - | RR > 6000/9537 Satisfaction/methodology: 95% rated modules as good/excellent Self-assessed attitudes: modules reinforced needs for emergency plans; enhanced awareness of possible unusual clinical presentations; enhanced consciousness about infection control. Self-assessed knowledge: modules provided useful approaches to decontamination; | - | |
Fowkes V., 2010, [45] | Evaluation forms; assessment of the departments’ emergency plans; AARs completed by the exercise groups; debriefing of coordinators; interviews with trainees; follow-up interviews; quarterly reports by the organizing party | RR 1176/1496 System results: 37% of emergency operation plans complete | RR 1176/1496 System results: 46% of emergency operation plans complete Satisfaction/methodology: 92–98% rating the training experiences as good to excellent.; Self-assessed knowledge/skills: 92–98% rates knowledge and skills gained form the exercises as good to excellent.; Skills: 72–90% were able to describe events and steps necessary to activate their plan, actions that should be taken when it is activated, roles of individuals, internal and external communications needed, how to participate in a coordinated response, and who is responsible for the oversight of the plan. 69% how to correct the plan when needed; 62% to lessen the spread of disease to staff, patients, and families. 42% to plan for a surge of infectious patients. | RR 1176/1496. System results: 74% of emergency operation plans complete; 91% of clinics made improvements in their emergency plans compared to post-test with statistically significant changes in 2/3 of the 15 criteria (p = 0.001–0.46). | |
Gershon R.R., 2010, [46] | Pre- and post-test of 7 questions on knowledge and 9 items course evaluation. | RR 129/unknown Knowledge: mean 6.3 +/− 1.1 | RR 129/unknown Knowledge: mean 6.6 +/−0.8 (p < 0.001) Satisfaction: 98% thought the program was valuable; 97% risk was addressed; 95% length of training is acceptable. Self-assessed knowledge: 95% their PPE understanding was reinforced; > 92% knowledge of respiratory illnesses improved; | - | |
TOT | Grillo M., 2017, [47] | 1 h pre- and post test with 40 MPC and true/false questions testing knowledge. | 136/136 (100) Knowledge: 50.5% mean score | 136/136 (100) Knowledge: 67.9% mean score (p < 0.001) | Results from the second wave of training: - |
Hegle J., 2011, [48] | Observation by at least 2 researchers using an observation guide; semi-structured interviews with exercise leaders; review of planning and exercise materials | Unknown/682 System performance: Direct result of the exercise: 1) building relationships among response partners across counties and agencies; enhancing social capital. 2) promoting visibility of public health and assets in an emergency response; 3) testing multiple communication systems. The use of these systems was problematic in all exercises observed; 4) training public health practitioners in exercise evaluation. | - | ||
Hoeppner M.M., 2010, USA, [49] | At 6 and 12 months after participation in one of the courses, a survey with eight questions. Skills/knowledge gained, changed attitudes, changed behavior, affected others, face barriers. Based on the Kirkpatrick model | - | - | RR 244/387 (63) Satisfaction: 6- months: 64% agreed that courses helped them develop the competencies; 12 months: 63.5%. Self-assessed knowledge/skills: Development of critical thinking and systems-thinking. Deeper appreciation for the complexity of planning and response and the need for collaboration across all agencies and levels of government. Recognised the national incident management system as a organizing mechanism. Better able to gauge the resources needed and better advocate for these. Better able data collection; better able to inform and educate others. Self-assessed attitude: Recognized the need for improved surveillance, use of technology to organize data, shared data management across agencies. Behavioral change: Performed better data collection; inform and educate others; developed wider range of exercises, revised emergency plans, enhanced and eexpanded cross-agency collaboration. Barriers: financial and human resources, lack of time, unavailable software, how to convince others?, underutilization of expertise due to changing jobs. | |
Horney J.A., 2005, USA, [50] | Knowledge post-test after each module; survey; | - | 416/3030 (14) Knowledge: no results available Satisfaction: 98% indicated that the training module provided the information they were looking for (100% for PH nurses); 62% had recommended the course to collegues. Self-assessed knowledge: 2/3 module introduced them to terms and concept they were previously unfamiliar with; 87% module clarified terms or concepts they had not enough information about; 97% module reinforced familiar terms and concepts; 81% module specifically adressed prof roles and responsibilities (80% for PH nurses); Self-assessed attitudes: 92% module made them feel better equipped to do their job (94% for PH nurses); | - | |
Hueston W.D., 2008, USA, [51] | Representatives of partner universities shared their lessons learned. | - | - | Unknown Several years after the start of the program: Methodology: Strong collaborations and good communication are essential; differences in DVM and MPH professional cultures must be understood; differences in CVM and SPH organizational cultures must be bridged; human and financial resources provide significant challenges; faculty efforts systems may differ; different curriculum design and delivery call for innovative approaches; CVM/SPH partnerships are mutually beneficial; Technology is not a panacea - developing good distance learning courses takes a lot of time and money; no single standard for distance learning exists to unify the competing software platforms and various computer operating systems.; be prepared for succes. | |
Johnson Y.J., 2009, USA, [52] | Post-exercise evaluation form with 11 five-point-Likert-scale questions | - | RR Unknown Satisfaction/ Training methodology: Exercise was well structured and organized (4.0); scenario was plausible & realistic (4.14); knowledgeable facilitators about (4.14); facilitators kept exercise on target (3.86); technologies were useful (4.14); participation was appropriate for my position (4.00); correct mix of people (3.79); adequate facilities utilized (4.36); meals and breaks (4.71); Self-reported skill: could practice and improve responsiveness (3.86); after the exercise my agency/jurisdiction is better prepared to successfully deal with the scenario (4.00). | - | |
Kohn S., 2010, USA, [53] | Observation. Post-test: evaluation sheet with Likert-scale answer options; informal conversations with participants and trainers. | - | RR 213/unknown 80% agreed enough interactive exercises; 89% that session provided useful information now/ in near future; 90% that overall session was valuable.; conversations identified: participants saw the course as effective and helpful in their professional setting; overwhelming volume of information; wish for more examples/ scenarios to illustrate practical application. ‘Many’ thought the material as not relevant to them. INFLUENZA TRAINING: RR unknown/37 Positive ratings of course content, facilitator, and presentation; information provided helped frame their emergency response roles & responsibilities within a PH context; training is essential in clarifying some response activities; core concepts taught help operate more effectively during a variety of future emergencies; still the course material could have been better tailored. | - | |
TOT | Livet M. 2005, USA, [54] | Participants surveys using 6-point-likert scales; observation forms; and evaluation instrument for a TTX | 67–70/80 (84–88) Networking/relationship building: 1.60 – 4.10 on a 6-point scale Competence: Describe PH role during emergency response: 4.09 Describe chain of command: 4.58 Identify response plan: 5.06 Describe functional roles: 5.07 Describe communication roles: 4.61 Identify limits to own competence: 4.48 Recognize unusual events: 4.66 Apply creative problem solving and evaluate effectiveness: 4.62 Plan a TTX: 3.84 Implement a TTX: 3.80 Evaluate a TTX: 3.76 Write an AAR: 3.72 Describe the ICS: 4.62 Describe risk communication strategies: 4.04 Describe criminal and epidemiological investigative methods: 3.84 | RR 67–70/80(84–88) Satisfaction: Organization of the sessions and TTX 60–90% thought it good-excellent. Usefulness and applicability 60–84% agreed or strongly agreed. Networking/relationship building: 4.91–5.67 on a 6-point scale (p < 0.001) Competence: Describe PH role during emergency response: 5.30 (p < 0.001) Describe chain of command: 5.35 (p < 0.001) Identify response plan: 5.57 (p = 0.001) Describe functional roles: 5.49 (p < 0.001) Describe communication roles: 5.31(p < 0.001) Identify limits to own competence: 5.30 Recognize unusual events: 5.40 (p < 0.001) Apply creative problem solving and evaluate effectiveness: 5.26 (p < 0.001) Plan a TTX: 5.44 (p < 0.001) Implement a TTX: 5.42 (p < 0.001) Evaluate a TTX: 5.36 (p < 0.001) Write an AAR: 5.23 (p < 0.001) Describe the ICS: 5.46 (p < 0.001) Describe risk communication strategies: 5.29 (p < 0.001) Describe criminal and epidemiological investigative methods: 5.20 (p < 0.001) | Results of the second wave of training: - |
Macario E., 2007, USA, [55] | Broadcast evaluation through a survey and knowledge post-test for those who wished further education; Tabletop evaluation through participant & facilitator surveys; telecall interview | - | RR: broadcast 821/25000 RR: tabletop 164/unknown RR knowledge test: 735/25000 RR teleconference: 21/unknown Satisfaction: 75–98% agree on satisfaction statements for the broadcast; 4.1/5 tabletop is helpful in feeling more prepared; 4.1 for tabletop overall. 2 h was too short for real effectiveness. Tabletop helped to learn the language of such preparedness activities and identified major gaps; 4.1 on how deliverables helped them in familiarizing themselves with pp. plans; 3.7 for broadcast info; Survey tabletop facilitators: 4.7 program was appropriate for their health departments; 4.3 training helped increasing capacity to respond; program helped to identify key deficiencies in preparedness, as well as to learn what is and is not in their control. Future tabletops address more localized concerns and have materials earlier available; Knowledge: 90% had 100% score; survey tabletop particpant: teleconference satisfaction, skill, confidence: pairing of broadcast with tabletop was effective complementary training; training helped them better understand their plans, roles of decion makers in the chain of command; | - | |
CBS | Martin G., 2018, Ireland, [56] | Lessons learned in hot debrief + observers/evaluators during the exercise | Unknown/> 200 Methodology: Exercise planning should not be overly ambitious. If testing too many facets of the emergency response protocols, the public health response can be deprioritised; the practical implementation of communication protocols in a real time exercise of this scope proved challenging; the roles and responsibilities of the various agencies involved needs to be clear. In the chaos of an incident it is easy for role confusion to set in; equipment and infrastructure must be in place and must have been thought about before an actual incident (wheter ore not cell phone signals are available on site or requiring boosting e.g.). | ||
Mitka M., 2003, USA, [57] | Comments of participants | - | Reaction: exercise helped to improve the nation’s response by testing emergency plans and finetune them. | - | |
Morris J.G., 2012, USA, [58] | Post-test: survey 6 open-ended & 10 questions with Likert-scale answer options | - | RR 22/unknown Satisfaction/methodology 22/22 scenario plausible, 20/21 scenario comprehensive, 22/22 scenario generated productive discussion, 22/22 scenario helped identify strength/weaknesses, 17/22 scenario helped identify gaps in current planning, 19/21 helped build relationships with participant from key agencies, 16/20 relationships with participants from other states Knowledge 22/22 knowledge of this type of emergency event has increased, 17/22 understanding of my role has increased, 22/22 my understanding of others’ role has increased. | - | |
Olson D., 2008, USA, [59] | Post-test: Course evaluations (no results) Follow-up: 6 & 12 months after end of program (no results) Testimonials (unknown method) Based on the Kirkpatrick Model | - | RR unknown/1680 Satisfaction: “I would not be in the position I am today if I had not had the opportunity to advance my knowledge in their uniquely structured program” “It has helped me in everything I do related to emergency preparedness!” Behavior: “I have reviewed and updated our plans. I have trained staff on the topics I studied. We have addressed mental health needs more in our planning and training.” Self-assessed knowledge: These courses have made me understand what other training I need as well as training needed by other staff and partners in preparedness. “Gave me more knowledge about writing plans, designing plan exercises, working with media and developing media kit, discussing mental health issues with local providers, planning for emergencies with community leaders. | - | |
TOT | Orfaly R.A., Frances J.C., 2005, USA, [60] | Evaluation of trainings performed by the TRAINED-TRAINERs 6 months after the TTT: Survey; Open-ended questions; registered number of trainings and people trained in second wave; follow-up interviews with TRAINED-TRAINERs | - | - | TRAINED-TRAINERs RR: 21/21 (100) Participants RR unknown Behavior: 118 participants trained; 20% of TRAINED-TRAINERs had conducted a training themselves after 6 months. Barriers were: lack of time&resources; not enough confidence; trainings were too general and did not provide enough information specific to particpants’ funcitonal roles Satisfaction: high rankings for the instructors (4.5/5); similar high scores for knowledge, responsiveness and organization; content and materials 4.0; self reported findings on training objectives:> 4.0 for four of the trainers and 3.5 for the fifth; qualitative evaluation of participants were favorable; |
Orfaly R.A., Biddinger P.D., 2005a, USA, [61] | Evaluation survey (unknown method) | - | 24/24 (100) Satisfaction: 96% would recommend the course to classmates; 92% the course is superior to other courses they had taken at HSPH; 100% the course is useful to their profession; 83% there was appropriate emphasis on practical skills. | - | |
TOT | Otto J.L., 2010, USA, [62] | Post-test: survey with MCP/Lickert scale questions; Follow up in six months with a survey with MCP and open questions on implemented trainings, revised plans, and barriers | - | RR 65/85 (76) Satisfaction/methodology: 95% expectations & instructions clearly presented, 92% scenario is realistic and credible, 87% TTX better equiped them to plan and execute an installation-level pandemic influenza exercise. 89% TTX identified strengths and gaps in response, 77% identified consequences of interventions and described strategies for dealing with them. 94% TTX identified opportunities for enhanced military and civilian coordination. | 50/85 (59) Behavior: 68% had a new/revised pandemic influenza plan, 44% my installation conducted a pandemic influenza exercise. Barriers: 67% competing priorities, 37% time limitations, 19% personnel limitations, 11% budget limitations, 37% other. |
Peddecord K.M., 2007, USA, [63] | Pre-post- and 6-week follow-up: questionnaire with MCP, likert-scale and open-ended questions. Pre-and post on knowledge Follow-up: attitudes + behavior | 520/> 1658 (< 31) Knowledge: mean score 81% Attitudes: 76% thinks mass vaccination clinics are necessary and beneficial 8% thinks it is a good idea but not that important in the overall PH system 10% feels intimidated by the scope& responsibilities 3% ambivalent 3% no opinion | 520/> 1658 (< 31) Knowledge: mean score 85% (p < 0.001) Attitudes 92% thinks mass vaccination clinics are necessary and beneficial 3% thinks it is a good idea but not that important in the overall PH system 4% feels intimidated by the scope& responsibilities < 1% ambivalent < 1% no opinion (p < 0,001 compared with pre-test) | 291/> 1658 (< 18) Attitudes: 84% thinks mass vaccination clinics are necessary and beneficial 4% thinks it is a good idea but not that important in the overall PH system 8% feels intimidated by the scope& responsibilities 2% ambivalent 2% no opinion (p = 0.002 compared to pre-test, p = 0.005 compared with post-test) Behavior: 44% planned or taken action to change their health departments’ emergency preparedness plan | |
Potter M.A., 2005, USA, [64] | Post-test: survey with Likert-scale answer options, results of own project | 28/28(100) Satisfaction/methodology: high scores, see study in Table 1. Knowledge & skills: Mean scores between 4.00 and 4.93 for assessing change potential; system thinking; improving negotiation skills; solution mapping; legal mandates; implication wheel, interest-based dispute resolution. | - | ||
Quiram B.J., 2005, USA, [65] | Pre- and posttest: knowledge test; | Unknown/167 Knowledge: see data post-test | Unknown/167 Knowledge: increases on all topics. For different sessions: overall: between + 9.7 and + 32.2% Preparedness planning: between + 5.8 and + 24,5% Biological/chemical agents: between + 3.9 and 34.3% Laboratory capacity: − 2.4 and + 49.5% Risk communication: + 4.6 and 30.2% Communication and IT: + 12.9 and 47.1% Surveillance & epidemiology:+ 12.7 and + 45.6% | - | |
Qureshi K.A., 2004, USA, [66] | Pre-post- and follow-up (1–6 months after) had 10 MCP questions on knowledge, 8 on behavioral intentions and attitudes using 3-point-likert scales General evaluation to a random sample of 100 people | 678/764 (89) Knowledge: means score 8.24 Attitude: 84.3% important to respond during emergency 91.0% responding will help the community 72.9% other PH nurses will respond during an emergency 78.9% they are themselves responsible to assist during emergency 73.2% intend to respond when needed 70.1% coworkers approve my role in emergency response 59.2% believes significant other approves role in emergency response | 726/764 (95) Knowledge: mean score 8.38 (p < 0.05) Attitude: 85.3% important to respond during emergency (NS) 91.6% responding will help the community (NS) 76.9% other PH nurses will respond during an emergency (p < 0.05) 84.6% they are themselves responsible to assist during emergency (p < 0.01) 76.5% intends to respond when needed (p < 0.05) 73.0% coworkers approve my role in emergency response (NS) 64.1% believes significant other approves role in emergency response (p < 0.05) RR 94/100 (94) Satisfaction: 92% program was clear; 88% well-organized, 72% reinforced knowledge regarding emergency response | 230/764 (30) Knowledge mean score unknown Attitude: 93.5% important to respond during emergency (p < 0.05 with post) 94.0% responding will help the community (NS with post) 83.6% other PH nurses will respond during an emergency (NS with post) 91.8% they are themselves responsible to assist during emergency (NS with post) 26.2% intends to respond when needed (NS with post) 81.5% coworkers approve my role in emergency response (NS with post) 73.4% believes significant other approves role in emergency response (p < 0.05 with post) | |
Rega P.P., 2013, USA, [67] | Pre- and post-exercise survey with 5-point-Likert-scale answer options about knowledge, satisfaction | Self-assessed knowledge: few participants reported ‘excellent’ knowledge | RR unknown Self-assessed knowledge: 96–100% improved knowledge Satisfaction: ‘most participants’ valued the exercises for their usefulness and content; exercises are innovative, entertaining, educational; was recommended to be delivered to fellows. | - | |
CBS | Richter J., 2005, USA, [68] | Post-test: survey collecting qualitative and quantitative data | - | 32/50(64) Satisfaction/methodology: majority like small groups; all agreed the workshop would benefit their job. Majority information was useful and met expectations. Self-assessed knowledge: All agreed increased understanding of PH response protocols during outbreaks. 91% agreed better understood protocol during BT attack. Majority had gained greater understanding in public safety, law enforcement and security response protocols (97%) and a BT attack (94). 100% learned about others’ roles during outbreak, 96% during BT attack. Networking: 95% workshop enabled make contacts and network with staff with similar positions at different agencies. 97% had opportunity to network with different positions. | – |
Rottman S.J., 2005, USA, [69] | Pre- and post-test: survey | RR 403/unknown Knowledge: 75,5% mean test score Self-reported knowledge: no data available | RR 403/unknown 87.4% mean test score (p < 0.001). Known own and health department’s roles in disaster: increased (p < 0.001). | - | |
Sandstrom B.E., 2014, Sweden, [70] | Observation of the exercises; evaluation seminars after the exercise | - | Satisfaction/Methodology: applicability of the model was independent of the type of scenario, i.e. it could be used as a generic tool for exercises.; First test (3p): Mutual recognition of the scenario site benefitted participants; Discussion concerning preparedness took 1/3 of the exercise.; important having a highly professional exercise director; high flexibility of the concept; possibility to concentrate the exercise on a few cards without losing the strength of the exercise; participants considered it a cost-effective way of performing table-top exercises; suggested time-frames were removed. Second test (> 40 p): used as a tool to raise awareness; in a heterogeneous population overall understanding of different organizations needs and limitations is possible; not efficient for going into deeper detail in specific procedures and plans.; With a large audience, merely spectators, ineffective for experienced personnel; high flexibility of the concept supported conducting a multi-agency exercise on a more overarching level; allowed for giving different timeframes for various categories of participants; a few more topics were included.; Third test (35p in groups 5–7): the scenario was easily adapted to serve in an international emergency response exercise.; different phases of the response could be emphasized in depth by applying the Director card issues and the corresponding topics.; the tool supported emphasis on different areas of operations in specific phases.; participants confirmed the tests before and appreciated the simplicity of the exercise card concept; also usable for international cross-border exercises. | - | |
Sarpy S.A., 2005, USA, [71] | Pre-and post-test: survey with Likert-scale answer options on behavior, attitude) Post-test: additional Likert-scale questions on self-assessed objectives, satisfaction and open questions on most valuable and useable parts | 49/49 (100) Self-assessed skill Prepared to effectively respond to a SARS event: 3.98(1.52) Community is prepared: 3.10(1.39) Recognize a SARS outbreak: 4.53(1.40) Establish contact and coordinate others: 5.02(1.18) Maintain effective protocols for roles& responsibilities: 4.65(1.33) Use the chain of command: 5.24(1.22) Communicate relevant information in/externally: 5.37(1.20) Determine communication to public: 4.76(1.51) Determine communication to media: 4.63(1.51) Monitor progress and action: 4.71(1.30) Use investigation& management strategies: 4.88(1.31) Self-assessed attitude SARS is threat to my community: 5.65(1.15) | 44/49 (90) Self-assessed skill Prepared to effectively respond to a SARS event: 5.20(1.13) p < 0.001 Community is prepared: 3.93(1.26) NS Recognize a SARS outbreak: 5.75(0.92) p < 0.001 Establish contact and coordinate others: 5.73(0.90) p < 0.001 Maintain effective protocols for roles& responsibilities: 5.32(1.05) NS Use the chain of command: 5.80(0.93) p = 0.002 Communicate relevant information in/externally: 5.86(0.88) NS Determine communication to public: 5.41(1.15) NS Determine communication to media: 5.45(1.13) p = 0.004 Monitor progress and action: 5.41(1.00) p = 0.003 Use investigation& management strategies: 5.36(0.99) NS Self-assessed attitude SARS is threat to my community: 5.82(1.02) NS Self-assessed knowledge: improved understanding of biological agents: 5.34/7 Improved understanding of functional roles & resp.: 5.57/7 Satisfaction (selection): Content: 5.88; format 5.77; mix of participant:5.86; pre-ttx lecture:6.36; ttx: 5.93; overall effectiveness: 6.12 Qualitative answers: most valuable: mix of participants, experts present, discussion sessions, networking possibility. Relevance to job because of: symptom identification, communication, in-& external partners. Improvements on: individual answers, missing perspectives, missing formalized introductions, packaging for greater dissemination. | - | |
Savoia E., 2009, USA, [72] | Pre- and post-test surveys with Likert-scale answer options and % of substantive answers | 56/89(63) Knowledge Between 25 and 85% substantive answers for different questions Confidence See results post-test | 56/89(63) Knowledge Between 70 and 100% substantive answers for different questions (p < 0.05 in all topic areas) Confidence Legal authorities: Between the 32 and 46% difference between pre-and post-test (p < 0.05 for all topic areas) Policies and procedures: Between 30 and 43% difference between pre- and post-test (p < 0.05 for all but one topic (declaration of emergencies). | - | |
Savoia E., 2013, USA, [73] | Guided group discussions where lists of recommendations were developed | - | RR unknown Training methodology: Include practitioners playing key leadership roles in the real world; representatives from agencies and disciplines across the range of jurisdictions that would respond to a specific public health threat; senior level players and specific agencies; a plausible scenario and timeline; clear an measurable exercise objectives; link exercises to prior years’ efforts and prior-tested capabilities; Expertise in exercise planning is a limited or unavailable resource in local health departments; heterogeneity in scopes has implications for the exercise design, required level of participation, and the approach to evaluation. | - | |
TOT | Soeters H.M., 2018, Guinea & USA, [74] | Pre-and post-test with 30 MCP questions on knowledge. Post-test demonstration of skills | 1625/1625(100) Knowledge: Median score 17 | 1625/1625(100) Knowledge: median score 25 Skills: donning/doffing: 70% ‘acceptable’ score for HCW, 97% for supervisors, 83% for trainers Chlorinated water preparation: 80% for HCW, 79% for supervisors, 81% for trainers. | |
Taylor J.L., 2005, USA, [75] | Comments of participants during exercise; post-test: written evaluations (unknown methods) | - | 69/150 (46) Self-assesed attitude: particpants realized needs to continue to build surge capacity specific to the challenges of an influenza pandemic; elected officials and decision makers must have clear understanding of the potential implications of an influenza pandemic and the additional efforts are needed to assure that such officials are adequately informed; pandemic influenza planning needs to be further coordinated with the existing emergency response infrastructure and additional training in incident command is needed; More detailed operational planning is required to achieve an effective overall response; additional support is needed at the federal level | - | |
Umble K.E., 2000, USA, [76] | Pre- and post-tests and follow-up (3 months): Knowledge 5 MCP questions; Agreement, Self-efficacy, Adherence, Setting factor, awareness using Likert-scales. The model used to examine the program’s effects was rooted in several health behavior theories, including the health belief model,15,16 social cognitive theory,16–18 the transtheoretical model,19,20 and the theory of reasoned action. | RR 196/470 for classroom Knowledge: Polio schedule 3.82/5 (1.40) Attitude: Agreement with the polio schedule 33.80/40 (5.55) Adherence to general recommendations: 11.50/15 (4.22) Adherence to the polio schedule 9.30 (5.71) Self-assessed skill: self-efficacy of the polio schedule 4.68/8 (2.09) RR 116/251 for broadcast Knowledge: Polio schedule 2.68/5 (1.61) Attitude: Agreement with the polio schedule 32.37/40 (5.91) Adherence to general recommendations: 9.50/15 (4.49) Adherence to the polio schedule 11.84 (4.31) Self-assessed skill: self-efficacy of the polio schedule 3.53/8 (1.99) | RR 196/470 for classroom Knowledge: Polio schedule 4.48/5 (0.89) (p < 0.001) Attitude: Agreement with the polio schedule 38.38/40 (2.27) (p < 0.001) Adherence to general recommendations: unknown Adherence to the polio schedule: unknown Self-assessed skill: self-efficacy of the polio schedule 6.93/8 (1.84) (p < 0.001) RR 116/251 for broadcast Knowledge: Polio schedule 4.19/5 (1.16) (p < 0.001) Attitude: Agreement with the polio schedule 37.37/40 (3.33) (p < 0.001) Adherence to general recommendations: unknown Adherence to the polio schedule: unknown Self-assessed skill: self-efficacy of the polio schedule 6.55/8 (1.96) (p < 0.001) Significant (p = 0.006) higher increase in knowledge for classroom education compared to broadcasts. | RR 196/470 for classroom Knowledge: Polio schedule 4.52/5 (0.85) (p < 0.001 with pre) Attitude: Agreement with the polio schedule 37.36/40 (2.71) (p < 0.001 with pre) Adherence to general recommendations: 12.38/15 (3.36) (p < 0.05 with pre) Adherence to the polio schedule 16.25 (4.90) (p < 0.001 with pre) Self-assessed skill: self-efficacy of the polio schedule 6.88/8 (1.64) (p < 0.001 with pre) RR 116/251 for broadcast Knowledge: Polio schedule 4.24/5 (1.07) (p < 0.001 with pre) Attitude: Agreement with the polio schedule 37.01/40 (3.60) (p < 0.001 with pre) Adherence to general recommendations: 11.84/15 (4.31) (p < 0.05 with pre) Adherence to the polio schedule 14.42 (5.44) (p = 0.084 with pre) Self-assessed skill: self-efficacy of the polio schedule 6.55/8 (1.96) (p = 0.001 with pre) No significant difference between classroom and broadcast | |
Waltz E.C., 2010, USA, [77] | ARS: post-test evaluation form Broadcasts: # views Web-based lecture: post-test evaluation form | - | ARS RR 93/93 (100) Satellite broadcast RR none Web-based education RR 20.000/44.000 (48) Satisfaction: ARS: 95% agreed that the technology was beneficial to the training; Broadcast: 3871 views on average where in three years for some courses the number of later views exceeded the live views.; Web-based education: 96% rated course quality as good, 99% would recommend the course, 88% the course would help perform their job more effectively | - | |
Wang C., Wei S., Xiang H., Wu J., 2008, China, [78] | Pre-, post- and follow-up (12 months) tests of 40 MCP questions (knowledge) and 5-point-Likert scales for skills. Observation by colleagues, and evaluation of a subsequent real outbreak. | RR 41/43 (95) Knowledge: mean score 21.62 +/− 6.37 Self-assessed skills: mean scores between 2.54–3.05 | RR 41/43 (95) Knowledge: mean score 31.59 +/− 5.85 (p < 0.01 with pre-test) Self-assessed skills: mean scores between 3.49–4.12. Increase in mean scores for all individual skills (p < 0.01) Satisfaction/ Training methodology: 90% thought the training to be excellent. 98% very satisfied with venue, logistics, communication. | RR 41/43 (95) Knowledge: mean score 32.39 +/− 5.15 (p < 0.01 with pre-test) Increase p < 0.05 for assessment knowledge post vs. follow-up test Self-assessed skills: mean scores between 3.68–4.07. Increase in mean scores for all individual skills (p < 0.01) compared with pre-test | |
Wang C., Wei S., Xiang H., 2008a, China, [79] | pre- and post and follow-up (12 months) test of 30 MCP questions on knowledge (objective)& 8 questions on attitude and behaviroal intentions (subjective); during training semistructured interveiws on training method | 76/78 (97) Knowledge: mean score 19.79 +/− 2.41 Self-assessed skills: Assessment: 2.77 (0.81); Policy development: 2.11 (0.69); Communication: 2.68 (0.78); Cultural competency: 2.55 (0.96); Community dimensions of practice: 2.82 (0.73); Basic PH sciences: 2.68 (0.72); Financial planning & management:2.32 (0.89); Leadership & system thinking: 2.86 (0.99). | 76/78 (97) Knowledge: mean score 24.49 +/− 0.86 (p < 0.001 with pre-test) Self-assessed skills: Assessment: 3.69 (0.61); Policy development: 3.95 (0.51); Communication: 3.95 (0.51); Cultural competency: 3.95 (0.69); Community dimensions of practice: 3.84 (0.59); Basic PH sciences: 4.11 (0.45); Financial planning & management: 3.47 (0.82); Leadership & system thinking: 3.89 (0.55); All p < 0.05 compared with pre-test. Satisfaction: 96% thought the methods good/excellent; 100% thought the training innovative; | Knowledge: mean score 24.24 +/− 1.58 (p < 0.001 with pre-test) Self-assessed skills: Assessment: 4.35 (0.72) p < 0.05 compared with post-test; Policy development: 2.94 (0.55) p < 0.05 compared with post-test; Communication: 3.82 (0.61); Cultural competency: 3.56 (0.49); Community dimensions of practice: 3.99 (0.51); Basic PH sciences: 3.74 (0.69); Financial planning & management: 2.66 (0.74) p < 0.05 compared with post-test; Leadership & system thinking: 3.82 (0.62). All p < 0.001 compared with pre-test | |
Wang C., Xiang H., 2010, China, [80] | pre- and post and follow-up (12 months) test of 30 MCP questions on knowledge (objective)& 8 questions on attitude and behaviroal intentions (subjective); during training semistructured interveiws on training method | RR 226/237(95) Knowledge: mean score 18.50 +/− 3.23 Self-assessed skills: Assessment: 2.54 (0.76); Policy development: 2.33 (1.06); Communication: 3.16 (0.84); Cultural competency: 2.26 (0.76); Community dimensions of practice: 2.69 (0.81); Basic PH sciences: 3.12 (0.93); Financial planning & management: 2.07 (1.03); Leadership & system thinking: 2.71 (0.99). | RR 226/237(95) Knowledge: mean score 22.78 +/− 1.14 (p < 0.001 with pre-test) Self-assessed skills: Assessment: 3.91 (0.65) p < 0.05; Policy development: 3.48 (0.70) p < 0.05; Communication: 4.13 (0.65) p < 0.05; Cultural competency: 3.44 (0.61) p < 0.05; Community dimensions of practice: 3.87 (0.73) p < 0.05; Basic PH sciences: 4.69 (0.49) p < 0.05; Financial planning & management: 3.26 (0.74) p < 0.05; Leadership & system thinking: 3.05 (0.69). NS Satisfaction: 92% thought training methods excellent, 96% satisfied with trainers’ performance, 89% training approach scientific and feasible, 99% very satisfied with venue, training, logistics. | RR unknown Knowledge: mean score 22.69 +/− 2.49 (p < 0.001 with pre-test) Self-assessed skills: Assessment: 4.46 (0.73) p < 0.05 with pre- and post-test; Policy development: 2.82 (0.82) NS; Communication: 4.27 (0.61) p < 0.05 with pre-test; Cultural competency: 3.21 (0.79) p < 0.05 with pre-test; Community dimensions of practice: 3.79 (0.65) NS; Basic PH sciences: 4.35 (0.54) p < 0.05 with pre-test; Financial planning & management: 2.79 (0.92) NS; Leadership & system thinking: 2.84 (0.77) NS. | |
Yamada S., 2007, Hawaii/USA, [81] | Attendance logs; Post-test: course evaluation questionnaires, interviews with participants | - | RR 85/unknown Satisfaction/methodology: 83/85 thought the case appropriate to own setting; 84/85 liked group work; 80/86 thought mixed groups to be helpful for learning; 84/85 thought the case overall worthwhile as a continuing education activity. Suggestions for better case: longer time to analyze the case and prepare for the discussion on the learning issues; provide pictures. Suggestions for the facilitator: more in-depth knowledge, all members included in discussion | - | |
Yellowlees P., 2008, USA, [82] | Post-test: survey with 4 questions and 5-point-Likert-scale answer options + open ended questions | - | RR 13/25(52) Satisfaction/methodology: 3.95 practicality of the program 4.5: time well spent 4.3: met the objectives. | - |