Interval follow up of a 4-day pilot program to implement the WHO surgical safety checklist at a Congolese hospital
Globalization and Health volume 13, Article number: 42 (2017)
The World Health Organisation Surgical Safety Checklist (SSC) improves surgical outcomes and the research question is no longer ‘does the SSC work?’ but, ‘how to make the SSC work?’ Evidence for implementation strategies in low-income countries is sparse and existing strategies are heavily based on long-term external support. Short but effective implementation programs are required if widespread scale up is to be achieved. We designed and delivered a four-day pilot SSC training course at a single hospital centre in the Republic of Congo, and evaluated the implementation after one year. We hypothesised that participants would still be using the checklist over 50% of the time.
We taught the four-day SSC training course at Dolisie hospital in February 2014, and undertook a mixed methods impact evaluation based on the Kirkpatrick model in May 2015. SSC implementation was evaluated using self-reported questionnaire with a 3 point Likert scale to assess six key process measures. Learning, behaviour, organisational change and facilitators and inhibitors to change were evaluated with questionnaires, interviews and focus group discussion.
Seventeen individuals participated in the training and seven (40%) were available for impact evaluation at 15 months. No participant had used the SSC prior to training. Over half the participants were following the six processes measures always or most of the time: confirmation of patient identity and the surgical procedure (57%), assessment of difficult intubation risk (72%), assessment of the risk of major blood loss (86%), antibiotic prophylaxis given before skin incision (86%), use of a pulse oximeter (86%), and counting sponges and instruments (71%). All participants reported positive improvements in teamwork, organisation and safe anesthesia. Most participants reported they worked in helpful, supportive and respectful atmosphere; and could speak up if they saw something that might harm a patient. However, less than half felt able to challenge those in authority.
Our study demonstrates that a 4-day pilot course for SSC implementation resulted in over 50% of participants using the SSC at 15 months, positive changes in learning, behaviour and organisational change, but less impact on hierarchical culture. The next step is to test our novel implementation strategy in a larger hospital setting.
The World Health Organisation (WHO) Surgical Safety Checklist (SSC) improves compliance with basic safety processes and surgical outcomes [1, 2] but the most effective methods of implementation in low and middle-income countries (LMIC) are unknown. Reports of successful SSC implementation in LMICs exist [3–7], but rely on significant time and resource commitments from high-income country (HIC) providers, which limits wide-spread implementation. Therefore, with scale-up in mind, we piloted a four-day SSC training course using a small team of HIC providers.
Team training and supportive hospital leadership are important for sustained implementation . Barriers to successful SSC implementation include lack of adaptation to local practice, paucity of buy-in due to poor understanding, general lack of supplies and functioning equipment [9, 10]. Therefore our four-day SSC training was based on the principles of (a) understanding the rationale for the SSC; (b) local ownership through locally-driven adaptation; (c) donation of equipment; (d) team training and operating room simulation. We aimed to pilot the four-day course at the main hospital in Dolisie, and hypothesised that one year after the course, over 50% of participants would still be using the SSC and following six basic safety processes as described by Haynes et al. ; and that there would be sustained positive changes in learning, behaviour and organisational practice.
The Minister of Health of the Republic of Congo gave permission for the training program and the subsequent evaluation as part of a Mercy Ships countrywide healthcare education program. Individuals gave verbal consent to participate but written signed consent was not required. The study was given ethics approval by the Mercy Ships Institutional Review Board (study number MS2016004) and the requirement for individual written consent was waived.
Mercy Ships is a global charity operating the world’s largest non-governmental hospital ship, the Africa Mercy. Mercy Ships visits coastal African countries at the invitation of the President and works closely with the Ministry of Health to deliver surgical services, and training. From August 2013 to June 2014 Mercy Ships was docked in Pointe Noire, Republic of Congo. The Mercy Ships Project Manager undertook a one-day baseline assessment visit, four months prior to the training.
Participants and setting
The Republic of Congo has a population of 4.6 million , and Dolisie is the third largest city. The hospital in Dolisie serves a population of approximately 90,000 people. At the time of the baseline assessment there was one surgeon, two obstetricians and five nurse anaesthetists. No one had ever heard of the SSC, there were no pulse oximeters available for three operating rooms and the recovery room, and nurses did not know how to count needles, sponges and instruments. There was no formal process for discussing the risk or difficult intubation or estimated blood loss. The SSC training was explained to the hospital director, classroom facilities identified and dates agreed for training.
The SSC course outline is given in Fig. 1. The training occurred over four days in February 2014, and used five training facilitators from HICs. Three were Mercy Ships crew with experience teaching the SSC (a physician anaesthetist, operating room nurse and training project manager); one British trainee surgeon, and one French teacher providing support and translation.
Participants were all operating team personnel at Dolisie hospital, and were invited to attend the training by the hospital director.
The Mercy Ships project manager and physician anaesthetist returned to Dolisie hospital three months after training to encourage continued use of the SSC and trouble shoot obstacles to implementation. Data was not formally collected as part of this visit which was primarily for training purposes but field notes were made in notebooks.
We used a mixed-methods design to assess impact of SSC implementation at 15 months based on the Kirkpatrick model for evaluation , (Table 1). Structured interviews, focused group discussion and questionnaires were used to collect data.
All participants still working at the hospital 15 months after training were asked by the hospital director to attend the impact evaluation. Interviews, focused group discussion, and observations in the operating room took place in the hospital and were conducted by the authors KC and MW. MW is a physician anaesthetist and KC is the training projects manager. Both KC and MW participated in the original SSC training. Participants were interviewed once in French with translation where necessary, and interviews lasted 15–30 min. There was one focused group discussion. Responses were not recorded and transcribed due to budget constraints but instead recorded with pen and paper contemporaneously.
Structured interviews and focused group discussion were based around the following questions:
What were the most important things you learnt from the surgical safety checklist training?
Have you made any changes in your personal practice since the training?
Have you noticed any changes in your hospital since the training?
Did anything help or hinder the changes that were made?
Questions 1, 2 and 3 correspond to Kirkpatrick level 2,3 and 4 evaluation respectively. Question 4 aimed to identify facilitators and inhibitors to change to aid future course development.
Participants also completed an anonymous questionnaire, written in French, made up of 17 questions in three parts (see Figure 2). Part 1 was designed to evaluate extent of SSC implementation, and parts 2 and 3 to further determine learning, behaviour and organisational change (Kirkpatrick Level 2,3 and 4 changes).
Observations in the operating room were planned to visually assess SCC implementation and recorded in field notebooks.
Hypothesis and primary outcome measure
The primary hypothesis was that after one year, over 50% of participants would be following the six basic safety processes  most of the time. Therefore the primary outcome measure was the number of participants following each safety measure, ‘always’ or ‘most of the time’.
The secondary hypothesis was that the SSC would lead to positive changes in learning, behaviour and organisational change (Kirkpatrick level 2, 3, and 4) evaluation.
To test the primary hypothesis, simple descriptive statistics were used. To test the secondary hypothesis we use mixed methods: quantitative data was analysed with descriptive statistics; and qualitative data was manually themed the interviews, focus group discussion, and questionnaire responses for analysis and reporting of descriptive patterns .
Between the time of the baseline assessment in October 2013 to the SSC training in February 2014, the general surgeon and one obstetrician had left the hospital and there was an increase from five to seven nurse anaesthetists. Therefore 17 individuals (all the operating room staff) participated in the initial SSC training in February 2014 (1 obstetrician, 7 nurse anaesthetists and 9 operating room nurses, see Table 2).
At the 3 month evaluation the SSC was seen on the wall of the operating room and staff (Hospital Director, nurse anaesthetists and operating room nurses) reported using the SSC including pulse oximetry and counting, and showed evidence of this in practice. A large pile of completed ‘counting sheets’ was seen.
At the impact evaluation in June 2015 only 7/17 (40%) participants were available for interview, 3 nurse anaesthetists and 4 operating room nurses. The obstetrician had left and had been replaced by temporary surgeons / obstetricians who came on a rotational basis to provide surgical cover. Details of training participants are given in Table 2.
At 15 months after training, the six basic safety processes were being performed by more than half of the attending participants either always or most of the time: confirmation of patient identity and surgical procedure (57% of participants), assessment of difficult intubation risk (72% of participants), assessment of the risk of major blood loss (86% of participants), antibiotic prophylaxis given before skin incision (86% of participants), use of a pulse oximeter (86% of participants), and counting (71% of participants). Details are given in Table 3.
Kirkpatrick levels 2,3 and 4 results from interviews and focussed group discussion responses
The most important things learnt recalled by participants were, introducing themselves to each other, confirming the patient identity and site of surgery, and counting (Kirkpatrick level 2). This was backed up by responses to the questions regarding personal and organisational change (Kirkpatrick level 3 and 4).
Several participants reported that when they introduced themselves in front of the patient, and verified the patient’s identity and operation site, then the patient was reassured and visibly calmed down. Participants also reported that visiting surgeons were asked to introduce themselves and participate in this process in front of the patient. Counting and giving antibiotics on time were also reported as personal and organisational changes. Further details are given in Table 4.
Kirkpatrick level 3 and 4 results from the questionnaire responses
Questionnaire part 2 responses
A selection of the commonest participants free text responses to questions asking about the impact of SSC training on perceptions of teamwork, communication, organisation and safe anaesthesia are given in Table 5. All seven participants reported the training had a positive effect on teamwork, organisation and safe anaesthesia practices. Six out of seven reported a positive effect on communication; and one did not answer the question.
Questionnaire part 3 responses
Responses to the seven questions concerning attitudes known to effect patient safety are given in Table 6. Most (71–100%) agreed that they worked in helpful, supportive, and respectful atmosphere; and would speak up if they saw something that might harm a patient. However less than half (43%) felt able to challenge those in authority or to ask questions when they see things that are wrong.
Observations in the operating room
On the day of the evaluation visit there was no surgery taking place therefore we were unable to observe the SSC in use. However, we were shown the pulse oximeters which showed signs of repeated use, and evidence of counting sheets being used.
Inhibitors and facilitators to change
Inhibitors to changes in personal and organisational practice were lack of support; and that they had learnt to count in pairs, but were often on their own which made counting difficult. Two facilitators to change were identified: ‘seeing the counting performed made a difference’, and ‘ we all get on well already so we are motivated to help each other’.
In this paper we report that 15 months after a four-day SSC pilot program, the six basic safety processes are being followed by 57–86% of participants; and self-reported positive improvements in teamwork, communication and organisation and safe anaesthesia persist. Evidence of pulse oximeter use and counting of needles, sponges and instruments was also observed.
Our study reports successful SSC implementation after a four-day pilot course in one LMIC hospital. This is a novel finding because to date, successful SSC implementation is reported in just a few single institutions in LMICs and only after sustained time-commitments from HIC providers. [3–7]. Shorter time-scale implementation strategies are needed to facilitate widespread uptake of the SSC in LMICs. Our pilot course offers novel insights in this regard.
The SSC is simple, inexpensive and effective  and also has a dose effect [14–16], meaning even if only used in part, surgical outcomes are still improved. Therefore it is clinically significant that in our pilot study the 6 basic safety processes were being followed by 57–86% of participants 15 months after SSC training. Haynes et al.  only achieved 67% compliance in all 6 basic safety process measures at 3 months, yet achieved a 47% reduction in mortality, and 50% reduction in post-operative infections. In our study, assessment of the risk of major blood loss, antibiotic prophylaxis given before skin incision, and use of a pulse oximeter (86% of participants) were the most common sustained practice changes at 15 months. It is significant that our participants also experienced positive changes in attitude related to teamwork, organisation and communication, because some studies show that partial completion of the SSC due to poor attitudes can substantially negate the benefits .
We specifically designed our pilot programme to take into account our prior lessons learnt ; factors known to influence surgical outcomes such as teamwork  and the retention of surgical items ; and to overcome previously identified implementation barriers such as lack of knowledge, poor buy-in from senior clinicians, inadequate resources and a hierarchical culture [3, 5, 9, 10]. All seven participants reported positive changes in teamwork, organisation and safe anaesthesia, and six out of seven reported improvements in communication. Many participants also noted organisational changes such as accompanying patients from the ward to the operating room, introducing themselves in front of the patient and asking visiting surgeons to do the same. Furthermore there was a perception of high levels of help (100%), respect (71%) and support (86%) among the operating room team. This may partially explain why our short four-day pilot course has sustained success despite the lack of a permanent surgeon/obstetrician. The nurses reported making the temporary surgeons do the SSC and in particular introduce themselves in front of the patient. The high degree of team coherence, culture of respect, as well as supportive hospital director may explain this finding [8, 9, 20].
involvement of the hospital director from the planning to implementation and follow- up stages
time spent understanding the rationale and adapting the SSC to the local environment which was led by the local staff and on going adaptations made through out the course
multidisciplinary team training
involvement of doctors and nurses working together through out the course and modelling how to speak up and help and support one another
operating room simulation
allowed the course to be not just theoretical but practical and highlighted challenges not foreseen when using simulation in the classroom
Additionally the hospital was small, and the operating team was a close-knit team as evidenced by one participant who said,
We all get on well already so we are motivated to help each other.
However, other participants commented that as a result of training:
We can help each other and encourage each other
We now have a team atmosphere, suggestions and mutual exchanges
Before I would just get angry if people weren’t interested, now we can talk about it
We are nicer to each other
These statements indicate that the SSC had made a noticeable improvement even if one participant felt the atmosphere was good to start with. Despite most participants (71–100%) reporting a helpful, respectful and supportive atmosphere, one area that still requires more attention is to the breaking down of the hierarchical culture and questioning those in authority. Our results show less than half (43%) of the participants feel able to challenge those in authority or ask questions when they see things that are wrong. This could be because hierarchy and questioning those in authority are implied in the course but not explicitly taught whereas teamwork, communication, organisation and safe anaesthesia are.
Our study has several limitations. The main one is that this is a pilot study at a single, small institution. The data was self-reported and therefore open to subjective bias, both under-reporting out of a desire to gain more training and equipment; or and over-reporting to impress the evaluators or hospital director. We did not measure attitudes to teamwork, communication, organisation and safe anaesthesia; or responses to the statements regarding the safety culture of the hospital, before the study and therefore have no baseline data for comparison. The questionnaire although based on solid Kirkpatrick principles was novel and not previously validated; therefore this was a pilot study in a small hospital and only 7 out of 17 (40%) participants were available for impact evaluation. Therefore results should be interpreted with caution and cultural differences may limit applicability of the course to other countries. The lack of permanent surgeon in this pilot study also limits the general applicability and may have introduced bias. Surgeons can become SSC champions and positively influence implementation, however disengaged surgeons are known to hinder implementation [9, 20]. That participants in this study reported persuading visiting temporary surgeons to introduce themselves and use the SCC shows positive buy-in from nursing staff and a willingness to promote safe surgical practice.
Despite these limitations we believe our study has a number of strengths. It was a pilot study designed to test the hypothesis that sustainable SSC implementation is possible after only a four-day course. Widespread SSC implementation in LMICs requires resource-light novel strategies and despite the small numbers our pilot course adds to the literature in this regard. We used a mix methods analysis based on the Kirkpatrick model of evaluation  to evaluate changes in attitudes and determine facilitators and inhibitors to personal and organisational change which will be used in further course development
Our pilot study shows that in one hospital in the Republic of Congo, SSC implementation was achieved and sustained at 15 months after a four-day pilot training course, with the six basic safety measures being followed by 57–86% of participants. Further work is required to test this model in larger hospitals to assess the suitability of a short course format for wide-scale national implementation.
High Income Country
Low Middle Income Country
Surgical Safety Checklist
World Health Organisation
Haynes AB, Weiser TG, Berry WR, Lipsitz SR, Breizat AH, Dellinger EP, et al. A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med. 2009;360(5):491–9.
Global Surgery Collaborative. Mortality of emergency abdominal surgery in high-, middle- and low-income countries. Br J Surg. 2016;103(8):971–88.
Kwok AC, Funk LM, Baltaga R, Lipsitz SR, Merry AF, Dziekan G, et al. Implementation of the World Health Organization surgical safety checklist, including introduction of pulse oximetry, in a resource-limited setting. Ann Surg. 2013;257(4):633–9.
Kasatpibal N, Senaratana W, Chitreecheur J, Chotirosniramit N, Pakvipas P, Junthasopeepun P. Implementation of the World Health Organization surgical safety checklist at a university hospital in Thailand. Surg Infect. 2012;13(1):50–6.
Lilaonitkul M, Kwikiriza A, Ttendo S, Kiwanuka J, Munyarungero E, Walker IA, et al. Implementation of the WHO surgical safety checklist and surgical swab and instrument counts at a regional referral hospital in Uganda - a quality improvement project. Anaesthesia. 2015;70(12):1345–55.
Bashford T, Reshamwalla S, McAuley J, Allen NH, McNatt Z, Gebremedhen YD. Implementation of the WHO surgical safety checklist in an Ethiopian referral hospital. Patient Saf Surg. 2014;8:16.
Yuan CT, Walsh D, Tomarken JL, Alpern R, Shakpeh J, Bradley EH. Incorporating the World Health Organization surgical safety checklist into practice at two hospitals in Liberia. Jt Comm J Qual Patient Saf. 2012;38(6):254–60.
White MC, Millimouno FS. Observational study of three different methods of implementing the WHO surgical safety checklist in Guinea. Global Anesthesia and Perioperative Medicine. 2016;2(2):158–61.
Aveling EL, McCulloch P, Dixon-Woods M. A qualitative study comparing experiences of the surgical safety checklist in hospitals in high-income and low-income countries. BMJ Open. 2013;3(8):e003039.
Conley DM, Singer SJ, Edmondson L, Berry WR, Gawande AA. Effective surgical safety checklist implementation. J Am Coll Surg. 2011;212(5):873–9.
World Bank. Republic of Congo Country Profile. Available from: http://www.worldbank.org/en/country/congo.
Kirkpatrick DL. The new world Kirkpatrick model 2009. Available from: http://kirkpatrickpartners.com.
Boyatzis R. Transforming qualitative information: thematic analysis and code development. Thousand Oaks, London: Sage Publications; 1998.
Bergs J, Hellings J, Cleemput I, Zurel O, De Troyer V, Van Hiel M, et al. Systematic review and meta-analysis of the effect of the World Health Organization surgical safety checklist on postoperative complications. Br J Surg. 2014;101(3):150–8.
van Klei WA, Hoff RG, van Aarnhem EE, Simmermacher RK, Regli LP, Kappen TH, et al. Effects of the introduction of the WHO “surgical safety checklist” on in-hospital mortality: a cohort study. Ann Surg. 2012;255(1):44–9.
Haugen AS, Softeland E, Almeland SK, Sevdalis N, Vonen B, Eide GE, et al. Effect of the World Health Organization checklist on patient outcomes: a stepped wedge cluster randomized controlled trial. Ann Surg. 2015;261(5):821–8.
Mayer EK, Sevdalis N, Rout S, Caris J, Russ S, Mansell J, et al. Surgical checklist implementation project: the impact of variable WHO checklist compliance on risk-adjusted clinical outcomes after National Implementation: a longitudinal study. Ann Surg. 2016;263(1):58–63.
Russ S, Rout S, Sevdalis N, Moorthy K, Darzi A, Vincent C. Do safety checklists improve teamwork and communication in the operating room? A systematic review. Ann Surg. 2013;258(6):856–71.
Stawicki SP, Cook CH, Anderson HL 3rd, Chowayou L, Cipolla J, Ahmed HM, et al. Natural history of retained surgical items supports the need for team training, early recognition, and prompt retrieval. Am J Surg. 2014;208(1):65–72.
Bergs J, Lambrechts F, Simons P, Vlayen A, Marneffe W, Hellings J, et al. Barriers and facilitators related to the implementation of surgical safety checklists: a systematic review of the qualitative evidence. BMJ Qual Saf. 2015;24(12):776–86.
The authors would like to thank the Lifebox Foundation for their support of the project and the use of training materials for pulse oximetry.
Ed Fitzgerald reports funding from the Lifebox Foundation for international flights to the Republic of Congo.
We received no other funding for the study.
Availability of data and materials
The datasets used and/or analysed during the current study available from the corresponding author on reasonable request.
MW conceived the original idea and study design. All authors were involved in the initial training programme and development. MW and KC were involved in the data acquisition, analysis and interpretation of data. MW reviewed the literature and drafted the manuscript. All authors read and approved the final manuscript.
Michelle White is a pediatric anesthesiologist and has worked for Mercy Ships for over 4 years, designing, implementing and evaluating medical capacity building and quality improvement projects in Guinea, Republic of Congo, Madagascar and Benin.
Jennifer Peterschmidt is a registered operating room nurse and has worked for Mercy Ships for over 4 years training operating room nurses in Guinea, Republic of Congo, Madagascar and Benin.
Edward Fitzgerald is a trainee in general surgery and Clinical Advisor to the Lifebox Foundation.
James Callahan is a french teacher and has experience in teaching, translating and interpreting for Mercy Ships Medical Capacity Building projects for 3 years in Republic of Congo, Madagascar
Kristin Close is a project manager and has worked for Mercy Ships for over 4 years, designing, implementing and evaluating medical capacity building and quality improvement projects in Guinea, Republic of Congo, Madagascar and Benin.
The authors have no conflicts of interest (including financial interests) to declare.
Consent for publication
As described above.
Ethics approval and consent to participate
The Minister of Health of Republic of Congo gave permission for the training program and the subsequent evaluation as part of a Mercy Ships countrywide healthcare education program. Individuals gave verbal consent to participate but written signed consent was not required.
The study was given ethics approval by the Mercy Ships Institutional Review Board (study number MS2016004) and the requirement for individual written consent was waived.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
White, M.C., Peterschmidt, J., Callahan, J. et al. Interval follow up of a 4-day pilot program to implement the WHO surgical safety checklist at a Congolese hospital. Global Health 13, 42 (2017). https://doi.org/10.1186/s12992-017-0266-0