1
|
Robinson SJ, Yin Mar Oo, Ljuhar D, McLeod E, Pacilli M, Nataraja RM. A guide to outcome evaluation of simulation-based education programmes in low and middle-income countries. ANZ J Surg 2024; 94:1011-1020. [PMID: 38553885 DOI: 10.1111/ans.18987] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2023] [Revised: 02/16/2024] [Accepted: 03/13/2024] [Indexed: 06/19/2024]
Abstract
Evaluation is a vital part of any learning activity and is essential to optimize and improve educational programmes. It should be considered and prioritized prior to the implementation of any learning activity. However, comprehensive programme evaluation is rarely conducted, and there are numerous barriers to high-quality evaluation. This review provides a framework for conducting outcome evaluation of simulation-based education programmes in low and middle-income countries (LMICs). The basis of evaluation, including core ideas of theory, purpose and structure are outlined, followed by an examination of the levels and healthcare applications of the Kirkpatrick model of evaluation. Then, methods of conducting evaluation of simulation-based education in LMICs are discussed through the lens of a successful surgical simulation programme in Myanmar, a lower-middle-income country. The programme involved the evaluation of 11 courses over 4 years in Myanmar and demonstrated evaluation at the highest level of the Kirkpatrick model. Reviewing this programme provides a bridge between evaluation theory and practical implementation. A range of evaluation methods are outlined, including surveys, interviews, and clinical outcome measurement. The importance of a mixed-methods approach, enabling triangulation of quantitative and qualitative analysis, is highlighted, as are methods of analysing data, including statistical and thematic analysis. Finally, issues and challenges of conducting evaluation are considered, as well as strategies to overcome these barriers. Ultimately, this review informs readers about evaluation theory and methods, grounded in a practical application, to enable other educators in low-resource settings to evaluate their own activities.
Collapse
Affiliation(s)
- Samuel Ja Robinson
- Department of Paediatrics, School of Clinical Sciences, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, Victoria, Australia
- Department of Surgery, School of Clinical Sciences, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, Victoria, Australia
- Department of Paediatric Surgery and Monash Children's Simulation, Monash Children's Hospital, Melbourne, Victoria, Australia
| | - Yin Mar Oo
- Department of Paediatric Surgery, Yangon Children's Hospital, Yangon, Myanmar
| | - Damir Ljuhar
- Department of Paediatrics, School of Clinical Sciences, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, Victoria, Australia
- Department of Surgery, School of Clinical Sciences, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, Victoria, Australia
- Department of Paediatric Surgery and Monash Children's Simulation, Monash Children's Hospital, Melbourne, Victoria, Australia
| | - Elizabeth McLeod
- Department of Paediatric and Neonatal Surgery, Royal Children's Hospital, Melbourne, Victoria, Australia
| | - Maurizio Pacilli
- Department of Paediatrics, School of Clinical Sciences, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, Victoria, Australia
- Department of Surgery, School of Clinical Sciences, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, Victoria, Australia
- Department of Paediatric Surgery and Monash Children's Simulation, Monash Children's Hospital, Melbourne, Victoria, Australia
| | - Ramesh M Nataraja
- Department of Paediatrics, School of Clinical Sciences, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, Victoria, Australia
- Department of Surgery, School of Clinical Sciences, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, Victoria, Australia
- Department of Paediatric Surgery and Monash Children's Simulation, Monash Children's Hospital, Melbourne, Victoria, Australia
| |
Collapse
|
2
|
Pollok F, Lund SB, Traynor MD, Alva-Ruiz R, MacArthur TA, Watkins RD, Mahony CR, Woerster M, Yeh VJH, Matovu A, Clarke DL, Laack TA, Rivera M. Systematic Review of Procedural Skill Simulation in Health Care in Low- and Middle-Income Countries. Simul Healthc 2023:01266021-990000000-00076. [PMID: 37440427 DOI: 10.1097/sih.0000000000000737] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/15/2023]
Abstract
ABSTRACT Low- and middle-income countries (LMICs) have adopted procedural skill simulation, with researchers increasingly investigating simulation efforts in resource-strained settings. We aim to summarize the current state of procedural skill simulation research in LMICs focusing on methodology, clinical area, types of outcomes and cost, cost-effectiveness, and overall sustainability. We performed a comprehensive literature review of original articles that assessed procedural skill simulation from database inception until April 2022.From 5371 screened articles, 262 were included in this review. All included studies were in English. Most studies were observational cohort studies (72.9%) and focused on obstetrics and neonatal medicine (32.4%). Most measured outcome was the process of task performance (56.5%). Several studies mentioned cost (38.9%) or sustainability (29.8%). However, few articles included actual monetary cost information (11.1%); only 1 article assessed cost-effectiveness. Based on our review, future research of procedural skill simulation in LMICS should focus on more rigorous research, cost assessments, and on less studied areas.
Collapse
Affiliation(s)
- Franziska Pollok
- From the Multidisciplinary Simulation Center (F.P., S.B.L., M.W., T.A.L.), Mayo Clinic, Rochester, MN; Department for Anesthesiology (F.P., M.W.), University Hospital Hamburg-Eppendorf, Hamburg, Germany; Department of Surgery (S.B.L., M.D.T., R.A.-R., T.A.M., R.D.W., C.R.M., V.J.-H.Y., M.R.), Mayo Clinic, Rochester, MN; Department of Surgery (A.M.), Mubende Regional Referral Hospital, Mubende, Uganda; Department of Molecular Medicine and Surgery (A.M.), Karolinska Institutet, Sweden; University of KwaZulu Natal, Pietermaritzburg (D.L.C.), KwaZulu Natal, South Africa; University of Witwatersrand, Johannesburg (D.L.C.), Gauteng, South Africa; and Department of Emergency Medicine (T.A.L.), Mayo Clinic, Rochester, MN
| | | | | | | | | | | | | | | | | | | | | | | | | |
Collapse
|
3
|
Terning G, El-Thalji I, Brun EC. The Impact of Patient Infection Rate on Emergency Department Patient Flow: Hybrid Simulation Study in a Norwegian Case. Healthcare (Basel) 2023; 11:1904. [PMID: 37444740 DOI: 10.3390/healthcare11131904] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2023] [Revised: 06/23/2023] [Accepted: 06/27/2023] [Indexed: 07/15/2023] Open
Abstract
The COVID-19 pandemic put emergency departments all over the world under severe and unprecedented distress. Previous methods of evaluating patient flow impact, such as in-situ simulation, tabletop studies, etc., in a rapidly evolving pandemic are prohibitively impractical, time-consuming, costly, and inflexible. For instance, it is challenging to study the patient flow in the emergency department under different infection rates and get insights using in-situ simulation and tabletop studies. Despite circumventing many of these challenges, the simulation modeling approach and hybrid agent-based modeling stand underutilized. This study investigates the impact of increased patient infection rate on the emergency department patient flow by using a developed hybrid agent-based simulation model. This study reports findings on the patient infection rate in different emergency department patient flow configurations. This study's results quantify and demonstrate that an increase in patient infection rate will lead to an incremental deterioration of the patient flow metrics average length of stay and crowding within the emergency department, especially if the waiting functions are introduced. Along with other findings, it is concluded that waiting functions, including the waiting zone, make the single average length of stay an ineffective measure as it creates a multinomial distribution of several tendencies.
Collapse
Affiliation(s)
- Gaute Terning
- Department of Safety, Economics, and Planning, University of Stavanger, 4036 Stavanger, Norway
| | - Idriss El-Thalji
- Department of Mechanical and Structural Engineering and Materials Science, University of Stavanger, 4036 Stavanger, Norway
| | - Eric Christian Brun
- Department of Safety, Economics, and Planning, University of Stavanger, 4036 Stavanger, Norway
| |
Collapse
|
4
|
Nataraja RM, Yin Mar Oo, Ljuhar D, Pacilli M, Nyo Nyo Win, Stevens S, Aye Aye, Nestel D. Long-Term Impact of a Low-Cost Paediatric Intussusception Air Enema Reduction Simulation-Based Education Programme in a Low-Middle Income Country. World J Surg 2022; 46:310-321. [PMID: 34671841 DOI: 10.1007/s00268-021-06345-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 10/02/2021] [Indexed: 10/20/2022]
Abstract
INTRODUCTION Intussusception is one of the commonest causes of bowel obstruction in infants. Most infants in Low- and Middle-Income Countries (LMICs) undergo an invasive operative intervention. Supported by simulation-based education (SBE), the Air Enema (AE) non-operative technique was introduced in 2016 in Myanmar. This study assesses the long-term outcomes. METHODS Mixed methods study design over 4 years including clinical outcomes and surgeon's attitudes towards the AE technique and SBE. Prospectively collected clinical outcomes and semi-structured interview with reflexive thematic analysis (RTA). Primary outcome measure was a long-term shift to non-operative intervention. SECONDARY OUTCOMES Length of Stay (LoS), recurrence rates, intestinal resection rates, compared to the operative group. The data was analysed according to intention to treat. Quantitative data analysis with Mann-Whitney U test, Fisher's exact test, Student's T-Test or Wilcoxon Signed-Rank Test utilised. A p-value of <.05 was considered significant. RESULTS A total of 311 infants with intussusception were included. A sustained shift to AE was revealed with high success rates (86.1-91.2%). AE had a reduced LoS (4 vs. 7 days p ≤ 0.0001), Duration of Symptoms (DoS) was lower with AE (1.9/7 vs. 2.5/7, p = 0.002). Low recurrence rates (0-5.8%) and intestinal resection rates stabilised at 30.5-31.8% vs.15.3% pre-intervention. Four RTA themes were identified: Expanding conceptions of healthcare professional education and training; realising far reaching advantages; promoting critical analysis and reflective practice of clinicians; and adapting clinical practice to local context. RTA revealed an overall positive paradigm shift in attitudes and application of SBE. CONCLUSIONS A sustained change in clinical outcomes and appreciation of the value of SBE was demonstrated following the intervention.
Collapse
Affiliation(s)
- R M Nataraja
- Department of Paediatric Surgery, Urology and Surgical Simulation, Monash Children's Hospital, 246 Clayton Road, Clayton, Melbourne, VIC, 3168, Australia.
- Department of Paediatrics, School of Clinical Sciences, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, VIC, Australia.
| | - Yin Mar Oo
- Department of Paediatric Surgery, Yangon Children's Hospital, Yangon, Myanmar
| | - D Ljuhar
- Department of Paediatric Surgery, Urology and Surgical Simulation, Monash Children's Hospital, 246 Clayton Road, Clayton, Melbourne, VIC, 3168, Australia
- Department of Paediatrics, School of Clinical Sciences, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, VIC, Australia
| | - M Pacilli
- Department of Paediatric Surgery, Urology and Surgical Simulation, Monash Children's Hospital, 246 Clayton Road, Clayton, Melbourne, VIC, 3168, Australia
- Department of Paediatrics, School of Clinical Sciences, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, VIC, Australia
| | - Nyo Nyo Win
- Department of Paediatric Surgery, Yangon Children's Hospital, Yangon, Myanmar
| | - S Stevens
- Austin Clinical School, University of Melbourne, Melbourne, VIC, Australia
- Austin Precinct, Department of Surgery, University of Melbourne, Melbourne, VIC, Australia
| | - Aye Aye
- Department of Paediatric Surgery, Yangon Children's Hospital, Yangon, Myanmar
| | - D Nestel
- Austin Precinct, Department of Surgery, University of Melbourne, Melbourne, VIC, Australia
- School of Clinical Sciences, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, VIC, Australia
| |
Collapse
|
5
|
Servin-Rojas M, Olivas-Martinez A, Dithurbide-Hernandez M, Chavez-Vela J, Petricevich VL, García-Juárez I, Gallo de Moraes A, Zendejas B. Impact of the COVID-19 pandemic on the clinical training of last year medical students in Mexico: a cross-sectional nationwide study. BMC MEDICAL EDUCATION 2022; 22:24. [PMID: 34998416 PMCID: PMC8742662 DOI: 10.1186/s12909-021-03085-w] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/22/2021] [Accepted: 12/13/2021] [Indexed: 06/02/2023]
Abstract
BACKGROUND The COVID-19 pandemic has brought unprecedented changes to medical education. However, no data are available regarding the impact the pandemic may have on medical training in Mexico. The aim of our study was to evaluate and identify the medical school students' perceptions of the changes in their clinical training due to the pandemic in Mexico. METHODS This was a cross-sectional study where a previous validated online survey was translated and adapted by medical education experts and applied to senior medical students from March to April of 2021. The 16-item questionnaire was distributed online combining dichotomous, multiple-choice, and 5-point Likert response scale questions. Descriptive and multivariate analyses were performed to compare the student's perceptions between public and private schools. RESULTS A total of 671 responses were included in the study period. Most participants were from public schools (81%) and female (61%). Almost every respondent (94%) indicated it was necessary to obtain COVID-19 education, yet only half (54%) received such training. Students in private schools were less likely to have their clinical instruction canceled (53% vs. 77%, p = 0.001) and more likely to have access to virtual instruction (46% vs. 22%, p = 0.001) when compared to students from public schools. Four out of every five students considered their training inferior to that of previous generations, and most students (82%) would consider repeating their final year of clinical training. CONCLUSIONS The impact of the COVID-19 on medical education in Mexico has been significant. Most final-year medical students have been affected by the cancellation of their in-person clinical instruction, for which the majority would consider repeating their final year of training. Efforts to counterbalance this lack of clinical experience with virtual or simulation instruction are needed.
Collapse
Affiliation(s)
| | | | | | - Julio Chavez-Vela
- Faculty of Medicine, Universidad Autónoma del Estado de Morelos, Cuernavaca, Morelos, Mexico
| | - Vera L Petricevich
- Faculty of Medicine, Universidad Autónoma del Estado de Morelos, Cuernavaca, Morelos, Mexico
| | - Ignacio García-Juárez
- Liver Transplant Unit, Instituto Nacional de Ciencias Médicas Y Nutrición "Salvador Zubirán", Mexico City, Mexico
| | - Alice Gallo de Moraes
- Division of Pulmonary and Critical Care Medicine, Mayo Clinic, Rochester, Minnesota, USA
| | - Benjamin Zendejas
- Department of Surgery, Boston Children Hospital, Harvard Medical School, 300 Longwood Avenue, Boston, MA, 02115, USA.
| |
Collapse
|
6
|
Nataraja RM, Oo YM, Ljuhar D, Webb NR, Pacilli M, Win NN, Aye A. Overview of a novel paediatric surgical simulation‐based medical education programme in Myanmar. ANZ J Surg 2020; 90:1925-1932. [DOI: 10.1111/ans.16200] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/11/2020] [Revised: 06/29/2020] [Accepted: 07/10/2020] [Indexed: 12/21/2022]
Affiliation(s)
- Ramesh M. Nataraja
- Department of Paediatric Surgery and Surgical Simulation Monash Children's Hospital Melbourne Victoria Australia
- Departments of Paediatrics and Surgery, School of Clinical Sciences, Faculty of Medicine, Nursing and Health Sciences Monash University Melbourne Victoria Australia
| | - Yin Mar Oo
- Department of Paediatric Surgery Yangon Children's Hospital Yangon Myanmar
| | - Damir Ljuhar
- Department of Paediatric Surgery and Surgical Simulation Monash Children's Hospital Melbourne Victoria Australia
- Departments of Paediatrics and Surgery, School of Clinical Sciences, Faculty of Medicine, Nursing and Health Sciences Monash University Melbourne Victoria Australia
| | - Nathalie R. Webb
- Department of Paediatric Surgery and Surgical Simulation Monash Children's Hospital Melbourne Victoria Australia
- Departments of Paediatrics and Surgery, School of Clinical Sciences, Faculty of Medicine, Nursing and Health Sciences Monash University Melbourne Victoria Australia
| | - Maurizio Pacilli
- Department of Paediatric Surgery and Surgical Simulation Monash Children's Hospital Melbourne Victoria Australia
- Departments of Paediatrics and Surgery, School of Clinical Sciences, Faculty of Medicine, Nursing and Health Sciences Monash University Melbourne Victoria Australia
| | - Nyo Nyo Win
- Department of Paediatric Surgery Yankin Children's Hospital Yangon Myanmar
| | - Aye Aye
- Department of Paediatric Surgery Yangon Children's Hospital Yangon Myanmar
| |
Collapse
|