1
|
Birman NA, Vashdi DR, Miller-Mor Atias R, Riskin A, Zangen S, Litmanovitz I, Sagi D. Unveiling the paradoxes of implementing post graduate competency based medical education programs. MEDICAL TEACHER 2024:1-8. [PMID: 38803298 DOI: 10.1080/0142159x.2024.2356826] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/05/2023] [Accepted: 05/14/2024] [Indexed: 05/29/2024]
Abstract
PURPOSE Competency-based medical education (CBME) has gained prominence as an innovative model for post-graduate medical education, yet its implementation poses significant challenges, especially with regard to its sustainability. Drawing on paradox theory, we suggest that revealing the paradoxes underlying these challenges may contribute to our understanding of post graduate competency-based medical education (PGCBME) implementation processes and serve as a first-step in enhancing better implementation. Thus, the purpose of the current study is to identify the paradoxes associated with PGCBME implementation. METHOD A qualitative study was conducted, as part of a larger action research, using in-depth semi-structured interviews with fellows and educators in eight Neonatal wards. RESULTS Analysis revealed that the PGCBME program examined in this study involves three different levels of standardization, each serving as one side of paradoxical tensions; (1) a paradox between the need for standardized assessment tools and for free-flow flexible assessment tools, (2) a paradox between the need for a standardized implementation process across all wards and the need for unique implementation protocols in each ward; and 3) a paradox between the need for a standardized meaning of competency proficiency and the need for flexible and personal competency achievement indicators. CONCLUSIONS Implementing PGCBME programs involves many challenges, some of which are paradoxical, i.e. two contradictory challenges in which solving one challenge exacerbates another. Revealing these paradoxes is important in navigating them successfully.
Collapse
Affiliation(s)
- Noa A Birman
- University of Haifa, The Herta and Paul Amir Faculty of Social Sciences, School of Political Science, Department of Public Administration, Haifa, Israel
| | - Dana R Vashdi
- University of Haifa, The Herta and Paul Amir Faculty of Social Sciences, School of Political Science, Department of Public Administration, Haifa, Israel
| | - Rotem Miller-Mor Atias
- University of Haifa, The Herta and Paul Amir Faculty of Social Sciences, School of Political Science, Department of Public Administration, Haifa, Israel
| | - Arieh Riskin
- Technion Israel Institute of Technology, The Ruth and Bruce Rappaport Faculty of Medicine, Haifa, Israel
| | - Shmuel Zangen
- Ben- Gurion University of the Negev, Faculty of Health Sciences, Be'er-Sheva, Israel
| | - Ita Litmanovitz
- Tel Aviv University, Faculty of Medicine & Health Sciences, Tel-Aviv, Israel
| | - Doron Sagi
- The Israel Center for Medical Simulation, Sheba Medical Center, Tel-Hashomer, Ramat-Gan, Israel
| |
Collapse
|
2
|
Andreou V, Peters S, Eggermont J, Schoenmakers B. Co-designing Entrustable Professional Activities in General Practitioner's training: a participatory research study. BMC MEDICAL EDUCATION 2024; 24:549. [PMID: 38760773 PMCID: PMC11100052 DOI: 10.1186/s12909-024-05530-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/25/2023] [Accepted: 05/07/2024] [Indexed: 05/19/2024]
Abstract
BACKGROUND In medical education, Entrustable Professional Activities (EPAs) have been gaining momentum for the last decade. Such novel educational interventions necessitate accommodating competing needs, those of curriculum designers, and those of users in practice, in order to be successfully implemented. METHODS We employed a participatory research design, engaging diverse stakeholders in designing an EPA framework. This iterative approach allowed for continuous refinement, shaping a comprehensive blueprint comprising 60 EPAs. Our approach involved two iterative cycles. In the first cycle, we utilized a modified-Delphi methodology with clinical competence committee (CCC) members, asking them whether each EPA should be included. In the second cycle, we used semi-structured interviews with General Practitioner (GP) trainers and trainees to explore their perceptions about the framework and refine it accordingly. RESULTS During the first cycle, 14 CCC members agreed that all the 60 EPAs should be included in the framework. Regarding the formulation of each EPAs, 20 comments were given and 16 adaptations were made to enhance clarity. In the second cycle, the semi-structured interviews with trainers and trainees echoed the same findings, emphasizing the need of the EPA framework for improving workplace-based assessment, and its relevance to real-world clinical scenarios. However, trainees and trainers expressed concerns regarding implementation challenges, such as the large number of EPAs to be assessed, and perception of EPAs as potentially high-stakes. CONCLUSION Accommodating competing stakeholders' needs during the design process can significantly enhance the EPA implementation. Recognizing users as experts in their own experiences empowers them, enabling a priori identification of implementation barriers and potential pitfalls. By embracing a collaborative approach, wherein diverse stakeholders contribute their unique viewpoints, we can only create effective educational interventions to complex assessment challenges.
Collapse
Affiliation(s)
- Vasiliki Andreou
- Academic Centre for General Practice, Department of Public Health and Primary Care, KU Leuven, Leuven, Belgium.
- Department of Public Health and Primary Care, KU Leuven, Box 7001, Kapucijnenvoer 7, Leuven, 3000, Belgium.
| | - Sanne Peters
- Academic Centre for General Practice, Department of Public Health and Primary Care, KU Leuven, Leuven, Belgium
- School of Health Sciences, Faculty of Medicine, Dentistry and Health Sciences, The University of Melbourne, Melbourne, Australia
| | - Jan Eggermont
- Department of Cellular and Molecular Medicine, KU Leuven, Leuven, Belgium
| | - Birgitte Schoenmakers
- Academic Centre for General Practice, Department of Public Health and Primary Care, KU Leuven, Leuven, Belgium
| |
Collapse
|
3
|
D’Abbondanza JA, Shih JG, Knox ADC, Zhygan N, Brown MH, Fish JS, Courtemanche DJ. Resident Exposure and Involvement in Core Procedural Competencies within Pediatric Plastic Surgery. Plast Surg (Oakv) 2024; 32:347-354. [PMID: 38681244 PMCID: PMC11046279 DOI: 10.1177/22925503221109072] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/19/2022] [Revised: 05/02/2022] [Accepted: 05/11/2022] [Indexed: 05/01/2024] Open
Abstract
Introduction: The implementation of competency-based residency training in plastic surgery is underway. Key competencies in plastic surgery have been previously identified, however, within the domain of pediatrics, data suggest limited exposure throughout training for Canadian graduates. This study aims to identify the exposure and involvement of residents in core pediatric cases. Methods: We performed a retrospective, multicenter review of plastic surgery resident case logs (T-Res, POWER, New Innovations) across 10 Canadian, English-speaking training programs between 2004 and 2014. Case logs were coded according to the 8 core pediatric competencies previously identified by a modified Delphi technique. Results: A total of 3061 of 59 405 cases (5.2%) logged by 55 graduating residents were core pediatric procedures with an average of 55.6 ± 23.0 cases logged per resident. The top 3 most commonly logged procedures were cleft lip repair, cleft palate repair, and setback otoplasty. The number of cases per program varied widely with the most at 731 and least at 85 logged cases. Roles across procedures have wide variation and residents are most commonly identified as the assistant rather than surgeon or co-surgeon. Conclusion: These findings highlight variability both within and across residency programs with a paucity of exposure and involvement in pediatric plastic surgery cases. This may present a conflict between current recommendations for residency-specific procedural competencies and true clinical exposure. Further curriculum development and simulation may be of benefit.
Collapse
Affiliation(s)
- Josephine A. D’Abbondanza
- Division of Plastic, Reconstructive and Aesthetic Surgery, Department of Surgery, University of Toronto, Toronto, Ontario, Canada
| | - Jessica G. Shih
- Division of Plastic, Reconstructive and Aesthetic Surgery, Department of Surgery, University of Toronto, Toronto, Ontario, Canada
| | - Aaron D. C. Knox
- Division of Plastic and Reconstructive Surgery, Department of Surgery, University of Calgary, Calgary, Alberta, Canada
| | - Nick Zhygan
- Division of Plastic and Reconstructive Surgery, Department of Surgery, University of British Columbia, Victoria, British Columbia, Canada
| | - Mitchell H. Brown
- Division of Plastic, Reconstructive and Aesthetic Surgery, Department of Surgery, University of Toronto, Toronto, Ontario, Canada
| | - Joel S. Fish
- Division of Plastic, Reconstructive and Aesthetic Surgery, Department of Surgery, University of Toronto, Toronto, Ontario, Canada
| | - Douglas J. Courtemanche
- Division of Plastic and Reconstructive Surgery, Department of Surgery, University of British Columbia, Victoria, British Columbia, Canada
| |
Collapse
|
4
|
Bryant BH, Anderson SR, Brissette M, Childs JM, Gratzinger D, Johnson K, Powell DE, Zein-Eldin Powell S, Timmons CF, McCloskey CB. Leveraging faculty development to support validation of entrustable professional activities assessment tools in anatomic and clinical pathology training. Acad Pathol 2024; 11:100111. [PMID: 38560424 PMCID: PMC10978475 DOI: 10.1016/j.acpath.2024.100111] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/25/2023] [Revised: 01/26/2024] [Accepted: 02/03/2024] [Indexed: 04/04/2024] Open
Abstract
Entrustable professional activities (EPAs) are observable activities that define the practice of medicine and provide a framework of evaluation that has been incorporated into US medical school curricula in both undergraduate and graduate medical education. This manuscript describes the development of an entrustment scale and formative and summative evaluations for pathology EPAs, outlines a process for faculty development that was employed in a pilot study implementing two Anatomic Pathology and two Clinical Pathology EPAs in volunteer pathology residency programs, and provides initial validation data for the proposed pathology entrustment scales. Prior to implementation, faculty development was necessary to train faculty on the entrustment scale for each given activity. A "train the trainer" model used performance dimension training and frame of reference training to train key faculty at each institution. The session utilized vignettes to practice determination of entrustment ratings and development of feedback for trainees as to strengths and weaknesses in the performance of these activities. Validity of the entrustment scale is discussed using the Messick framework, based on concepts of content, response process, and internal structure. This model of entrustment scales, formative and summative assessments, and faculty development can be utilized for any pathology EPA and provides a roadmap for programs to design and implement EPA assessments into pathology residency training.
Collapse
Affiliation(s)
- Bronwyn H. Bryant
- University of Vermont Medical Center, Department of Pathology and Laboratory Medicine, Burlington, VT, USA
| | - Scott R. Anderson
- University of Vermont Medical Center, Department of Pathology and Laboratory Medicine, Burlington, VT, USA
| | - Mark Brissette
- University of Colorado Anschutz Medical Campus, Department of Pathology, Aurora, CO, USA
| | - John M. Childs
- Geisinger Medical Center, Department of Pathology, Danville, PA, USA
| | - Dita Gratzinger
- Stanford University School of Medicine, Department of Pathology, Stanford, CA, USA
| | | | - Deborah E. Powell
- University of Minnesota Medical School, Department of Laboratory Medicine and Pathology, Minneapolis, MN, USA
| | | | - Charles F. Timmons
- UT Southwestern Medical Center, Department of Pathology, Dallas, TX, USA
| | - Cindy B. McCloskey
- University of Oklahoma College of Medicine, Department of Pathology, Oklahoma City, OK, USA
| |
Collapse
|
5
|
Bryant BH, Anderson SR, Brissette M, Childs JM, Gratzinger D, Johnson K, Powell DE, Zein-Eldin Powell S, Timmons CF, Chute D, Cummings TJ, Furlong MA, Hébert TM, Reeves HM, Rush D, Vitkovski T, McCloskey CB. National pilot of entrustable professional activities in pathology residency training. Acad Pathol 2024; 11:100110. [PMID: 38560425 PMCID: PMC10978478 DOI: 10.1016/j.acpath.2024.100110] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2023] [Revised: 10/25/2023] [Accepted: 01/21/2024] [Indexed: 04/04/2024] Open
Abstract
Entrustable professional activities (EPAs) are observable clinical skills and/or procedures that have been introduced into medical education at the student and resident levels in most specialties to determine readiness to advance into residency or independent practice, respectively. This publication describes the process and outcomes of a pilot study looking at the feasibility of using two anatomic pathology and two clinical pathology EPAs in pathology residency in 6 pathology residency programs that volunteered for the study. Faculty development on EPAs and their assessment was provided to pilot program faculty, and EPA assessment tools were developed and used by the pilot programs. Pre- and post-study surveys were given to participating residents, faculty, and program directors to gauge baseline practices and to gather feedback on the EPA implementation experience. Results demonstrated overall good feasibility in implementing EPAs. Faculty acceptance of EPAs varied and was less than that of program directors. Residents reported a significant increase in the frequency with which faculty provided formative assessments that included specific examples of performance and specific ways to improve, as well as increased frequency with which faculty provided summative assessments that included specific ways to improve. EPAs offered the most benefit in setting clear expectations for performance of each task, for providing more specific feedback to residents, and in increasing Program director's understanding of resident strengths abilities and weaknesses.
Collapse
Affiliation(s)
- Bronwyn H. Bryant
- University of Vermont Medical Center, Department of Pathology and Laboratory Medicine, Burlington, VT, USA
| | - Scott R. Anderson
- University of Vermont Medical Center, Department of Pathology and Laboratory Medicine, Burlington, VT, USA
| | - Mark Brissette
- University of Colorado Anschutz Medical Campus, Department of Pathology, Aurora, CO, USA
| | - John M. Childs
- Geisinger Medical Center, Department of Pathology, Danville, PA, USA
| | - Dita Gratzinger
- Stanford University School of Medicine, Department of Pathology, Stanford, CA, USA
| | | | - Deborah E. Powell
- University of Minnesota Medical School, Department of Laboratory Medicine and Pathology, Minneapolis, MN, USA
| | | | - Charles F. Timmons
- UT Southwestern Medical Center, Department of Pathology, Dallas, TX, USA
| | - Deborah Chute
- Cleveland Clinic, Department of Pathology and Laboratory Medicine, Cleveland, OH, USA
| | | | - Mary A. Furlong
- Georgetown University School of Medicine, Department of Pathology and Laboratory Medicine, Washington, DC, USA
| | - Tiffany M. Hébert
- Montefiore Health System/Albert Einstein College of Medicine, Department of Pathology, Bronx, NY, USA
| | - Hollie M. Reeves
- University Hospitals Cleveland Medical Center, Case Western Reserve University School of Medicine, Department of Pathology, Cleveland, OH, USA
| | - Demaretta Rush
- University of Arizona College of Medicine, Department of Pathology, Tucson, AZ, USA
| | - Taisia Vitkovski
- Zucker School of Medicine at Hofstra/Northwell Health, Department of Pathology and Laboratory Medicine, Hempstead, NY, USA
| | - Cindy B. McCloskey
- University of Oklahoma College of Medicine, Department of Pathology, Oklahoma City, OK, USA
| |
Collapse
|
6
|
Balvardi S, Kaneva P, Semsar-Kazerooni K, Vassiliou M, Al Mahroos M, Mueller C, Fiore JF, Schwartzman K, Feldman LS. Effect of video-based self-reflection on intraoperative skills: A pilot randomized controlled trial. Surgery 2024; 175:1021-1028. [PMID: 38154996 DOI: 10.1016/j.surg.2023.11.028] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2023] [Revised: 10/13/2023] [Accepted: 11/26/2023] [Indexed: 12/30/2023]
Abstract
BACKGROUND The value of video-based self-assessment in enhancing surgical skills is uncertain. This study investigates the feasibility and estimates sample size for a full-scale randomized controlled trial to evaluate the effectiveness of video-based self-assessment to improve surgical performance of laparoscopic cholecystectomy in trainees. METHODS This parallel pilot randomized controlled trial included general surgery trainees performing supervised laparoscopic cholecystectomy randomized 1:1 to control (traditional intraoperative teaching) or intervention group (traditional teaching plus video-based self-assessment). Operative performance was measured by the attending surgeon blinded to group assignment at the time of surgery using standardized assessment tools (Global Operative Assessment of Laparoscopic Skills and Operative Performance Rating System). The intervention group had access to their video recordings on a web-based platform for review and self-assessment using the same instruments. The primary outcome for the estimation of sample size was the difference in faculty-assessed final operative performance (third submitted case). Feasibility criteria included >85% participation, >85% adherence to case submission and >85% completion of self-assessment. RESULTS Of 37 eligible trainees approached, 32 consented and were randomized (86%). There were 16 in the intervention group, 15 in the control group (55% male, 55% junior trainees), and 1 was excluded for protocol violation. Twenty-four (75%) of participants submitted 3 cases. Thirteen trainees (81%) accessed the platform and completed 26 (63.2%) case self-assessments. Fifty-five trainees per arm will be needed to power a full-scale laparoscopic cholecystectomy with Global Operative Assessment of Laparoscopic Skills and 130 trainees per arm with Operative Performance Rating System as the assessment tool. CONCLUSION This pilot study contributes important data to inform the design of an adequately powered randomized controlled trial of video-based self-assessment to improve trainee performance of laparoscopic cholecystectomy. Although a priori trial feasibility criteria were not achieved, automated video capture and storage could significantly improve adherence in future trials.
Collapse
Affiliation(s)
- Saba Balvardi
- Department of Surgery, McGill University, Montreal, Quebec, Canada; Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, Quebec, Canada
| | - Pepa Kaneva
- Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, Quebec, Canada
| | - Koorosh Semsar-Kazerooni
- Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, Quebec, Canada
| | - Melina Vassiliou
- Department of Surgery, McGill University, Montreal, Quebec, Canada; Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, Quebec, Canada
| | | | - Carmen Mueller
- Department of Surgery, McGill University, Montreal, Quebec, Canada; Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, Quebec, Canada
| | - Julio F Fiore
- Department of Surgery, McGill University, Montreal, Quebec, Canada; Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, Quebec, Canada
| | - Kevin Schwartzman
- Respiratory Division, Department of Medicine, McGill University and McGill International Tuberculosis Centre, Research Institute of the McGill University Health Centre, Montreal, Quebec, Canada
| | - Liane S Feldman
- Department of Surgery, McGill University, Montreal, Quebec, Canada; Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, Quebec, Canada.
| |
Collapse
|
7
|
Frank JR, Karpinski J, Sherbino J, Snell LS, Atkinson A, Oswald A, Hall AK, Cooke L, Dojeiji S, Richardson D, Cheung WJ, Cavalcanti RB, Dalseg TR, Thoma B, Flynn L, Gofton W, Dudek N, Bhanji F, Wong BMF, Razack S, Anderson R, Dubois D, Boucher A, Gomes MM, Taber S, Gorman LJ, Fulford J, Naik V, Harris KA, St. Croix R, van Melle E. Competence By Design: a transformational national model of time-variable competency-based postgraduate medical education. PERSPECTIVES ON MEDICAL EDUCATION 2024; 13:201-223. [PMID: 38525203 PMCID: PMC10959143 DOI: 10.5334/pme.1096] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/04/2023] [Accepted: 02/16/2024] [Indexed: 03/26/2024]
Abstract
Postgraduate medical education is an essential societal enterprise that prepares highly skilled physicians for the health workforce. In recent years, PGME systems have been criticized worldwide for problems with variable graduate abilities, concerns about patient safety, and issues with teaching and assessment methods. In response, competency based medical education approaches, with an emphasis on graduate outcomes, have been proposed as the direction for 21st century health profession education. However, there are few published models of large-scale implementation of these approaches. We describe the rationale and design for a national, time-variable competency-based multi-specialty system for postgraduate medical education called Competence by Design. Fourteen innovations were bundled to create this new system, using the Van Melle Core Components of competency based medical education as the basis for the transformation. The successful execution of this transformational training system shows competency based medical education can be implemented at scale. The lessons learned in the early implementation of Competence by Design can inform competency based medical education innovation efforts across professions worldwide.
Collapse
Affiliation(s)
- Jason R. Frank
- Centre for Innovation in Medical Education and Professor, Department of Emergency Medicine, Faculty of Medicine, University of Ottawa, ON, Canada
| | - Jolanta Karpinski
- Department of Medicine, University of Ottawa, Ottawa, ON, Canada
- Competency Based Medical Education, University of Ottawa, Ottawa, ON, Canada
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
| | | | - Linda S. Snell
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Medicine and Health Sciences Education, McGill University, Montreal, QC, Canada
| | - Adelle Atkinson
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Department of Paediatrics, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada
| | - Anna Oswald
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Department of Medicine, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, AB, Canada
- Competency Based Medical Education, University of Alberta, Edmonton, AB, Canada
| | - Andrew K. Hall
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Department of Emergency Medicine, University of Ottawa, Ottawa, ON, Canada
| | - Lara Cooke
- Division of Neurology, Department of Clinical Neurosciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
| | - Susan Dojeiji
- Physical Medicine and Rehabilitation, University of Ottawa, Ottawa, ON, Canada
| | - Denyse Richardson
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Department of Physical Medicine and Rehabilitation, Queen’s University, Kingston, ON, Canada
| | - Warren J. Cheung
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Department of Medicine, University of Toronto, Toronto, ON, Canada
| | - Rodrigo B. Cavalcanti
- Department of Medicine, University of Toronto, Toronto, ON, Canada
- HoPingKong Centre, University Health Network, Toronto, ON, Canada
| | - Timothy R. Dalseg
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Division of Emergency Medicine, University of Toronto, Toronto, ON, Canada
| | - Brent Thoma
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Emergency Medicine, University of Saskatchewan, Saskatoon, SK, Canada
| | - Leslie Flynn
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Departments of Psychiatry and Family Medicine, and Co-Director Master of Health Sciences Education, Queen’s University, Kingston, ON, Canada
| | - Wade Gofton
- Department of Surgery (Division of Orthopedic Surgery), The Ottawa Hospital and University of Ottawa, Ottawa, ON, Canada
| | - Nancy Dudek
- Department of Medicine (Division of Physical Medicine & Rehabilitation) and The Ottawa Hospital, University of Ottawa, Ottawa, Ontario, Canada
| | - Farhan Bhanji
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Faculty of Medicine and Health Sciences, McGill University, Montreal, QC, Canada
| | - Brian M.-F. Wong
- Centre for Quality Improvement and Patient Safety, University of Toronto, Toronto, Canada
| | - Saleem Razack
- Centre for Health Education Scholarship, University of British Columbia and BC Children’s Hospital, Vancouver, BC, Canada
| | - Robert Anderson
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Northern Ontario School of Medicine University, Sudbury, ON, Canada
| | - Daniel Dubois
- Department of Anesthesiology and Pain Medicine, University of Ottawa, Ottawa, ON, Canada
| | - Andrée Boucher
- Department of Medicine (Division of Endocrinology), Universitéde Montréal, Montréal, QC, Canada
| | - Marcio M. Gomes
- Department of Pathology and Laboratory Medicine, University of Ottawa, Ottawa, ON, Canada
| | - Sarah Taber
- Office of Standards and Assessment, Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
| | - Lisa J. Gorman
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
| | - Jane Fulford
- Canadian Internet Registration Authority, Canada
| | - Viren Naik
- Department of Anesthesiology and Pain Medicine, University of Ottawa, Ottawa, ON, Canada
- Medical Council of Canada, Ottawa, ON, Canada
| | - Kenneth A. Harris
- Royal College of Physicians and Surgeons of Canada, Canada
- Emeritus, Western University, Canada
| | - Rhonda St. Croix
- Learning and Connecting at the Royal College of Physicians and Surgeons of Canada, Canada
| | - Elaine van Melle
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Department of Family Medicine, Queen’s University, Kingston, ON, Canada
| |
Collapse
|
8
|
Gholami S, Manon A, Yao K, Billard A, Meling TR. An objective skill assessment framework for microsurgical anastomosis based on ALI scores. Acta Neurochir (Wien) 2024; 166:104. [PMID: 38400918 DOI: 10.1007/s00701-024-05934-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/29/2023] [Accepted: 12/01/2023] [Indexed: 02/26/2024]
Abstract
INTRODUCTION The current assessment and standardization of microsurgical skills are subjective, posing challenges in reliable skill evaluation. We aim to address these limitations by developing a quantitative and objective framework for accurately assessing and enhancing microsurgical anastomosis skills among surgical trainees. We hypothesize that this framework can differentiate the proficiency levels of microsurgeons, aligning with subjective assessments based on the ALI score. METHODS We select relevant performance metrics from the literature on laparoscopic skill assessment and human motor control studies, focusing on time, instrument kinematics, and tactile information. This information is measured and estimated by a set of sensors, including cameras, a motion capture system, and tactile sensors. The recorded data is analyzed offline using our proposed evaluation framework. Our study involves 12 participants of different ages ([Formula: see text] years) and genders (nine males and three females), including six novice and six intermediate subjects, who perform surgical anastomosis procedures on a chicken leg model. RESULTS We show that the proposed set of objective and quantitative metrics to assess skill proficiency aligns with subjective evaluations, particularly the ALI score method, and can effectively differentiate novices from more proficient microsurgeons. Furthermore, we find statistically significant disparities, where microsurgeons with intermediate level of skill proficiency surpassed novices in both task speed, reduced idle time, and smoother, briefer hand displacements. CONCLUSION The framework enables accurate skill assessment and provides objective feedback for improving microsurgical anastomosis skills among surgical trainees. By overcoming the subjectivity and limitations of current assessment methods, our approach contributes to the advancement of surgical education and the development of aspiring microsurgeons. Furthermore, our framework emerges to precisely distinguish and classify proficiency levels (novice and intermediate) exhibited by microsurgeons.
Collapse
Affiliation(s)
- Soheil Gholami
- Learning Algorithms and Systems Laboratory (LASA), École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland.
| | - Anaëlle Manon
- Learning Algorithms and Systems Laboratory (LASA), École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Kunpeng Yao
- Learning Algorithms and Systems Laboratory (LASA), École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Aude Billard
- Learning Algorithms and Systems Laboratory (LASA), École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Torstein R Meling
- Department of Neurosurgery, The National Hospital of Denmark, Rigshospitalet, Copenhagen, Denmark
| |
Collapse
|
9
|
Ng IKS, Mok SF, Teo D. Competency in medical training: current concepts, assessment modalities, and practical challenges. Postgrad Med J 2024:qgae023. [PMID: 38376156 DOI: 10.1093/postmj/qgae023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2023] [Accepted: 01/23/2024] [Indexed: 02/21/2024]
Abstract
In the past two decades, competency-based medical education (CBME) has rapidly become the cornerstone of medical training and accreditation programmes worldwide. It has increasingly replaced traditional time-based educational approaches which were often rigid, fragmented, and overly emphasized clinical content knowledge over practical skillsets and attitudes. CBME adoption was in the hope of better preparing medical graduates for the demands and responsibilities of real-world clinical practice. For all the supposed merits of CBME, there hitherto remains difficulties in arriving at comprehensive and practical 'competency' definitions, and actual challenges with implementation of clinical competency assessment modalities pertaining to construct validity, reliability, and applicability with the use and interpretation of evaluation metrics. Therefore, in this article, we describe the various conceptualizations of 'competency' in medical education literature and attempt to refine its usage in practice to meet the evolving needs and expectations of healthcare stakeholders, as well as incorporate emerging concepts in the medical education discourse. We herein propose that clinical 'competencies' should be defined as multi-domain clinical expertise, comprising medical knowledge, skills, attitudes and metacognitive capabilities that reflects the prevailing needs of healthcare stakeholders, and is inferred from performance evaluations of medical trainees. In order to attain 'competence', there must then be a process of integrating multi-domain competencies into meaningful professional identity formation that is commensurate with the context and stage of medical training. In addition, we review the current competency assessment modalities, including common pitfalls with their usage, and sought to provide practical strategies to mitigate the identified challenges.
Collapse
Affiliation(s)
- Isaac K S Ng
- Department of Medicine, National University Hospital, Singapore 119228, Singapore
- Yong Loo Lin School of Medicine, National University of Singapore 117597, Singapore
| | - Shao Feng Mok
- Yong Loo Lin School of Medicine, National University of Singapore 117597, Singapore
- Division of Endocrinology, Department of Medicine, National University Hospital, Singapore 119228, Singapore
| | - Desmond Teo
- Yong Loo Lin School of Medicine, National University of Singapore 117597, Singapore
- Fast and Chronic Programmes, Alexandra Hospital, Singapore 159964, Singapore
- Division of Advanced Internal Medicine, Department of Medicine, National University Hospital, Singapore 119228, Singapore
| |
Collapse
|
10
|
Cheung WJ, Bhanji F, Gofton W, Hall AK, Karpinski J, Richardson D, Frank JR, Dudek N. Design and Implementation of a National Program of Assessment Model - Integrating Entrustable Professional Activity Assessments in Canadian Specialist Postgraduate Medical Education. PERSPECTIVES ON MEDICAL EDUCATION 2024; 13:44-55. [PMID: 38343554 PMCID: PMC10854461 DOI: 10.5334/pme.956] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 03/08/2023] [Accepted: 09/04/2023] [Indexed: 02/15/2024]
Abstract
Traditional approaches to assessment in health professions education systems, which have generally focused on the summative function of assessment through the development and episodic use of individual high-stakes examinations, may no longer be appropriate in an era of competency based medical education. Contemporary assessment programs should not only ensure collection of high-quality performance data to support robust decision-making on learners' achievement and competence development but also facilitate the provision of meaningful feedback to learners to support reflective practice and performance improvement. Programmatic assessment is a specific approach to designing assessment systems through the intentional selection and combination of a variety of assessment methods and activities embedded within an educational framework to simultaneously optimize the decision-making and learning function of assessment. It is a core component of competency based medical education and is aligned with the goals of promoting assessment for learning and coaching learners to achieve predefined levels of competence. In Canada, postgraduate specialist medical education has undergone a transformative change to a competency based model centred around entrustable professional activities (EPAs). In this paper, we describe and reflect on the large scale, national implementation of a program of assessment model designed to guide learning and ensure that robust data is collected to support defensible decisions about EPA achievement and progress through training. Reflecting on the design and implications of this assessment system may help others who want to incorporate a competency based approach in their own country.
Collapse
Affiliation(s)
- Warren J. Cheung
- Department of Emergency Medicine, University of Ottawa, Ottawa, ON, CA
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada, 1053 Carling Avenue, Rm F660, Ottawa, ON K1Y 4E9, CA
| | - Farhan Bhanji
- Department of Pediatrics (Critical Care), Faculty of Medicine and Health Sciences, McGill University, Montreal, QC, CA
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, CA
| | - Wade Gofton
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, CA
- Department of Surgery, Division of Orthopaedic Surgery, University of Ottawa, Ottawa, ON, CA
| | - Andrew K. Hall
- Department of Emergency Medicine, University of Ottawa, Ottawa, ON, CA
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, CA
| | - Jolanta Karpinski
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, CA
- Department of Medicine, University of Ottawa, Ottawa, ON, CA
| | - Denyse Richardson
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, CA
- Department of Physical Medicine and Rehabilitation, Queen’s University, Kingston, ON, CA
| | - Jason R. Frank
- Department of Emergency Medicine, Director, Centre for Innovation in Medical Education, Faculty of Medicine, University of Ottawa, Ottawa, ON, CA
| | - Nancy Dudek
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, CA
- Department of Medicine, Division of Physical Medicine and Rehabilitation, University of Ottawa, Ottawa, ON, CA
| |
Collapse
|
11
|
Mattioli DD, Thomas GW, Long S, Rölfing JD, Anderson DD. Fluoroscopic image-based behavior analysis can objectively explain subjective expert assessment of wire navigation skill. J Orthop Res 2024; 42:404-414. [PMID: 37652571 DOI: 10.1002/jor.25685] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/23/2023] [Revised: 06/05/2023] [Accepted: 08/24/2023] [Indexed: 09/02/2023]
Abstract
Psychomotor skill and decision-making efficiency in surgical wire navigation can be objectively evaluated by analysis of intraoperative fluoroscopic image sequences. Prior work suggests that such image-based behavior analysis of operating room (OR) performance can predict performer experience level (R2 = 0.62) and agree with expert opinion (the current standard) on the quality of a final implant construct (R2 = 0.59). However, it is unclear how objective image-based evaluation compares with expert assessments for entire technical OR performances. This study examines the relationships between three key variables: (1) objective image-based criteria, (2) expert opinions, and (3) performing surgeon experience level. A paired-comparison survey of seven experts, based upon eight OR fluoroscopic wire navigation image sequences, shows that the experts' preferences are best explained by objective metrics that reflect psychomotor and decision-making behaviors which are counter-productive to successful implant placement, like image count (R2 = 0.83) and behavior tally (R2 = 0.74). One such behavior, adjustments away from goal, uniquely correlated well with all three key variables: a fluoroscopic image-based analysis composite score (R2 = 0.40), expert consensus (R2 = 0.76), and performer experience (R2 = 0.41). These results confirm that experts view less efficient technical behavior as indicative of lesser technical proficiency. While expert assessments of technical skill were reliable and consistent, neither individual nor consensus expert opinion appears to correlate with performer experience (R2 = 0.11).
Collapse
Affiliation(s)
- Dominik D Mattioli
- Department of Industrial and Systems Engineering, University of Iowa, Iowa City, Iowa, USA
| | - Geb W Thomas
- Department of Industrial and Systems Engineering, University of Iowa, Iowa City, Iowa, USA
- Department of Orthopedics and Rehabilitation, University of Iowa, Iowa City, Iowa, USA
| | - Steven Long
- Department of Orthopedics and Rehabilitation, University of Iowa, Iowa City, Iowa, USA
- Department of Biomedical Engineering, University of Iowa, Iowa City, Iowa, USA
| | - Jan Duedal Rölfing
- Children's Orthopaedics and Reconstruction, Department of Orthopaedics, Aarhus University Hospital, Aarhus, Denmark
- Corporate HR, MidtSim, Aarhus, Denmark
| | - Donald D Anderson
- Department of Industrial and Systems Engineering, University of Iowa, Iowa City, Iowa, USA
- Department of Orthopedics and Rehabilitation, University of Iowa, Iowa City, Iowa, USA
- Department of Biomedical Engineering, University of Iowa, Iowa City, Iowa, USA
| |
Collapse
|
12
|
Mitchell EC, Ott M, Ross D, Grant A. Development of a Tool to Assess Surgical Resident Competence On-Call: The Western University Call Assessment Tool (WUCAT). JOURNAL OF SURGICAL EDUCATION 2024; 81:106-114. [PMID: 38008642 DOI: 10.1016/j.jsurg.2023.10.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/10/2023] [Revised: 09/13/2023] [Accepted: 10/02/2023] [Indexed: 11/28/2023]
Abstract
BACKGROUND A central tenet of competency-based medical education is the formative assessment of trainees. There are currently no assessments designed to examine resident competence on-call, despite the on-call period being a significant component of residency, characterized by less direct supervision compared to daytime. The purpose of this study was to design a formative on-call assessment tool and collect valid evidence on its application. METHODS Nominal group technique was used to identify critical elements of surgical resident competence on-call to inform tool development. The tool was piloted over six months in the Division of Plastic & Reconstructive Surgery at our institution. Quantitative and qualitative evidence was collected to examine tool validity. RESULTS A ten-item tool was developed based on the consensus group results. Sixty-three assessments were completed by seven staff members on ten residents during the pilot. The tool had a reliability coefficient of 0.67 based on a generalizability study and internal item consistency was 0.92. Scores were significantly associated with years of training. We found the tool improved the quantity and structure of feedback given and that the tool was considered feasible and acceptable by both residents and staff members. CONCLUSIONS The Western University Call Assessment Tool (WUCAT) has multiple sources of evidence supporting its use in assessing resident competence on-call.
Collapse
Affiliation(s)
- Eric C Mitchell
- Department of Surgery, Western University, London, Ontario, Canada
| | - Michael Ott
- Department of Surgery, Western University, London, Ontario, Canada
| | - Douglas Ross
- Department of Surgery, Western University, London, Ontario, Canada
| | - Aaron Grant
- Department of Surgery, Western University, London, Ontario, Canada.
| |
Collapse
|
13
|
Quinonez RB, Broome A, Nesbit S, Tittemore A, Kowlowitz V, Duqum I, Uyan D, Musskopf ML, Phillips K, Moretti A, Mason M, Jacobs A, Brian Z, Simancas-Pallares M, Swift EJ, Ramaswamy V, Guthmiller JM. Developing entrustable professional activities for general dentistry at the University of North Carolina. J Dent Educ 2023; 87:1718-1724. [PMID: 37740716 DOI: 10.1002/jdd.13374] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/06/2023] [Revised: 08/25/2023] [Accepted: 09/03/2023] [Indexed: 09/25/2023]
Abstract
INTRODUCTION As part of curriculum innovation, the University of North Carolina (UNC) Adams School of Dentistry identified core entrustable professional activities (EPAs) that graduates must demonstrate for practice readiness. This paper describes the development of the UNC EPAs and the perceptions of the general dentistry faculty. METHODS Upon establishing a blueprint of knowledge, skills, and attitudes of UNC graduates, using a distributed leadership approach, faculty teams developed EPAs focused on the patient care process. The American Dental Education Association Compendium of Clinical Competency Assessments and Commission on Dental Accreditation Standards informed the team's work. Perceptions of the assessment framework were examined using a questionnaire completed by 13 general dentistry faculty considering the importance, accuracy, and agreement of each EPA, associated domains of competence, and encounter management on a 6-point rating scale. RESULTS Distributed leadership was a useful strategy in EPA development to disperse decision-making and build ownership. Through multiple iterations, four EPAs (assessment, plan of care, collaborative care, and provision of care) with associated sub-EPAs emerged. EPAs included a description, required knowledge and skills, and rubrics for assessment. The general dentistry faculty reported a high level of importance, accuracy, and agreement with EPAs, domains of competence, and encounter management. DISCUSSION EPAs provide a standardized manner to describe the comprehensive work dentists perform, shifting away from individual competencies. The UNC EPAs provide the foundation for longitudinal measures of competence preparing graduates for independent practice. With limited EPAs frameworks available in dentistry, we aim to inform the development and implementation of EPAs across dental education.
Collapse
Affiliation(s)
- Rocio B Quinonez
- University of North Carolina at Chapel Hill Adams School of Dentistry, Chapel Hill, North Carolina, USA
| | - Angela Broome
- Division of Diagnostic Sciences, University of North Carolina at Chapel Hill Adams School of Dentistry, Chapel Hill, North Carolina, USA
| | - Sam Nesbit
- Division of Comprehensive Oral Health, University of North Carolina at Chapel Hill Adams School of Dentistry, Chapel Hill, North Carolina, USA
| | - Ashley Tittemore
- University of North Carolina at Chapel Hill Adams School of Dentistry, Chapel Hill, North Carolina, USA
| | - Vicki Kowlowitz
- University of North Carolina at Chapel Hill Adams School of Dentistry, Chapel Hill, North Carolina, USA
| | - Ibrahim Duqum
- Division of Comprehensive Oral Health, University of North Carolina at Chapel Hill Adams School of Dentistry, Chapel Hill, North Carolina, USA
| | - Dilek Uyan
- Division of Comprehensive Oral Health, University of North Carolina at Chapel Hill Adams School of Dentistry, Chapel Hill, North Carolina, USA
| | - Marta L Musskopf
- Division of Comprehensive Oral Health, University of North Carolina at Chapel Hill Adams School of Dentistry, Chapel Hill, North Carolina, USA
| | - Keith Phillips
- Division of Comprehensive Oral Health, University of North Carolina at Chapel Hill Adams School of Dentistry, Chapel Hill, North Carolina, USA
| | - Antonio Moretti
- Division of Comprehensive Oral Health, University of North Carolina at Chapel Hill Adams School of Dentistry, Chapel Hill, North Carolina, USA
| | - Matt Mason
- Division of Comprehensive Oral Health, University of North Carolina at Chapel Hill Adams School of Dentistry, Chapel Hill, North Carolina, USA
| | - Alexa Jacobs
- University of North Carolina at Chapel Hill Adams School of Dentistry, Chapel Hill, North Carolina, USA
| | - Zachary Brian
- Division of Pediatric and Public Health, University of North Carolina at Chapel Hill Adams School of Dentistry, Chapel Hill, North Carolina, USA
| | - Miguel Simancas-Pallares
- Division of Pediatric and Public Health, University of North Carolina at Chapel Hill Adams School of Dentistry, Chapel Hill, North Carolina, USA
| | - Edward J Swift
- University of North Carolina at Chapel Hill Adams School of Dentistry, Chapel Hill, North Carolina, USA
| | - Vidya Ramaswamy
- University of Michigan School of Dentistry, Ann Arbor, Michigan, USA
| | - Janet M Guthmiller
- University of North Carolina at Chapel Hill Adams School of Dentistry, Chapel Hill, North Carolina, USA
| |
Collapse
|
14
|
Gutierrez G, Lunsky IO, Van Heer S, Szulewski A, Chaplin T. Cognitive load theory in action: e-learning modules improve performance in simulation-based education. A pilot study. CAN J EMERG MED 2023; 25:893-901. [PMID: 37751082 DOI: 10.1007/s43678-023-00586-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2023] [Accepted: 08/17/2023] [Indexed: 09/27/2023]
Abstract
PURPOSE Learners participating in simulation-based education may experience cognitive overload with potential detrimental effects to learning and performance. Multiple strategies have been proposed to mitigate this detrimental response. However, these strategies have not fully considered the potential benefits of using online platforms, such as accessibility, cost-effectiveness, efficiency, and scalability. Addressing this gap in the literature, preparatory online modules were developed by applying concepts from cognitive load theory and simulation-based education. This study assessed whether using preparatory online modules to deliver weekly pre-briefing content could impact cognitive load and performance. The participants were first-year postgraduate medical trainees participating in a simulation-based resuscitation curriculum. METHODS Fifty-three trainees were allocated to receive preparatory online modules (online modules group, n = 27) or not (control group, n = 26) during the course component of a simulation-based resuscitation curriculum. Then, these trainees participated in a simulation-based objective structured clinical examination (OSCE). Sources of cognitive load (intrinsic, extraneous, and germane) were measured using a modified cognitive load questionnaire. Performance was assessed using the Ottawa Surgical Competency Operating Room Evaluation. Data were analyzed with descriptive statistics, and principal component analysis. RESULTS During the course component, the online modules group was found to have higher intrinsic and germane cognitive load, and lower extraneous cognitive load compared to the control group. During the OSCE, the online modules group scored significantly higher in performance scores (p = 0.0077, d = 0.39, 95% confidence interval = 0.10;0.68) compared to the control group. Principal component analysis supported the results obtained with the modified cognitive load questionnaire. CONCLUSION Trainees using preparatory online modules during the course component of a simulation-based resuscitation curriculum experienced cognitive load changes consistent with cognitive optimization. This may have contributed to their superior performance in the subsequent OSCE. Future research should explore the long-term impacts of online preparatory training and consider potential barriers to implementation in diverse healthcare environments.
Collapse
Affiliation(s)
- Gilmar Gutierrez
- Department of Psychiatry, Kingston Health Sciences Center, Kingston, ON, Canada
| | - Isis Olivia Lunsky
- Faculty of Health Sciences, Queen's School of Medicine, Kingston, ON, Canada
| | - Shyan Van Heer
- Department of Family Medicine, University of Toronto, Toronto, ON, Canada
| | - Adam Szulewski
- Department of Emergency Medicine, Queen's University, Kingston, ON, Canada
- Department of Psychology, Queen's University, Kingston, ON, Canada
| | - Timothy Chaplin
- Department of Emergency Medicine, Queen's University, Kingston, ON, Canada.
- Department of Emergency Medicine, Kingston General Hospital, Kingston, ON, Canada.
| |
Collapse
|
15
|
Faber DA, Hinman JM, Knauer EM, Hechenbleikner EM, Badell IR, Lin E, Srinivasan JK, Chahine AA, Papandria DJ. Implementation of an Online Intraoperative Assessment of Technical Performance for Surgical Trainees. J Surg Res 2023; 291:574-585. [PMID: 37540975 DOI: 10.1016/j.jss.2023.07.008] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2023] [Revised: 06/29/2023] [Accepted: 07/02/2023] [Indexed: 08/06/2023]
Abstract
INTRODUCTION Assessment of surgical resident technical performance is an integral component of any surgical training program. Timely assessment delivered in a structured format is a critical step to enhance technical skills, but residents often report that the quality and quantity of timely feedback received is lacking. Moreover, the absence of written feedback with specificity can allow residents to seemingly progress in their operative milestones as a junior resident, but struggle as they progress into their postgraduate year 3 and above. We therefore designed and implemented a web-based intraoperative assessment tool and corresponding summary "dashboard" to facilitate real-time assessment and documentation of technical performance. MATERIALS AND METHODS A web form was designed leveraging a cloud computing platform and implementing a modified Ottawa Surgical Competency Operating Room Evaluation instrument; this included additional, procedure-specific criteria for select operations. A link to this was provided to residents via email and to all surgical faculty as a Quick Response code. Residents open and complete a portion of the form on a smartphone, then relinquish the device to an attending surgeon who then completes and submits the assessment. The data are then transferred to a secure web-based reporting interface; each resident (together with a faculty advisor) can then access and review all completed assessments. RESULTS The Assessment form was activated in June 2021 and formally introduced to all residents in July 2021, with residents required to complete at least one assessment per month. Residents with less predictable access to operative procedures (night float or Intensive Care Unit) were exempted from the requirement on those months. To date a total of 559 assessments have been completed for operations performed by 56 trainees, supervised by 122 surgical faculty and senior trainees. The mean number of procedures assessed per resident was 10.0 and the mean number per assessor was 4.6. Resident initiation of Intraoperative Assessments has increased since the tool was introduced and scores for technical and nontechnical performance reliably differentiate residents by seniority. CONCLUSIONS This novel system demonstrates that an online, resident-initiated technical assessment tool is feasible to implement and scale. This model's requirement that the attending enter performance ratings into the trainee's electronic device ensures that feedback is delivered directly to the trainee. Whether this aspect of our assessment ensures more direct and specific (and therefore potentially actionable) feedback is a focus for future study. Our use of commercial cloud computing services should permit cost-effective adoption of similar systems at other training programs.
Collapse
Affiliation(s)
- David A Faber
- Department of Surgery, Emory University School of Medicine, Atlanta, Georgia
| | - Johanna M Hinman
- Department of Surgery, Emory University School of Medicine, Atlanta, Georgia
| | - Eric M Knauer
- Department of Surgery, Emory University School of Medicine, Atlanta, Georgia
| | | | - I Raul Badell
- Department of Surgery, Emory University School of Medicine, Atlanta, Georgia
| | - Edward Lin
- Department of Surgery, Emory University School of Medicine, Atlanta, Georgia
| | | | - A Alfred Chahine
- Department of Surgery, Emory University School of Medicine, Atlanta, Georgia
| | - Dominic J Papandria
- Department of Surgery, Emory University School of Medicine, Atlanta, Georgia.
| |
Collapse
|
16
|
Dewhirst S, Wood TJ, Cheung WJ, Frank JR. Assessing the utility of a novel entrustment-supervision assessment tool. MEDICAL EDUCATION 2023; 57:949-957. [PMID: 37387266 DOI: 10.1111/medu.15156] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/30/2022] [Revised: 06/05/2023] [Accepted: 06/10/2023] [Indexed: 07/01/2023]
Abstract
BACKGROUND Work-based assessments (WBAs) are increasingly used to inform decisions about trainee progression. Unfortunately, WBAs often fail to discriminate between trainees of differing abilities and have poor reliability. Entrustment-supervision scales may improve WBA performance, but there is a paucity of literature directly comparing them to traditional WBA tools. METHODS The Ottawa Emergency Department Shift Observation Tool (O-EDShOT) is a previously published WBA tool employing an entrustment-supervision scale with strong validity evidence. This pre-/post-implementation study compares the performance of the O-EDShOT with that of a traditional WBA tool using norm-based anchors. All assessments completed in 12-month periods before and after implementing the O-EDShOT were collected, and generalisability analysis was conducted with year of training, trainees within year and forms within trainee as nested factors. Secondary analysis included assessor as a factor. RESULTS A total of 3908 and 3679 assessments were completed by 99 and 116 assessors, for 152 and 138 trainees in the pre- and post-implementation phases respectively. The O-EDShOT generated a wider range of awarded scores than the traditional WBA, and mean scores increased more with increasing level of training (0.32 vs. 0.14 points per year, p = 0.01). A significantly greater proportion of overall score variability was attributable to trainees using the O-EDShOT (59%) compared with the traditional tool (21%, p < 0.001). Assessors contributed less to overall score variability for the O-EDShOT than for the traditional WBA (16% vs. 37%). Moreover, the O-EDShOT required fewer completed assessments than the traditional tool (27 vs. 51) for a reliability of 0.8. CONCLUSION The O-EDShOT outperformed a traditional norm-referenced WBA in discriminating between trainees and required fewer assessments to generate a reliable estimate of trainee performance. More broadly, this study adds to the body of literature suggesting that entrustment-supervision scales generate more useful and reliable assessments in a variety of clinical settings.
Collapse
Affiliation(s)
| | - Timothy J Wood
- Department of Innovation in Medical Education, University of Ottawa, Ottawa, Canada
| | - Warren J Cheung
- Department of Emergency Medicine, University of Ottawa, Ottawa, Canada
| | - Jason R Frank
- Department of Emergency Medicine, University of Ottawa, Ottawa, Canada
| |
Collapse
|
17
|
Sebok-Syer SS, Lingard L, Panza M, Van Hooren TA, Rassbach CE. Supportive and collaborative interdependence: Distinguishing residents' contributions within health care teams. MEDICAL EDUCATION 2023; 57:921-931. [PMID: 36822577 DOI: 10.1111/medu.15064] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/21/2022] [Revised: 02/04/2023] [Accepted: 02/21/2023] [Indexed: 06/18/2023]
Abstract
INTRODUCTION Individual assessments disregard team contributions, while team assessments disregard an individual's contributions. Interdependence has been put forth as a conceptual bridge between our educational traditions of assessing individual performance and our imminent challenge of assessing team-based performance without losing sight of the individual. The purpose of this study was to develop a more refined conceptualisation of interdependence to inform the creation of measures that can assess the interdependence of residents within health care teams. METHODS Following a constructivist grounded theory approach, we conducted 49 semi-structured interviews with various members of health care teams (e.g. physicians, nurses, pharmacists, social workers and patients) across two different clinical specialties-Emergency Medicine and Paediatrics-at two separate sites. Data collection and analysis occurred iteratively. Constant comparative inductive analysis was used, and coding consisted of three stages: initial, focused and theoretical. RESULTS We asked participants to reflect upon interdependence and describe how it exists in their clinical setting. All participants acknowledged the existence of interdependence, but they did not view it as part of a linear spectrum where interdependence becomes independence. Our analysis refined the conceptualisation of interdependence to include two types: supportive and collaborative. Supportive interdependence occurs within health care teams when one member demonstrates insufficient expertise to perform within their scope of practice. Collaborative interdependence, on the other hand, was not triggered by lack of experience/expertise within an individual's scope of practice, but rather recognition that patient care requires contributions from other team members. CONCLUSION In order to assess a team's collective performance without losing sight of the individual, we need to capture interdependent performances and characterise the nature of such interdependence. Moving away from a linear trajectory where independence is seen as the end goal can also help support efforts to measure an individual's competence as an interdependent member of a health care team.
Collapse
Affiliation(s)
| | - Lorelei Lingard
- Department of Medicine and Centre for Education Research and Innovation, Schulich School of Medicine & Dentistry, Western University, London, Ontario, Canada
| | - Michael Panza
- Centre for Education Research and Innovation, Western University, London, Ontario, Canada
| | - Tamara A Van Hooren
- Department of Pediatrics, Schulich School of Medicine and Dentistry, Western University, London, Ontario, Canada
| | | |
Collapse
|
18
|
Clanahan JM, Yee A, Awad MM. Active control time: an objective performance metric for trainee participation in robotic surgery. J Robot Surg 2023; 17:2117-2123. [PMID: 37237112 DOI: 10.1007/s11701-023-01628-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/08/2023] [Accepted: 05/21/2023] [Indexed: 05/28/2023]
Abstract
Trainee participation and progression in robotic general surgery remain poorly defined. Computer-assisted technology offers the potential to provide and track objective performance metrics. In this study, we aimed to validate the use of a novel metric-active control time (ACT)-for assessing trainee participation in robotic-assisted cases. Performance data from da Vinci Surgical Systems was retrospectively analyzed for all robotic cases involving trainees with a single minimally invasive surgeon over 10 months. The primary outcome metric was percent ACT-the amount of trainee console time spent in active system manipulations over total active time from both consoles. Kruskal-Wallis and Mann-Whitney U statistical tests were applied in analyses. A total of 123 robotic cases with 18 general surgery residents and 1 fellow were included. Of these, 56 were categorized as complex. Median %ACT was statistically different between trainee levels for all case types taken in aggregate (PGY1s 3.0% [IQR 2-14%], PGY3s 32% [IQR 27-66%], PGY4s 42% [IQR 26-52%], PGY5s 50% [IQR 28-70%], and fellow 61% [IQR 41-85%], p = < 0.0001). When stratified by complexity, median %ACT was higher in standard versus complex cases for PGY5 (60% vs. 36%, p = 0.0002) and fellow groups (74% vs. 47%, p = 0.0045). In this study, we demonstrated an increase in %ACT with trainee level and with standard versus complex robotic cases. These findings are consistent with hypotheses, providing validity evidence for ACT as an objective measurement of trainee participation in robotic-assisted cases. Future studies will aim to define task-specific ACT to guide further robotic training and performance assessments.
Collapse
Affiliation(s)
- Julie M Clanahan
- Department of Surgery, Section of Minimally Invasive Surgery, Washington University School of Medicine, 660 South Euclid Avenue, Mailstop 8109-22-9905, Campus Box 8109, St. Louis, MO, 63110-1093, USA.
| | - Andrew Yee
- Data and Analytics, Intuitive Surgical, Inc., Peachtree Corners, GA, 30092, USA
| | - Michael M Awad
- Department of Surgery, Section of Minimally Invasive Surgery, Washington University School of Medicine, 660 South Euclid Avenue, Mailstop 8109-22-9905, Campus Box 8109, St. Louis, MO, 63110-1093, USA
| |
Collapse
|
19
|
Bladin O, Young N, Nordquist J, Roy J, Järnbert-Pettersson H, Sandblom G, Löfgren J. Learning curve in open groin hernia surgery: nationwide register-based study. BJS Open 2023; 7:zrad108. [PMID: 37882629 PMCID: PMC10601449 DOI: 10.1093/bjsopen/zrad108] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2023] [Revised: 08/07/2023] [Accepted: 08/22/2023] [Indexed: 10/27/2023] Open
Abstract
BACKGROUND Current recommendations regarding the number of open groin hernia repairs that surgical trainees are required to perform during their residency are arbitrarily defined and vary between different curricula. This register-based study sought to investigate the learning curve of surgeons performing open anterior mesh repair for groin hernia by assessing hernia recurrence rates, surgical complications and operating times in relation to the number of procedures performed. METHOD Nationwide data on open anterior mesh repair for groin hernia performed by surgical residents were collected from the Swedish Hernia Register between 2005 and 2020. The data were analysed in a cohort undergoing procedures carried out by surgeons performing their first registered repair as resident general surgeons. Repairs by surgeons with fewer than 30 repairs were excluded. RESULTS A total of 38 845 repairs carried out by 663 surgeons were included. Operation time decreased with increasing number of performed procedures, mean (s.d.) operation time was 79 (26) min for the first 15 procedures and 60 (23) min after 241 procedures (P <0.001). A turning point where complication rates began to decrease was seen after 60 procedures. Complication rates were 3.6 per cent (396 of 10 978) for procedures 31-60 and 2.7 per cent (157 of 5 798) for procedures 61-120 (P = 0.002). There was no significant relationship between the number of procedures performed and the rate of operation on for recurrence (P = 0.894). CONCLUSION Sixty performed procedures during surgical residency is a reasonable target for achieving competency to perform open anterior mesh repair for groin hernia safely without supervision.
Collapse
Affiliation(s)
- Olof Bladin
- Department of Molecular Medicine and Surgery, Karolinska Institute, Stockholm, Sweden
| | - Nathalie Young
- Acute and Trauma Surgery, Karolinska University Hospital, Stockholm, Sweden
- Department of Clinical Science and Education, Södersjukhuset Karolinska Institute, Stockholm, Sweden
| | - Jonas Nordquist
- Department of Medicine (Huddinge), Karolinska Institute, Stockholm, Sweden
| | - Joy Roy
- Department of Molecular Medicine and Surgery, Karolinska Institute, Stockholm, Sweden
| | - Hans Järnbert-Pettersson
- Department of Clinical Science and Education, Södersjukhuset Karolinska Institute, Stockholm, Sweden
| | - Gabriel Sandblom
- Department of Clinical Science and Education, Södersjukhuset Karolinska Institute, Stockholm, Sweden
- Department of Surgery, Södersjukhuset, Stockholm, Sweden
| | - Jenny Löfgren
- Department of Molecular Medicine and Surgery, Karolinska Institute, Stockholm, Sweden
| |
Collapse
|
20
|
Payen C, Gallet P, Lechien JR, Favier V. Teachers should apply the principle of reduction for more sustainable surgical simulation practice: the example of training pharyngolaryngeal surgery in a porcine model. Front Med (Lausanne) 2023; 10:1226475. [PMID: 37711745 PMCID: PMC10499042 DOI: 10.3389/fmed.2023.1226475] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/21/2023] [Accepted: 08/11/2023] [Indexed: 09/16/2023] Open
Affiliation(s)
- Caroline Payen
- Faculty of Medicine, Montpellier University, Montpellier, France
| | - Patrice Gallet
- Otolaryngology–Head and Neck Surgery Department, Nancy Regional University Hospital, Lorraine University, Nancy, France
- Nancy-Lorraine School of Surgery, Virtual Hospital of Lorraine (HVL), Lorraine University, Nancy, France
- NGERE Team, INSERM U1256, Lorraine University, Nancy, France
| | - Jérôme R. Lechien
- Otolaryngology–Head and Neck Surgery Department, EpiCURA Hospital, Mons University, Mons, Belgium
- Otolaryngology–Head and Neck Surgery Department, Foch Hospital, Suresnes, France
| | - Valentin Favier
- Faculty of Medicine, Montpellier University, Montpellier, France
- Research-Team ICAR, Laboratory of Computer Science, Robotics and Microelectronics of Montpellier (LIRMM), Univ. Montpellier, French National Centre for Scientific Research (CNRS), Montpellier, France
- YO-IFOS Group for Sustainable Development, Young Members of International Federation of Otolaryngology Societies, Paris, France
| |
Collapse
|
21
|
Nelson K, McQuillan S, Gingerich A, Regehr G. Residents as supervisors: How senior residents make ad hoc entrustment decisions. MEDICAL EDUCATION 2023; 57:723-731. [PMID: 36606657 DOI: 10.1111/medu.15017] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/25/2022] [Revised: 12/29/2022] [Accepted: 01/04/2023] [Indexed: 06/17/2023]
Abstract
BACKGROUND As residency programmes transition to competency-based medical education, there has been substantial inquiry into understanding how ad hoc entrustment decisions are made by attending supervisors in the clinical context. However, although attendings are ultimately responsible for the decisions and actions of resident trainees, senior residents are often the ones directly supervising junior residents enrolled in competency-based training programmes. This clinical dynamic has been largely overlooked in the ad hoc entrustment literature. The purpose of this study was to explore the considerations senior residents entertain when making ad hoc entrustment decisions for their junior resident colleagues. METHODS In semi-structured interviews, 11 senior resident supervisors (third, fourth and fifth year) in obstetrics and gynaecology described how they entrust junior residents with clinical activities in the moment. Following constructivist grounded theory methodology, data were iteratively collected and coded with constant comparison until theoretical sufficiency was determined. RESULTS Senior residents described many similar considerations as attendings regarding ad hoc entrustment of junior residents, including patient safety, desire to optimise the learning environment, junior resident qualities (such as discernment and communication skills), learner handover from colleagues, and situational factors. Uniquely, senior residents discussed how their role as a middle manager and their desire to protect the junior resident (from burnout, becoming a second victim and from attendings) impacts their decisions. CONCLUSIONS Although senior residents make ad hoc entrustment decisions with some similar considerations to attendings, they also seem to think about additional factors. It may be that these different considerations need to be accommodated in documentation of ad hoc entrustment decisions if these documents are to be used for high-stakes summative entrustment decisions made by competency committees.
Collapse
Affiliation(s)
- Kayla Nelson
- Obstetrics and Gynecology, Pediatric and Adolescent Gynecology, University of Calgary, Calgary, Alberta, Canada
| | - Sarah McQuillan
- Obstetrics and Gynecology, Pediatric and Adolescent Gynecology, University of Calgary, Calgary, Alberta, Canada
| | - Andrea Gingerich
- Northern Medical Program, University of Northern British Columbia, Prince George, British Columbia, Canada
| | - Glenn Regehr
- Faculty of Medicine, University of British Columbia, Vancouver, British Columbia, Canada
| |
Collapse
|
22
|
Masada KM, Cristino DM, Dear KA, Hast MW, Mehta S. 3-D Printed Fracture Models Improve Resident Performance and Clinical Outcomes in Operative Fracture Management. JOURNAL OF SURGICAL EDUCATION 2023; 80:1020-1027. [PMID: 37198080 DOI: 10.1016/j.jsurg.2023.04.004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/28/2022] [Revised: 12/30/2022] [Accepted: 04/09/2023] [Indexed: 05/19/2023]
Abstract
OBJECTIVE To determine if preoperative examination of patient additive manufactured (AM) fracture models can be used to improve resident operative competency and patient outcomes. DESIGN Prospective cohort study. Seventeen matched pairs of fracture fixation surgeries (for a total of 34 surgeries) were performed. Residents first performed a set of baseline surgeries (n = 17) without AM fracture models. The residents then performed a second set of surgeries randomly assigned to include an AM model (n = 11) or to omit it (n = 6). Following each surgery, the attending surgeon evaluated the resident using an Ottawa Surgical Competency Operating Room Evaluation (O-Score). The authors also recorded clinical outcomes including operative time, blood loss, fluoroscopy duration, and patient reported outcome measurement information system (PROMIS) scores of pain and function at 6 months. SETTING Single-center academic level one trauma center. PARTICIPANTS Twelve orthopaedic residents, between postgraduate year (PGY) 2 and 5, participated in this study. RESULTS Residents significantly improved their O-Scores between the first and second surgery when they trained with AM models for the second surgery (p = 0.004, 2.43 ± 0.79 versus 3.73 ± 0.64). Similar improvements were not observed in the control group (p = 0.916, 2.69 ± 0.69 versus 2.77 ± 0.36). AM model training also significantly improved clinical outcomes, including surgery time (p = 0.006), fluoroscopy exposure time (p = 0.002), and patient reported functional outcomes (p = 0.0006). CONCLUSIONS Conclusions: Training with AM fracture models improves the performance of orthopaedic surgery residents during fracture surgery.
Collapse
Affiliation(s)
- Kendall M Masada
- Hospital of the University of Pennsylvania, Department of Orthopaedic Surgery, University of Pennsylvania, Philadelphia, Pennsylvania.
| | - Danielle M Cristino
- Hospital of the University of Pennsylvania, Department of Orthopaedic Surgery, University of Pennsylvania, Philadelphia, Pennsylvania
| | - Kayley A Dear
- Hospital of the University of Pennsylvania, Department of Orthopaedic Surgery, University of Pennsylvania, Philadelphia, Pennsylvania
| | - Michael W Hast
- Hospital of the University of Pennsylvania, Department of Orthopaedic Surgery, University of Pennsylvania, Philadelphia, Pennsylvania
| | - Samir Mehta
- McKay Laboratory, Department of Orthopaedic Surgery, University of Pennsylvania, Philadelphia, Pennsylvania
| |
Collapse
|
23
|
Luscan R, Malheiro E, Sisso F, Wartelle S, Parc Y, Fauroux B, Bégué T, Johanet H, Denoyelle F, Garabédian EN, Simon F. What defines a great surgeon? A survey study confronting perspectives. Front Med (Lausanne) 2023; 10:1210915. [PMID: 37457585 PMCID: PMC10338833 DOI: 10.3389/fmed.2023.1210915] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/23/2023] [Accepted: 06/05/2023] [Indexed: 07/18/2023] Open
Abstract
Background The definition of a great surgeon is usually reported by surgeons themselves. The objective of the study was to define a multifaceted definition of a great surgeon, by confronting patients', healthcare workers', and surgeons' perspectives. Study design An online open-ended questionnaire was created to identify three qualities and three shortcomings defining a great surgeon. Age, gender, and profession of respondents were collected. Responses with a similar meaning were combined into word groups and labeled within four themes: human qualities, technical surgical skills (TSS), non-technical skills (NTS), and knowledge. Multivariate analyses were conducted between themes and respondent characteristics. Results Four thousand seven hundred and sixty qualities and 4,374 shortcomings were obtained from 1,620 respondents including 385 surgeons, 291 patients, 565 operating theater (OT) health professionals, and 379 non-OT health professionals. The main three qualities were dexterity (54% of respondents), meticulousness (18%), and empathy (18%). There was no significant difference between professional categories for TSS. Compared with surgeons, non-OT health professionals and patients put more emphasis on human qualities (29 vs. 39% and 42%, respectively, p < .001). OT health professionals referred more to NTS than surgeons (35 vs. 22%, p < 0.001). Knowledge was more important for surgeons (19%) than for all other professional categories (p < 0.001). Conclusions This survey illustrates the multifaceted definition of a great surgeon. Even if dexterity is a major quality, human qualities are of paramount importance. Knowledge seems to be underestimated by non-surgeons, although it essential to understand the disease and preparing the patient and OT team for the procedure.
Collapse
Affiliation(s)
- Romain Luscan
- Université Paris Cité, Faculté de Médecine, Paris, France
- Department of Paediatric Otolaryngology, AP-HP, Hôpital Necker-Enfants Malades, Paris, France
| | - Emeline Malheiro
- Department of Otolaryngology, Centre Hospitalier Intercommunal de Créteil, Créteil, France
| | - Fiona Sisso
- Université Paris Cité, Faculté de Médecine, Paris, France
- Department of Paediatric Anaesthesiology, AP-HP, Hôpital Necker-Enfants Malades, Paris, France
| | - Sébastien Wartelle
- Department of Paediatric Otolaryngology, Clinique Marcel Sembat, Boulogne-Billancourt, France
| | - Yann Parc
- Paris Sorbonne University, Faculté de Médecine, Paris, France
- Department of Digestive Surgery, AP-HP, Hôpital Saint Antoine, Paris, France
| | - Brigitte Fauroux
- Pediatric Noninvasive Ventilation and Sleep Unit, AP-HP, Hôpital Necker Enfants-Malades, Paris, France
- Université Paris Cité, EA 7330 VIFASOM (Vigilance Fatigue Sommeil et Santé Publique), Paris, France
| | - Thierry Bégué
- Paris-Saclay University, Faculté de Médecine Paris-Saclay, Le Kremlin-Bicêtre, France
- Department of Orthopaedic Surgery and Traumatology, AP-HP, Hôpital Antoine Béclère, Clamart, France
| | - Hubert Johanet
- Department of General Surgery, Clinique Turin, Paris, France
- Académie Nationale de Chirurgie, Paris, France
| | - Françoise Denoyelle
- Université Paris Cité, Faculté de Médecine, Paris, France
- Department of Paediatric Otolaryngology, AP-HP, Hôpital Necker-Enfants Malades, Paris, France
| | - Erea-Noël Garabédian
- Université Paris Cité, Faculté de Médecine, Paris, France
- Department of Paediatric Otolaryngology, AP-HP, Hôpital Necker-Enfants Malades, Paris, France
| | - François Simon
- Université Paris Cité, Faculté de Médecine, Paris, France
- Department of Paediatric Otolaryngology, AP-HP, Hôpital Necker-Enfants Malades, Paris, France
| |
Collapse
|
24
|
DeBiasio C, Pageau P, Shefrin A, Woo MY, Cheung WJ. Point-of-Care-ultrasound in undergraduate medical education: a scoping review of assessment methods. Ultrasound J 2023; 15:30. [PMID: 37302105 DOI: 10.1186/s13089-023-00325-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2023] [Accepted: 05/08/2023] [Indexed: 06/13/2023] Open
Abstract
BACKGROUND Point-of-Care-Ultrasound (POCUS) curricula have rapidly expanded in undergraduate medical education (UME). However, the assessments used in UME remain variable without national standards. This scoping review characterizes and categorizes current assessment methods using Miller's pyramid for skills, performance, and competence of POCUS in UME. A structured protocol was developed using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR). A literature search of MEDLINE was performed from January 1, 2010, to June 15, 2021. Two independent reviewers screened all titles and abstracts for articles that met inclusion criteria. The authors included all POCUS UME publications in which POCUS-related knowledge, skills, or competence were taught and objectively assessed. Articles were excluded if there were no assessment methods used, if they exclusively used self-assessment of learned skills, were duplicate articles, or were summaries of other literature. Full text analysis and data extraction of included articles were performed by two independent reviewers. A consensus-based approach was used to categorize data and a thematic analysis was performed. RESULTS A total of 643 articles were retrieved and 157 articles met inclusion criteria for full review. Most articles (n = 132; 84%) used technical skill assessments including objective structured clinical examinations (n = 27; 17%), and/or other technical skill-based formats including image acquisition (n = 107; 68%). Retention was assessed in n = 98 (62%) studies. One or more levels of Miller's pyramid were included in 72 (46%) articles. A total of four articles (2.5%) assessed for students' integration of the skill into medical decision making and daily practice. CONCLUSIONS Our findings demonstrate a lack of clinical assessment in UME POCUS that focus on integration of skills in daily clinical practice of medical students corresponding to the highest level of Miller's Pyramid. There exists opportunities to develop and integrate assessment that evaluate higher level competencies of POCUS skills of medical students. A mixture of assessment methods that correspond to multiple levels of Miller's pyramid should be used to best assess POCUS competence in UME.
Collapse
Affiliation(s)
- Celina DeBiasio
- Division of Dermatology, Ottawa Hospital and University of Ottawa, Ottawa, ON, Canada
| | - Paul Pageau
- Department of Emergency Medicine, University of Ottawa, Ottawa Hospital, 1053 Carling Avenue Ottawa, Ottawa, ON, Canada.
| | - Allan Shefrin
- Division of Emergency Medicine, Department of Pediatrics, Children's Hospital of Eastern Ontario, University of Ottawa, Ottawa, ON, Canada
| | - Michael Y Woo
- Department of Emergency Medicine, University of Ottawa, Ottawa Hospital, 1053 Carling Avenue Ottawa, Ottawa, ON, Canada
- Ottawa Hospital Research Institute, Ottawa, ON, Canada
| | - Warren J Cheung
- Department of Emergency Medicine, University of Ottawa, Ottawa Hospital, 1053 Carling Avenue Ottawa, Ottawa, ON, Canada
- Ottawa Hospital Research Institute, Ottawa, ON, Canada
| |
Collapse
|
25
|
Seed JD, Gauthier S, Zevin B, Hall AK, Chaplin T. Simulation vs workplace-based assessment in resuscitation: a cross-specialty descriptive analysis and comparison. CANADIAN MEDICAL EDUCATION JOURNAL 2023; 14:92-98. [PMID: 37465738 PMCID: PMC10351640 DOI: 10.36834/cmej.73692] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Indexed: 07/20/2023]
Abstract
Background Simulation-based assessment can complement workplace-based assessment of rare or difficult to assess Entrustable Professional Activities (EPAs). We aimed to compare the use of simulation-based assessment for resuscitation-focused EPAs in three postgraduate medical training programs and describe faculty perceptions of simulation-based assessment. Methods EPA assessment scores and setting (simulation or workplace) were extracted from 2017-2020 for internal medicine, emergency medicine, and surgical foundations residents at the transition to discipline and foundations of discipline stages. A questionnaire was distributed to clinical competency committee members. Results Eleven percent of EPA assessments were simulation-based. The proportion of simulation-based assessment did not differ between programs but differed between transition (38%) and foundations (4%) stages within surgical foundations only. Entrustment scores differed between settings in emergency medicine at the transition level only (simulation: 4.82 ± 0.60 workplace: 3.74 ± 0.93). 70% of committee members (n=20) completed the questionnaire. Of those that use simulation-based assessment, 45% interpret them differently than workplace-based assessments. 73% and 100% trust simulation for high-stakes and low-stakes assessment, respectively. Conclusions The proportion of simulation-based assessment for resuscitation focused EPAs did not differ between three postgraduate medical training programs. Interpretation of simulation-based assessment data between committee members was inconsistent. All respondents trust simulation-based assessment for low-stakes, and the majority for high-stakes assessment. These findings have practical implications for the integration simulation into programs of assessment.
Collapse
Affiliation(s)
- Jeremy D Seed
- Department of Emergency Medicine, Queen's University, Ontario, Canada
| | | | - Boris Zevin
- Department of Surgery, Queen's University, Ontario, Canada
| | - Andrew K Hall
- Department of Emergency Medicine, University of Ottawa, Ontario, Canada
| | - Timothy Chaplin
- Department of Emergency Medicine, Queen's University, Ontario, Canada
| |
Collapse
|
26
|
Herrmann LE, Elliott LE, Sucharew H, Jerardi K, Zackoff MW, Klein M, Real FJ. Impact of a Remote Virtual Reality Curriculum Pilot on Clinician Conflict Communication Skills. Hosp Pediatr 2023; 13:527-540. [PMID: 37161716 DOI: 10.1542/hpeds.2022-006990] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/11/2023]
Abstract
OBJECTIVES Conflict management skills are essential for interprofessional team functioning, however existing trainings are time and resource intensive. We hypothesized that a curriculum incorporating virtual reality (VR) simulations would enhance providers' interprofessional conflict communication skills and increase self-efficacy. METHODS We conducted a randomized controlled pilot study of the Conflict Instruction through Virtual Immersive Cases (CIVIC) curriculum among inpatient clinicians at a pediatric satellite campus. Participants viewed a 30-minute didactic presentation on conflict management and subsequently completed CIVIC (intervention group) or an alternative VR curriculum on vaccine counseling (control group), both of which allowed for verbal interactions with screen-based avatars. Three months following VR training, all clinicians participated in a unique VR simulation focused on conflict management that was recorded and scored using a rubric of observable conflict management behaviors and a Global Entrustment Scale (GES). Differences between groups were evaluated using generalized linear models. Self-efficacy was also assessed immediately pre, post, and 3 months postcurriculum. Differences within and between groups were assessed with paired independent and 2-sample t-tests, respectively. RESULTS Forty of 51 participants (78%) completed this study. The intervention group (n = 17) demonstrated better performance on the GES (P = .003) and specific evidence-based conflict management behaviors, including summarizing team member's concerns (P = .02) and checking for acceptance of the plan (P = .02), as well as statistical improvements in 5 self-efficacy measures compared with controls. CONCLUSIONS Participants exposed to CIVIC demonstrated enhanced conflict communication skills and reported increased self-efficacy compared with controls. VR may be an effective method of conflict communication training.
Collapse
Affiliation(s)
| | | | - Heidi Sucharew
- Emergency Medicine, University of Cincinnati College of Medicine, Cincinnati, Ohio
| | - Karen Jerardi
- Departments of Pediatrics
- Divisions of Hospital Medicine
| | | | - Melissa Klein
- Departments of Pediatrics
- General and Community Pediatrics, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio
| | - Francis J Real
- Departments of Pediatrics
- General and Community Pediatrics, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio
| |
Collapse
|
27
|
Chin M, Pack R, Cristancho S. "A whole other competence story": exploring faculty perspectives on the process of workplace-based assessment of entrustable professional activities. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2023; 28:369-385. [PMID: 35997910 DOI: 10.1007/s10459-022-10156-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/24/2022] [Accepted: 08/07/2022] [Indexed: 05/11/2023]
Abstract
The centrality of entrustable professional activities (EPAs) in competency-based medical education (CBME) is predicated on the assumption that low-stakes, high-frequency workplace-based assessments used in a programmatic approach will result in accurate and defensible judgments of competence. While there have been conversations in the literature regarding the potential of this approach, only recently has the conversation begun to explore the actual experiences of clinical faculty in this process. The purpose of this qualitative study was to explore the process of EPA assessment for faculty in everyday practice. We conducted 18 semi-structured interviews with Anesthesia faculty at a Canadian academic center. Participants were asked to describe how they engage in EPA assessment in daily practice and the factors they considered. Interviews were audio-recorded, transcribed, and analysed using the constant comparative method of grounded theory. Participants in this study perceived two sources of tension in the EPA assessment process that influenced their scoring on official forms: the potential constraints of the assessment forms and the potential consequences of their assessment outcome. This was particularly salient in circumstances of uncertainty regarding the learner's level of competence. Ultimately, EPA assessment in CBME may be experienced as higher-stakes by faculty than officially recognized due to these tensions, suggesting a layer of discomfort and burden in the process that may potentially interfere with the goal of assessment for learning. Acknowledging and understanding the nature of this burden and identifying strategies to mitigate it are critical to achieving the assessment goals of CBME.
Collapse
Affiliation(s)
- Melissa Chin
- Department of Anesthesia and Perioperative Medicine, London Health Sciences Centre, Schulich School of Medicine and Dentistry, University of Western Ontario, London, ON, Canada.
| | - Rachael Pack
- Center for Education Research and Innovation, University of Western Ontario, London, ON, Canada
| | - Sayra Cristancho
- Center for Education Research and Innovation, University of Western Ontario, London, ON, Canada
| |
Collapse
|
28
|
Jarrett JB, Elmes AT, Schwartz A. Which Entrustment-Supervision Scale is Right for Pharmacy Education? AMERICAN JOURNAL OF PHARMACEUTICAL EDUCATION 2023; 87:100021. [PMID: 37288689 DOI: 10.1016/j.ajpe.2022.12.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/13/2022] [Revised: 11/16/2022] [Accepted: 12/23/2022] [Indexed: 06/09/2023]
Abstract
Entrustable professional activities (EPAs) are an outgrowth of the competency-based educational model to support workplace-based learner assessments and evaluation. A learner's performance of EPAs is assessed by the degree of provided entrustment and required supervision rather than by a score, percentage, or letter grade typically assigned in traditional academic coursework. Entrustment-supervision (ES) scales are used to document learner progression and steer learner development over time. The purpose of this article is to critique various ES tools in health professions education for utilization within an EPA framework for learner assessment in workplace-based settings and to determine which will best suit pharmacy education. Exploring the advantages and disadvantages across all types of ES scales is a critical step in determining the most useful ES tool for use within a specific pharmacy institution and across the Academy. An ES scale with the traditional 5 levels, a prospective assessment frame, and increased stratification at lower levels should be recommended by the Academy and utilized in workplace-based settings for formative and summative assessment to provide more valid assessment of learners, support the ideal of life-long learning, and give more meaning for pharmacy faculty and learners within assessment.
Collapse
Affiliation(s)
- Jennie B Jarrett
- Department of Pharmacy Practice, University of Illinois Chicago College of Pharmacy, Chicago, IL, USA.
| | - Abigail T Elmes
- Department of Pharmacy Practice, University of Illinois Chicago College of Pharmacy, Chicago, IL, USA
| | - Alan Schwartz
- Department of Medical Education, University of Illinois Chicago College of Medicine, Chicago, IL, USA
| |
Collapse
|
29
|
Huang JS, Walsh CM. Electronic Medical Records: Use as Tools for Improving Quality in Pediatric Endoscopy. Gastrointest Endosc Clin N Am 2023; 33:267-290. [PMID: 36948746 DOI: 10.1016/j.giec.2022.11.004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/24/2023]
Abstract
Quality indicators and standards for pediatric endoscopy have recently been developed by the inaugural working group of the international Pediatric Endoscopy Quality Improvement Network (PEnQuIN). Currently available electronic medical record (EMR) functionalities can enable real-time capture of quality indicators to support continuous quality measurement and improvement within pediatric endoscopy facilities. Ultimately, EMR interoperability and cross-institutional data sharing can serve to validate PEnQuIN standards of care and permit benchmarking across endoscopy services, in the pursuit of elevating the quality of endoscopic care for children everywhere.
Collapse
Affiliation(s)
- Jeannie S Huang
- Department of Pediatrics, University of California San Diego, 9500 Gilman Drive, MC 0984, La Jolla, CA 92093, USA; Division of Pediatric Gastroenterology, Rady Children's Hospital San Diego, 3020 Children's Way, San Diego, CA, USA.
| | - Catharine M Walsh
- Division of Gastroenterology, Hepatology, and Nutrition, and the SickKids Research and Learning Institutes, The Hospital for Sick Children, Toronto, ON, Canada; Department of Paediatrics and The Wilson Centre, University of Toronto, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
30
|
Imm MR, Agarwal G, Zhang C, Deshpande AR, Issenberg B, Chandran L. EPMO: A novel medical student assessment tool that integrates entrustable professional activities, prime, and the modified Ottawa coactivity scale. MEDICAL TEACHER 2023; 45:419-425. [PMID: 36288734 DOI: 10.1080/0142159x.2022.2137012] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
PURPOSE Alignment of workplace-based assessments (WPBA) with core entrustable professional activities (EPAs) for entering residency may provide opportunities to monitor student progress across the continuum of undergraduate medical education. Core EPAs, however, reflect tasks of varying degrees of difficulty and faculty assessors are not accustomed to rating students based on entrustability. Expectations of student progress should vary depending on the complexity of the tasks associated with the EPAs. An assessment tool that orients evaluators to the developmental progression of specific EPA tasks will be critical to fairly evaluate learners. METHODS The authors developed an EPA assessment tool combining the frameworks of Professionalism, Reporter, Interpreter, Manager, Educator (PRIME), and Modified Ottawa coactivity scales. Only those EPAs that could be repeatedly observed and assessed across clinical clerkships were included. From July 2019 to March 2020, third-year medical students across multiple clerkships were assessed using this tool. The authors hypothesized that if the tool was applied correctly, ratings of learner independence would be lower with higher complexity tasks and that such ratings would increase over the course of year with ongoing clinical learning. RESULTS Assessment data for 247 medical students were similar across clerkships suggesting that evaluators in diverse clinical contexts were able to use this tool to assign scores reflective of developing entrustability in the workplace. Faculty rated student entrustability highest in skills emphasized in the pre-clerkship curriculum (professionalism and reporter) and progressively lower in more advanced skills (interpreter and manager). Students' ratings increased over time with more clinical exposure. CONCLUSIONS The authors developed a composite WBPA tool that combines the frameworks of EPAs, PRIME, and Modified Ottawa Co- Activity and demonstrated the usability of applying it for learner assessments in clinical settings. Further multicenter studies with cohorts of pre- and post-clerkship students may provide additional validity evidence for the tool.
Collapse
Affiliation(s)
- Matthew R Imm
- Department of Medicine, The University of Miami Leonard M. Miller School of Medicine, Miami, FL, USA
| | - Gauri Agarwal
- Department of Medicine, The University of Miami Leonard M. Miller School of Medicine, Miami, FL, USA
| | - Chi Zhang
- Department of Medical Education, The University of Miami Leonard M. Miller School of Medicine, Miami, FL, USA
| | - Amar R Deshpande
- Department of Medicine, The University of Miami Leonard M. Miller School of Medicine, Miami, FL, USA
| | - Barry Issenberg
- Department of Medical Education, The University of Miami Leonard M. Miller School of Medicine, Miami, FL, USA
| | - Latha Chandran
- Department of Medical Education, The University of Miami Leonard M. Miller School of Medicine, Miami, FL, USA
| |
Collapse
|
31
|
Neurosurgeons' Perspectives on Vascular Entrustable Professional Activities. Can J Neurol Sci 2023; 50:287-289. [PMID: 35190007 DOI: 10.1017/cjn.2021.528] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
Abstract
This article discusses subspecialty Canadian neurosurgeons' perceptions of entrustable professional activities (EPAs) assessments and variabilities prior to the implementation of the Competence by Design (CBD) system in Canada. Vascular neurosurgeons were asked to reflect on how they would evaluate and give feedback to neurosurgery residents concerning the EPA "Performing surgery for patients with an intracranial aneurysm." Interviews were transcribed and analyzed using a deductive approach. Themes were derived from these interviews and reflected on the subjectivity and biases present in the EPA assessment forms. Indeed, faculty may require more training in the transitioning to a CBD evaluation system.
Collapse
|
32
|
Balvardi S, Semsar-Kazerooni K, Kaneva P, Mueller C, Vassiliou M, Al Mahroos M, Fiore JF, Schwartzman K, Feldman LS. Validity of video-based general and procedure-specific self-assessment tools for surgical trainees in laparoscopic cholecystectomy. Surg Endosc 2023; 37:2281-2289. [PMID: 35922607 PMCID: PMC9362520 DOI: 10.1007/s00464-022-09466-6] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/11/2022] [Accepted: 07/10/2022] [Indexed: 11/26/2022]
Abstract
INTRODUCTION Self-review of recorded surgical procedures offers new opportunities for trainees to extend technical learning outside the operating-room. Valid tools for self-assessment are required prior to evaluating the effectiveness of video-review in enhancing technical learning. Therefore, we aimed to contribute evidence regarding the validity of intraoperative performance assessment tools for video-based self-assessment by general surgery trainees when performing laparoscopic cholecystectomies. METHODS AND PROCEDURES Using a web-based platform, general surgery trainees in a university-based residency program submitted recorded laparoscopic cholecystectomy procedures where they acted as the supervised primary surgeon. Attending surgeons measured operative performance at the time of surgery using general and procedure-specific assessment tools [Global Operative Assessment of Laparoscopic Skills (GOALS) and Operative Performance Rating System (OPRS), respectively] and entrustability level (O-SCORE). Trainees self-evaluated their performance from video-review using the same instruments. The validity of GOALS and OPRS for trainee self-assessment was investigated by testing the hypotheses that self-assessment scores correlate with (H1) expert assessment scores, (H2) O-SCORE, and (H3) procedure time and that (H4) self-assessment based on these instruments differentiates junior [postgraduate year (PGY) 1-3] and senior trainees (PGY 4-5), as well as (H5)simple [Visual Analogue Scale (VAS) ≤ 4] versus complex cases (VAS > 4). All hypotheses were based on previous literature, defined a priori, and were tested according to the COSMIN consensus on measurement properties. RESULTS A total of 35 videos were submitted (45% female and 45% senior trainees) and self-assessed. Our data supported 2 out of 5 hypotheses (H1 and H4) for GOALS and 3 out of 5 hypotheses (H1, H4 and H5) for OPRS, for trainee self-assessment. CONCLUSIONS OPRS, a procedure-specific assessment tool, was better able to differentiate between groups expected to have different levels of intraoperative performance, compared to GOALS, a general assessment tool. Given the interest in video-based learning, there is a need to further develop valid procedure-specific tools to support video-based self-assessment by trainees in a range of procedures.
Collapse
Affiliation(s)
- Saba Balvardi
- Department of Surgery, McGill University, Montreal, QC, Canada.,Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, QC, Canada
| | - Koorosh Semsar-Kazerooni
- Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, QC, Canada
| | - Pepa Kaneva
- Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, QC, Canada
| | - Carmen Mueller
- Department of Surgery, McGill University, Montreal, QC, Canada.,Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, QC, Canada
| | - Melina Vassiliou
- Department of Surgery, McGill University, Montreal, QC, Canada.,Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, QC, Canada
| | | | - Julio F Fiore
- Department of Surgery, McGill University, Montreal, QC, Canada.,Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, QC, Canada
| | - Kevin Schwartzman
- Respiratory Division, Department of Medicine, McGill University and McGill International Tuberculosis Centre, Research Institute of the McGill University Health Centre, Montreal, QC, Canada
| | - Liane S Feldman
- Department of Surgery, McGill University, Montreal, QC, Canada. .,Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, QC, Canada. .,Department of Surgery, McGill University Health Centre, 1650 Cedar Ave, D6-156, Montreal, QC, H3G 1A4, Canada.
| |
Collapse
|
33
|
Al-Hubaishi O, Hillier T, Gillis M, Oxner W, Trenholm A, Richardson G, Leighton R, Glennie A. Video-based assessment (VBA) of an open, simulated orthopedic surgical procedure: a pilot study using a single-angle camera to assess surgical skill and decision making. J Orthop Surg Res 2023; 18:90. [PMID: 36750893 PMCID: PMC9904250 DOI: 10.1186/s13018-023-03557-3] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/11/2022] [Accepted: 01/21/2023] [Indexed: 02/09/2023] Open
Abstract
BACKGROUND Videos have been used in many settings including medical simulation. Limited information currently exists on video-based assessment in surgical training. Effective assessment tools have substantial impact on the future of training. The objectives of this study were as follows: to evaluate the inter-rater reliability of video-based assessment of orthopedic surgery residents performing open cadaveric simulation procedures and to explore the benefits and limitations of video-based assessment. METHODS A multi-method technique was used. In the quantitative portion, four residents participated in a Surgical Objective Structured Clinical Examination in 2017 at a quaternary care training center. A single camera bird's-eye view was used to videotape the procedures. Five orthopedic surgeons evaluated the surgical videos using the Ottawa Surgical Competency Operating Room Evaluation. Interclass correlation coefficient was used to calculate inter-rater reliability. In the qualitative section, semi-structured interviews were used to explore the perceived strengths and limitations of video-based assessment. RESULTS AND DISCUSSION The scores using video-based assessment demonstrated good inter-rater reliability (ICC = 0.832, p = 0.014) in assessing open orthopedic procedures on cadavers. Qualitatively, the strengths of video-based assessment in this study are its ability to assess global performance and/or specific skills, ability to reassess missed points during live assessment, and potential use for less common procedures. It also allows for detailed constructive feedback, flexible assessment time, anonymous assessment, multiple assessors and serves as a good coaching tool. The main limitations of video-based assessment are poor audio-video quality, and questionable feasibility for assessing readiness for practice. CONCLUSION Video-based assessment is a potential adjunct to live assessment in orthopedic open procedures with good inter-rater reliability. Improving audio-video quality will enhance the quality of the assessment and improve the effectiveness of using this tool in surgical training.
Collapse
Affiliation(s)
- Obaid Al-Hubaishi
- grid.55602.340000 0004 1936 8200Department of Surgery, Dalhousie University, Halifax, NS Canada
| | - Troy Hillier
- grid.55602.340000 0004 1936 8200Faculty of Medicine, Dalhousie University, Halifax, NS Canada
| | - Megan Gillis
- grid.55602.340000 0004 1936 8200Department of Surgery, Dalhousie University, Halifax, NS Canada
| | - William Oxner
- grid.55602.340000 0004 1936 8200Department of Surgery, Dalhousie University, Halifax, NS Canada
| | - Andrew Trenholm
- grid.55602.340000 0004 1936 8200Department of Surgery, Dalhousie University, Halifax, NS Canada
| | - Glen Richardson
- grid.55602.340000 0004 1936 8200Department of Surgery, Dalhousie University, Halifax, NS Canada
| | - Ross Leighton
- grid.55602.340000 0004 1936 8200Department of Surgery, Dalhousie University, Halifax, NS Canada
| | - Andrew Glennie
- Department of Surgery, Dalhousie University, Halifax, NS, Canada.
| |
Collapse
|
34
|
A novel assessment model for teaching robot-assisted living donor nephrectomy in abdominal transplant surgery fellowship. Am J Surg 2023; 225:420-424. [PMID: 36253318 DOI: 10.1016/j.amjsurg.2022.09.058] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/08/2022] [Revised: 09/23/2022] [Accepted: 09/29/2022] [Indexed: 11/20/2022]
Abstract
BACKGROUND An increasing number of transplant centers have adopted robot-assisted living donor nephrectomy. Thus, a transplant fellow assessment tool is needed for promoting operative independence in an objective and safe manner. METHODS In this pilot study, data was prospectively collected on both fellow performance with focus on technique, efficiency, and communication ("overall RO-SCORE"), and operative steps ("operative steps RO-SCORE"). Robotic user performance metrics were analyzed from the da Vinci Xi system, including fellow percent active control time (ACT) and handoff counts. RESULTS From July 2020 to February 2021, twenty-one robot-assisted donor nephrectomies were performed. In regression analysis, fellow performance (based on both RO-SCOREs and robot % ACT) was significantly associated with both time and case number, with time-to-independence modelled at 8.4-14.2 months, and case number-to-independence estimated at 15-22 cases. Robot user metrics provided valid objective measures alongside RO-SCOREs. CONCLUSIONS This pilot study provides an effective assessment tool for promoting operative competency in robot-assisted donor nephrectomy among transplant fellows.
Collapse
|
35
|
Kogan JR, Dine CJ, Conforti LN, Holmboe ES. Can Rater Training Improve the Quality and Accuracy of Workplace-Based Assessment Narrative Comments and Entrustment Ratings? A Randomized Controlled Trial. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2023; 98:237-247. [PMID: 35857396 DOI: 10.1097/acm.0000000000004819] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
PURPOSE Prior research evaluating workplace-based assessment (WBA) rater training effectiveness has not measured improvement in narrative comment quality and accuracy, nor accuracy of prospective entrustment-supervision ratings. The purpose of this study was to determine whether rater training, using performance dimension and frame of reference training, could improve WBA narrative comment quality and accuracy. A secondary aim was to assess impact on entrustment rating accuracy. METHOD This single-blind, multi-institution, randomized controlled trial of a multifaceted, longitudinal rater training intervention consisted of in-person training followed by asynchronous online spaced learning. In 2018, investigators randomized 94 internal medicine and family medicine physicians involved with resident education. Participants assessed 10 scripted standardized resident-patient videos at baseline and follow-up. Differences in holistic assessment of narrative comment accuracy and specificity, accuracy of individual scenario observations, and entrustment rating accuracy were evaluated with t tests. Linear regression assessed impact of participant demographics and baseline performance. RESULTS Seventy-seven participants completed the study. At follow-up, the intervention group (n = 41), compared with the control group (n = 36), had higher scores for narrative holistic specificity (2.76 vs 2.31, P < .001, Cohen V = .25), accuracy (2.37 vs 2.06, P < .001, Cohen V = .20) and mean quantity of accurate (6.14 vs 4.33, P < .001), inaccurate (3.53 vs 2.41, P < .001), and overall observations (2.61 vs 1.92, P = .002, Cohen V = .47). In aggregate, the intervention group had more accurate entrustment ratings (58.1% vs 49.7%, P = .006, Phi = .30). Baseline performance was significantly associated with performance on final assessments. CONCLUSIONS Quality and specificity of narrative comments improved with rater training; the effect was mitigated by inappropriate stringency. Training improved accuracy of prospective entrustment-supervision ratings, but the effect was more limited. Participants with lower baseline rating skill may benefit most from training.
Collapse
Affiliation(s)
- Jennifer R Kogan
- J.R. Kogan is associate dean, Student Success and Professional Development, and professor of medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania; ORCID: https://orcid.org/0000-0001-8426-9506
| | - C Jessica Dine
- C.J. Dine is associate dean, Evaluation and Assessment, and associate professor of medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania; ORCID: https://orcid.org/0000-0001-5894-0861
| | - Lisa N Conforti
- L.N. Conforti is research associate for milestones evaluation, Accreditation Council for Graduate Medical Education, Chicago, Illinois; ORCID: https://orcid.org/0000-0002-7317-6221
| | - Eric S Holmboe
- E.S. Holmboe is chief, research, milestones development and evaluation, Accreditation Council for Graduate Medical Education, Chicago, Illinois; ORCID: https://orcid.org/0000-0003-0108-6021
| |
Collapse
|
36
|
Evaluating the effect of assessment form design on the quality of feedback in one Canadian ophthalmology residency program as an early adopter of CBME. CANADIAN JOURNAL OF OPHTHALMOLOGY 2023:S0008-4182(23)00005-4. [PMID: 36702162 DOI: 10.1016/j.jcjo.2023.01.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/09/2022] [Revised: 12/19/2022] [Accepted: 01/06/2023] [Indexed: 01/25/2023]
|
37
|
Kurashima Y, Kitagami H, Teramura K, Poudel S, Ebihara Y, Inaki N, Nakamura F, Misawa K, Shibao K, Nagai E, Akashi Y, Takiguchi S, Kobayashi S, Hirano S. Validation study of a skill assessment tool for education and outcome prediction of laparoscopic distal gastrectomy. Surg Endosc 2022; 36:8807-8816. [PMID: 35578050 DOI: 10.1007/s00464-022-09305-8] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2022] [Accepted: 04/23/2022] [Indexed: 01/06/2023]
Abstract
BACKGROUND The Japanese operative-rating scale for laparoscopic distal gastrectomy (JORS-LDG) was developed through cognitive task analysis together with the Delphi method to measure intraoperative performance during laparoscopic distal gastrectomy. This study aimed to investigate the value of this rating scale as an educational tool and a surgical outcome predictor in laparoscopic distal gastrectomy. METHODS The surgical performance of laparoscopic distal gastrectomy was assessed by the first assistant, through self-evaluation in the operating room and by video raters blind to the case. We evaluated inter-rater reliability, internal consistency, and correlations between the JORS-LDG scores and the evaluation methods, patient characteristics, and surgical outcomes. RESULTS Fifty-four laparoscopic distal gastrectomy procedures performed by 40 surgeons at 16 institutions were evaluated in the operating room and with video recordings using the proposed rating scale. The video inter-rater reliability was > 0.8. Participating surgeons were divided into the low, intermediate, and high groups based on their total scores. The number of laparoscopic surgeries and laparoscopic gastrectomy procedures performed differed significantly among the groups according to laparoscopic distal gastrectomy skill levels. The low, intermediate, and high groups also differed in terms of median operating times (311, 266, and 229 min, respectively, P < 0.001), intraoperative complication rates (27.8, 11.8, and 0%, respectively, P = 0.01), and postoperative complication rates (22.2, 0, and 0%, respectively, P = 0.002). CONCLUSIONS The JORS-LDG is a reliable and valid measure for laparoscopic distal gastrectomy training and could be useful in predicting surgical outcomes.
Collapse
Affiliation(s)
- Yo Kurashima
- Department of Gastroenterological Surgery II, Hokkaido University School of Medicine, Kita-15, Nishi-7, Kita-ku, Sapporo, Hokkaido, 060-8638, Japan.
- Clinical Simulation Center, Hokkaido University School of Medicine, Sapporo, Japan.
| | - Hidehiko Kitagami
- Department of Gastroenterological Surgery, Keiyukai Sapporo Hospital, Sapporo, Japan
| | - Koichi Teramura
- Department of Colorectal Surgery, National Cancer Center Hospital East, Kashiwa, Japan
| | - Saseem Poudel
- Department of Gastroenterological Surgery II, Hokkaido University School of Medicine, Kita-15, Nishi-7, Kita-ku, Sapporo, Hokkaido, 060-8638, Japan
| | - Yuma Ebihara
- Department of Gastroenterological Surgery II, Hokkaido University School of Medicine, Kita-15, Nishi-7, Kita-ku, Sapporo, Hokkaido, 060-8638, Japan
| | - Noriyuki Inaki
- Department of Gastrointestinal Surgery, Kanazawa University Graduate School of Medical Sciences, Kanazawa, Japan
| | - Fumitaka Nakamura
- Department of Gastroenterological Surgery, Teine Keijinkai Hospital, Sapporo, Japan
| | - Kazunari Misawa
- Department of Gastroenterological Surgery, Aichi Cancer Center Hospital, Nagoya, Japan
| | - Kazunori Shibao
- Department of Surgey1, School of Medicine, University of Occupational and Environmental Health, Kitakyushu, Japan
| | - Eishi Nagai
- Department of Gastrointestinal Surgery, Japanese Red Cross Fukuoka Hospital, Fukuoka, Japan
| | - Yoshimasa Akashi
- Department of Gastrointestinal and Hepato-Billiary-Pancreatic Surgery, Faculty of Medicine, University of Tsukuba, Tsukuba, Japan
| | - Shuji Takiguchi
- Department of Gastroenterological Surgery, Nagoya City University Graduate School of Medical Sciences, Nagoya, Japan
| | - Shinichiro Kobayashi
- Department of Surgery, Nagasaki University Graduate School of Biomedical Sciences, Nagasaki, Japan
| | - Satoshi Hirano
- Department of Gastroenterological Surgery II, Hokkaido University School of Medicine, Kita-15, Nishi-7, Kita-ku, Sapporo, Hokkaido, 060-8638, Japan
| |
Collapse
|
38
|
Kitaguchi D, Fujino T, Takeshita N, Hasegawa H, Mori K, Ito M. Limited generalizability of single deep neural network for surgical instrument segmentation in different surgical environments. Sci Rep 2022; 12:12575. [PMID: 35869249 PMCID: PMC9307578 DOI: 10.1038/s41598-022-16923-8] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/09/2021] [Accepted: 07/18/2022] [Indexed: 12/05/2022] Open
Abstract
Clarifying the generalizability of deep-learning-based surgical-instrument segmentation networks in diverse surgical environments is important in recognizing the challenges of overfitting in surgical-device development. This study comprehensively evaluated deep neural network generalizability for surgical instrument segmentation using 5238 images randomly extracted from 128 intraoperative videos. The video dataset contained 112 laparoscopic colorectal resection, 5 laparoscopic distal gastrectomy, 5 laparoscopic cholecystectomy, and 6 laparoscopic partial hepatectomy cases. Deep-learning-based surgical-instrument segmentation was performed for test sets with (1) the same conditions as the training set; (2) the same recognition target surgical instrument and surgery type but different laparoscopic recording systems; (3) the same laparoscopic recording system and surgery type but slightly different recognition target laparoscopic surgical forceps; (4) the same laparoscopic recording system and recognition target surgical instrument but different surgery types. The mean average precision and mean intersection over union for test sets 1, 2, 3, and 4 were 0.941 and 0.887, 0.866 and 0.671, 0.772 and 0.676, and 0.588 and 0.395, respectively. Therefore, the recognition accuracy decreased even under slightly different conditions. The results of this study reveal the limited generalizability of deep neural networks in the field of surgical artificial intelligence and caution against deep-learning-based biased datasets and models. Trial Registration Number: 2020-315, date of registration: October 5, 2020.
Collapse
|
39
|
Eltayar AN, Aref SR, Khalifa HM, Hammad AS. Do entrustment scales make a difference in the inter-rater reliability of the workplace-based assessment? MEDICAL EDUCATION ONLINE 2022; 27:2053401. [PMID: 35311494 PMCID: PMC8942514 DOI: 10.1080/10872981.2022.2053401] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/15/2021] [Revised: 03/08/2022] [Accepted: 03/11/2022] [Indexed: 06/14/2023]
Abstract
BACKGROUND A workplace-based assessment (WBA) is used to assess learners' competencies in their workplaces. Many workplace assessment tools are available and validated to assess various constructs. The implementation of workplace-based assessment requires proper training of the staff. OBJECTIVE This study aimed to explore the impact of staff training on WBA practices and evaluate the inter-rater reliability of these practices while using entrustment scales, performance descriptors, and personal judgment. DESIGN A quasi-experimental study, in which the staff members of the orthopedic department were invited to participate in a training program on the use of entrustment scales and assessment descriptors within the WBA tools. As a response to the training, subjective judgment was replaced by entrustment scales and performance descriptors in a trauma course offered by the orthopedic department. The inter-rater reliability of the WBA was evaluated using various rating scales. RESULTS The entrustment scales had higher inter-rater reliability of the assessment tools than performance descriptors and the personal judgment. CONCLUSION The inter-rater reliability was highest when using entrustment scales for WBAs, which could indicate that the entrustment scales achieve good psychometric properties as regards consistency among different raters. Thus, they decrease the confounding effect of differences in assessors. They may also give a clearer image of the actual academic level of the learners.
Collapse
Affiliation(s)
- Ayat Nabil Eltayar
- Medical Education Department, Faculty of Medicine, Alexandria University, Egypt
| | - Soha Rashed Aref
- Community Medicine and Public Health Department, Faculty of Medicine, Alexandria University, Egypt
| | - Hoda Mahmoud Khalifa
- Histology and Cell biology Department, Faculty of Medicine, Alexandria University, Egypt
| | - Abdullah Said Hammad
- Orthopaedic and Traumatology Department, Faculty of Medicine, Alexandria University, Egypt
| |
Collapse
|
40
|
Schumacher DJ, Michelson C, Winn AS, Turner DA, Elshoff E, Kinnear B. Making prospective entrustment decisions: Knowing limits, seeking help and defaulting. MEDICAL EDUCATION 2022; 56:892-900. [PMID: 35263474 DOI: 10.1111/medu.14797] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/11/2021] [Revised: 02/28/2022] [Accepted: 03/02/2022] [Indexed: 06/14/2023]
Abstract
INTRODUCTION Many studies focused on entrustment have not considered prospective entrustment decisions, where clinical competency and entrustment committees determine how much supervision trainees will require in the future for tasks occurring in not completely known contexts. The authors sought to explore factors that influence and determine prospective entrustment decisions made by members of such committees in graduate medical education (GME) and undergraduate medical education (UME). METHODS The authors conducted a constructivist grounded theory study with 23 faculty participants from GME and UME clinical competency and entrustment committees in the United States between October 2020 and March 2021. Interviews sought to explore factors and considerations participants weigh in making prospective entrustment decisions about trainees. Data collection and analysis occurred in an iterative fashion, ensuring constant comparison. Theoretical sampling was used to confirm, disconfirm and elaborate on the evolving results. RESULTS Trainees' ability to know limits and seek help is the foundation of participants' prospective entrustment decision making. Most participants, however, describe a presumption of trainee readiness to progress and describe commonly making default prospective entrustment decisions unless 'red flags' in performance are present. Although participants desire sufficient and trusted data about trainee performance to inform decisions, they often lack it. Finally, the perceived permanence and consequences of prospective entrustment decisions influence how participants weigh other factors contributing to prospective entrustment decisions. CONCLUSION Trainees' ability to know limits and seek help appears to be the foundation of prospective entrustment decision making. Training programmes should strive to collect and employ robust data supporting and questioning the presence of these attributes.
Collapse
Affiliation(s)
- Daniel J Schumacher
- Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| | - Catherine Michelson
- Boston Medical Center, Boston University School of Medicine, Boston, Massachusetts, USA
| | - Ariel S Winn
- Boston Children's Hospital, Harvard Medical School, Boston, Massachusetts, USA
| | - David A Turner
- American Board of Pediatrics, Chapel Hill, North Carolina, USA
| | | | - Benjamin Kinnear
- Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| |
Collapse
|
41
|
Yilmaz Y, Carey R, Chan TM, Bandi V, Wang S, Woods RA, Mondal D, Thoma B. Developing a dashboard for program evaluation in competency-based training programs: a design-based research project. CANADIAN MEDICAL EDUCATION JOURNAL 2022; 13:14-27. [PMID: 36310899 PMCID: PMC9588183 DOI: 10.36834/cmej.73554] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
BACKGROUND Canadian specialist residency training programs are implementing a form of competency-based medical education (CBME) that requires the assessment of entrustable professional activities (EPAs). Dashboards could be used to track the completion of EPAs to support program evaluation. METHODS Using a design-based research process, we identified program evaluation needs related to CBME assessments and designed a dashboard containing elements (data, analytics, and visualizations) meeting these needs. We interviewed leaders from the emergency medicine program and postgraduate medical education office at the University of Saskatchewan. Two investigators thematically analyzed interview transcripts to identify program evaluation needs that were audited by two additional investigators. Identified needs were described using quotes, analytics, and visualizations. RESULTS Between July 1, 2019 and April 6, 2021 we conducted 17 interviews with six participants (two program leaders and four institutional leaders). Four needs emerged as themes: tracking changes in overall assessment metrics, comparing metrics to the assessment plan, evaluating rotation performance, and engagement with the assessment metrics. We addressed these needs by presenting analytics and visualizations within a dashboard. CONCLUSIONS We identified program evaluation needs related to EPA assessments and designed dashboard elements to meet them. This work will inform the development of other CBME assessment dashboards designed to support program evaluation.
Collapse
Affiliation(s)
- Yusuf Yilmaz
- Continuing Professional Development Office, and McMaster program for Education Research, Innovation, and Theory (MERIT), McMaster University, Ontario, Canada
- Department of Medical Education, Ege University, Turkey
| | - Robert Carey
- Department of Emergency Medicine, University of Saskatchewan, Saskatchewan, Canada
| | - Teresa M Chan
- Continuing Professional Development Office, and McMaster program for Education Research, Innovation, and Theory (MERIT), McMaster University, Ontario, Canada
- Division of Emergency Medicine, Department of Medicine at McMaster University
| | - Venkat Bandi
- Department of Computer Science, University of Saskatchewan, Saskatchewan, Canada
| | - Shisong Wang
- Department of Computer Science, University of Saskatchewan, Saskatchewan, Canada
| | - Robert A Woods
- Department of Emergency Medicine, University of Saskatchewan, Saskatchewan, Canada
| | - Debajyoti Mondal
- Department of Computer Science, University of Saskatchewan, Saskatchewan, Canada
| | - Brent Thoma
- Department of Emergency Medicine, University of Saskatchewan, Saskatchewan, Canada
- Royal College of Physicians and Surgeons of Canada, Ontario, Canada
| |
Collapse
|
42
|
Wilson CA, Davidson J, Chahine S, Chan EP, Stringer L, Quantz MA, Saklofske DH, Wang PZT. What is Transferred and How Much is Retained? A Simulation Study of Complex Surgical Skills. J Surg Res 2022; 280:411-420. [PMID: 36041341 DOI: 10.1016/j.jss.2022.07.040] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/14/2021] [Revised: 06/29/2022] [Accepted: 07/29/2022] [Indexed: 10/15/2022]
Abstract
INTRODUCTION Studies indicate that learning surgical skills on low-fidelity models is equally beneficial to learning on high-fidelity models in terms of skills retention and transfer. However, it is unclear how low-fidelity simulation training impacts retention and transfer in novice learners, particularly on complex surgical tasks that incorporate multiple challenging skills. This study explores the capacity of complete novices to learn and transfer complex surgical skills from a low-fidelity model to a high-fidelity simulation after a delay. METHODS Task-naïve medical and nonmedical undergraduate students (n = 62) participated in a three-phase prospective double-arm randomized (2:1) experimental study. Participants completed two skills training sessions (end-to-side anastomosis) on a low-fidelity bench model. After a 4-week delay, participants completed the task again either using the low-fidelity model or a high-fidelity model (cadaver) and were assessed using a validated checklist. RESULTS There was a significant time × fidelity group interaction (P = 0.004). Simple effects analysis indicated the high-fidelity group (Mdiff = 4.18, P < 0.001) performed significantly worse (P = 0.003) in phase 3 relative to phase 2 compared to the low-fidelity group (Mdiff = 0.75, P = 0.39). Post hoc logistic regression analysis indicated that radial suturing technique and economy of motion skills were less likely to be completed correctly for those in the high-fidelity group. CONCLUSIONS These findings suggest that for novice populations, relying on low-fidelity simulation training as a source of teaching complex skills may not provide a reliable transfer to high-fidelity models and in turn clinical settings.
Collapse
Affiliation(s)
- Claire A Wilson
- Division of Pediatric Surgery, London Health Sciences Centre, London, Ontario, Canada
| | - Jacob Davidson
- Division of Pediatric Surgery, London Health Sciences Centre, London, Ontario, Canada
| | - Saad Chahine
- Faculty of Education, Queen's University, Kingston, Ontario, Canada
| | | | - Leandra Stringer
- Department of Surgery, Western University, London, Ontario, Canada
| | | | | | | |
Collapse
|
43
|
Quero G, Mascagni P, Kolbinger FR, Fiorillo C, De Sio D, Longo F, Schena CA, Laterza V, Rosa F, Menghi R, Papa V, Tondolo V, Cina C, Distler M, Weitz J, Speidel S, Padoy N, Alfieri S. Artificial Intelligence in Colorectal Cancer Surgery: Present and Future Perspectives. Cancers (Basel) 2022; 14:cancers14153803. [PMID: 35954466 PMCID: PMC9367568 DOI: 10.3390/cancers14153803] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2022] [Revised: 07/29/2022] [Accepted: 08/03/2022] [Indexed: 02/05/2023] Open
Abstract
Artificial intelligence (AI) and computer vision (CV) are beginning to impact medicine. While evidence on the clinical value of AI-based solutions for the screening and staging of colorectal cancer (CRC) is mounting, CV and AI applications to enhance the surgical treatment of CRC are still in their early stage. This manuscript introduces key AI concepts to a surgical audience, illustrates fundamental steps to develop CV for surgical applications, and provides a comprehensive overview on the state-of-the-art of AI applications for the treatment of CRC. Notably, studies show that AI can be trained to automatically recognize surgical phases and actions with high accuracy even in complex colorectal procedures such as transanal total mesorectal excision (TaTME). In addition, AI models were trained to interpret fluorescent signals and recognize correct dissection planes during total mesorectal excision (TME), suggesting CV as a potentially valuable tool for intraoperative decision-making and guidance. Finally, AI could have a role in surgical training, providing automatic surgical skills assessment in the operating room. While promising, these proofs of concept require further development, validation in multi-institutional data, and clinical studies to confirm AI as a valuable tool to enhance CRC treatment.
Collapse
Affiliation(s)
- Giuseppe Quero
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
- Faculty of Medicine, Università Cattolica del Sacro Cuore di Roma, Largo Francesco Vito 1, 00168 Rome, Italy
| | - Pietro Mascagni
- Faculty of Medicine, Università Cattolica del Sacro Cuore di Roma, Largo Francesco Vito 1, 00168 Rome, Italy
- Institute of Image-Guided Surgery, IHU-Strasbourg, 67000 Strasbourg, France
| | - Fiona R. Kolbinger
- Department for Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, 01307 Dresden, Germany
| | - Claudio Fiorillo
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
- Correspondence: ; Tel.: +39-333-8747996
| | - Davide De Sio
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
| | - Fabio Longo
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
| | - Carlo Alberto Schena
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
- Faculty of Medicine, Università Cattolica del Sacro Cuore di Roma, Largo Francesco Vito 1, 00168 Rome, Italy
| | - Vito Laterza
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
- Faculty of Medicine, Università Cattolica del Sacro Cuore di Roma, Largo Francesco Vito 1, 00168 Rome, Italy
| | - Fausto Rosa
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
- Faculty of Medicine, Università Cattolica del Sacro Cuore di Roma, Largo Francesco Vito 1, 00168 Rome, Italy
| | - Roberta Menghi
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
- Faculty of Medicine, Università Cattolica del Sacro Cuore di Roma, Largo Francesco Vito 1, 00168 Rome, Italy
| | - Valerio Papa
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
- Faculty of Medicine, Università Cattolica del Sacro Cuore di Roma, Largo Francesco Vito 1, 00168 Rome, Italy
| | - Vincenzo Tondolo
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
| | - Caterina Cina
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
| | - Marius Distler
- Department for Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, 01307 Dresden, Germany
| | - Juergen Weitz
- Department for Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, 01307 Dresden, Germany
| | - Stefanie Speidel
- National Center for Tumor Diseases (NCT), Partner Site Dresden, 01307 Dresden, Germany
| | - Nicolas Padoy
- Institute of Image-Guided Surgery, IHU-Strasbourg, 67000 Strasbourg, France
- ICube, Centre National de la Recherche Scientifique (CNRS), University of Strasbourg, 67000 Strasbourg, France
| | - Sergio Alfieri
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
- Faculty of Medicine, Università Cattolica del Sacro Cuore di Roma, Largo Francesco Vito 1, 00168 Rome, Italy
| |
Collapse
|
44
|
The use of advanced robotic simulation labs to advance and assess senior resident robotic skills and operating room leadership competency: a pilot study. Surg Endosc 2022; 37:3053-3060. [PMID: 35922603 DOI: 10.1007/s00464-022-09474-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/24/2022] [Accepted: 07/12/2022] [Indexed: 10/16/2022]
Abstract
BACKGROUND General surgery has the fastest growing robotic operative volume in the United States, but most robotic curricula are focused on basic psychomotor skills. There are limited curricula focused on advanced robotic technical and related non-technical skills. We describe a novel pilot curriculum for robotic hiatal hernia repair developed for senior surgical residents to provide training and standardized assessment of higher-order robotic technical and leadership skills. METHODS Twelve senior residents, post-graduate year (PGY) 4 & 5, participated in a robotic hiatal hernia repair skills curriculum. Residents completed a pre- and post-survey on confidence and ability ratings on a 5-point Likert-type Scale, and a knowledge assessment. An informal faculty-led didactic was provided prior to the simulation. Residents were scored on two validated assessment tools: Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) and Global Ratings Scale of Operative Performance (GRS) by faculty proctors. RESULTS Confidence in ability to independently complete a robotic hiatal hernia case increased from mean of 2.6 ± 0.8 to 3.3 ± 0.6 (p = 0.0007). Following the simulation, residents reported increased overall confidence and ability to operate independently with mean scores of 3.3 ± 0.8 and 3.8 ± 0.9, respectively. Mean O-SCORE and GRS scores were 3.6 (range 2 - 4) and 25.4 (range 12 - 31), respectively. Number of prior live robotic cases was strongly positively correlated to O-SCORE (R = 0.84, p = 0.0006) and GRS (R = 0.88, p = 0.0002). CONCLUSION Our pilot study suggests live-operative robotic training is not sufficient alone for advanced robotic skill training. Simulations such as this can be used to (1) practice advanced robotic technical and relevant non-technical skills such as communication and operating room leadership in a low stake setting and (2) assess residents in a standardized environment to eventually evaluate robotic competency.
Collapse
|
45
|
Zelewski SK, Basson MD. Design and implementation of a core EPA-based acting internship curriculum. MEDICAL TEACHER 2022; 44:922-927. [PMID: 35358009 DOI: 10.1080/0142159x.2022.2049732] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
PURPOSE The Association of American Medical Colleges published Core Entrustable Professional Activities (Core EPAs), expected independent clinical skills for first-day interns. We sought to determine whether a required acting internship (AI) in the fourth-year curriculum could be used for summative assessment of students' mastery of Core EPAs to a predefined level that would readily generalize across disciplines and campuses. METHODS The University of North Dakota School of Medicine and Health Sciences MD Program created a standardized, required Core EPA-based AI curriculum for multiple specialties at multiple geographic sites providing a final entrustability assessment for 10 EPAs in a single course. RESULTS The course was successfully designed and launched for all students in a single class. During the AI, students functioned at the level of an acting intern, rated the courses as superior, and performed at satisfactory exit-level competence for 10 Core EPAs. CONCLUSIONS A standardized, EPA-based AI curriculum can provide an opportunity for exit level EPA assessment in the medical curriculum. This model functions well within multiple specialties and at diverse community-based, volunteer faculty teaching sites.
Collapse
Affiliation(s)
- Susan K Zelewski
- Department of Pediatrics, Assistant Dean Northeast Campus and Director of the Year 3 and 4 MD Curriculum, School of Medicine and Health Sciences, University of North Dakota, Grand Forks, ND, USA
| | - Marc D Basson
- Department of Surgery, Biomedical Sciences, and Pathology, and Senior Associate Dean for Medicine and Research, School of Medicine and the Health Sciences, University of North Dakota, Grand Forks, ND, USA
| |
Collapse
|
46
|
Spencer M, Sherbino J, Hatala R. Examining the validity argument for the Ottawa Surgical Competency Operating Room Evaluation (OSCORE): a systematic review and narrative synthesis. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2022; 27:659-689. [PMID: 35511356 DOI: 10.1007/s10459-022-10114-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/15/2021] [Accepted: 04/02/2022] [Indexed: 06/14/2023]
Abstract
The Ottawa Surgical Competency Operating Room Evaluation (OSCORE) is an assessment tool that has gained prominence in postgraduate competency-based training programs. We undertook a systematic review and narrative synthesis to articulate the underlying validity argument in support of this tool. Although originally developed to assess readiness for independent performance of a procedure, contemporary implementation includes using the OSCORE for entrustment supervision decisions. We used systematic review methodology to search, identify, appraise and abstract relevant articles from 2005 to September 2020, across MEDLINE, EMBASE and Google Scholar databases. Nineteen original, English-language, quantitative or qualitative articles addressing the use of the OSCORE for health professionals' assessment were included. We organized and synthesized the validity evidence according to Kane's framework, articulating the validity argument and identifying evidence gaps. We demonstrate a reasonable validity argument for the OSCORE in surgical specialties, based on assessing surgical competence as readiness for independent performance for a given procedure, which relates to ad hoc, retrospective, entrustment supervision decisions. The scoring, generalization and extrapolation inferences are well-supported. However, there is a notable lack of implications evidence focused on the impact of the OSCORE on summative decision-making within surgical training programs. In non-surgical specialties, the interpretation/use argument for the OSCORE has not been clearly articulated. The OSCORE has been reduced to a single-item global rating scale, and there is limited validity evidence to support its use in workplace-based assessment. Widespread adoption of the OSCORE must be informed by concurrent data collection in more diverse settings and specialties.
Collapse
Affiliation(s)
- Martha Spencer
- The University of British Columbia, Vancouver, BC, Canada.
| | | | - Rose Hatala
- The University of British Columbia, Vancouver, BC, Canada
| |
Collapse
|
47
|
Dunne D, Gielissen K, Slade M, Park YS, Green M. WBAs in UME-How Many Are Needed? A Reliability Analysis of 5 AAMC Core EPAs Implemented in the Internal Medicine Clerkship. J Gen Intern Med 2022; 37:2684-2690. [PMID: 34561828 PMCID: PMC9411433 DOI: 10.1007/s11606-021-07151-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/19/2021] [Accepted: 09/09/2021] [Indexed: 01/07/2023]
Abstract
BACKGROUND Reliable assessments of clinical skills are important for undergraduate medical education, trustworthy handoffs to graduate medical programs, and safe, effective patient care. Entrustable professional activities (EPAs) for entering residency have been developed; research is needed to assess reliability of such assessments in authentic clinical workspaces. DESIGN A student-driven mobile assessment platform was developed and used for clinical supervisors to record ad hoc entrustment decisions using the modified Ottawa scale on 5 core EPAs in an 8-week internal medicine (IM) clerkship. After a 12-month period, generalizability (G) theory analysis was performed to estimate the reliability of entrustment scores and determine the proportion of variance attributable to the student and the other facets, including particular EPA, evaluator type (attending versus resident), or case complexity. Decision (D) theory analysis determined the expected reliability based on the number of hypothetical observations. A g-coefficient of 0.7 was used as a generally agreed upon minimum reliability threshold. KEY RESULTS A total of 1368 ratings over the 5 EPAs were completed on 94 students. Variance attributed to person (true variance) was high for all EPAs; EPA-5 had the lowest person variance (9.8% across cases and four blocks). Across cases, reliability ranged from 0.02 to 0.60. Applying this to the Decision study, the estimated number of observations needed to reach a reliability index of 0.7 ranged between 9 and 11 for all EPAs except EPA5 which was sensitive to case complexity. CONCLUSIONS Work place-based clinical skills in IM clerkship students were assessed and logged using a convenient mobile platform. Our analysis suggests that 9-11 observations are needed for these EPA workplace-based assessments (WBAs) to achieve a reliability index of 0.7. Note writing was very sensitive to case complexity. Further reliability analyses of core EPAs are needed before US medical schools consider wider adoption into summative entrustment processes and GME handoffs.
Collapse
Affiliation(s)
- Dana Dunne
- Department of Internal Medicine, Section of Infectious Diseases, Yale School of Medicine, 15 York Street LMP 1074, New Haven, CT, 065111, USA.
| | - Katherine Gielissen
- Department of Internal Medicine, Section of General Medicine, Yale School of Medicine, New Haven, CT, 06511, USA
| | - Martin Slade
- Occupational Medicine, Yale School of Medicine, New Haven, CT, 06511, USA
| | | | - Michael Green
- Department of Internal Medicine, Section of General Medicine, Yale School of Medicine, New Haven, CT, 06511, USA
| |
Collapse
|
48
|
Khan RN, Siddiqui NA. The Use of Formative Assessment in Postgraduate Urology Training: A Systematic Review. Cureus 2022; 14:e27162. [PMID: 36017282 PMCID: PMC9393543 DOI: 10.7759/cureus.27162] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 07/22/2022] [Indexed: 11/05/2022] Open
Abstract
Formative assessment is an essential component of surgical training. However, it is not usually a mandatory component in postgraduate curricula. The purpose of this study is to identify and evaluate how formative assessments are integrated into postgraduate urology training in programs across the globe. This study consisted of a systemic review to see how formative assessments are being implemented in various urology programs globally. A total of 427 articles were identified for the literature review. Of these, only 10 were included and critically appraised. These studies explored various techniques for exploration of formative assessments in urology training programs, which included established tools, such as portfolio reviews, and direct observations of procedure skills (DOPS); novel tools, including the Dutch urology practical skills (D-UPS) program and Ottawa surgical competency operating room evaluation (O-SCORE); and curricular models. Nine of the 10 articles favored their potential utility in formative assessments. Current literature involving formative assessments in postgraduate urology programs is scarce, and available resources have a high heterogeneity between them. More structured formative assessments need to be incorporated into surgical training programs, and affiliated training institutions should be encouraged to integrate them into their curricula.
Collapse
|
49
|
Stockheim J, Perrakis A, Sabel BA, Waschipky R, Croner RS. RoCS: Robotic Curriculum for young Surgeons. J Robot Surg 2022; 17:495-507. [PMID: 35810233 PMCID: PMC10076401 DOI: 10.1007/s11701-022-01444-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2022] [Accepted: 06/26/2022] [Indexed: 11/26/2022]
Abstract
Robotic-assisted procedures gain increasing acceptance for daily surgical routine. However, structured training programs are designed for surgeons with high expertise. Hence, a comprehensive training curriculum was established to ensure a basic competence in robotic abdominal surgery for young surgeons during their residency. The aim of the current work is to propose a feasible and effective training concept. The development process of this training curriculum is based on a comprehensive literature review which led to the concept of "robotic curriculum for young surgeons" (RoCS). It was implemented in the daily routine of a German university hospital starting in 2020. The robotic assessment questionnaire (RAQ) was used for electronic data collection. After the initial phase adjustments, it led to an improvement of the initial version of the curriculum. RoCS is a multimodal training program containing basic training through assistance at the operation table during robotic-assisted operations and basic console training. Key elements are the robotic team time-out (rTTO), perioperative process standardization including feasible personnel scheduling and useful procedure clustering into organ systems, procedural steps and procedural step complexity. Evaluation of standardized communication, performance assessment, patient factors and individual overall workload using NASA Task Load Index is realizable. Flexibility and adaptability to internal organization processes of surgical departments are the main advantages of the concept. RoCS is a strong training tool to meet the specific needs of young surgeons and evaluate their learning success of robotic procedural training. Furthermore, comparison within the different robotic systems should be considered. Further studies are needed to validate a multicenter concept design.
Collapse
Affiliation(s)
- Jessica Stockheim
- Department of General, Visceral, Vascular and Transplant Surgery, University Hospital Magdeburg, Otto-Von-Guericke University Magdeburg, Leipziger Str. 44, 39120, Magdeburg, Germany.
| | - Aristotelis Perrakis
- Department of General, Visceral, Vascular and Transplant Surgery, University Hospital Magdeburg, Otto-Von-Guericke University Magdeburg, Leipziger Str. 44, 39120, Magdeburg, Germany
| | - Bernhard A Sabel
- Institute of Medical Psychology, Otto-Von-Guericke University Magdeburg, Magdeburg, Germany
| | - Robert Waschipky
- Department of Information Technology (IT) and Medical Engineering, Otto-Von-Guericke University Magdeburg, Magdeburg, Germany
| | - Roland S Croner
- Department of General, Visceral, Vascular and Transplant Surgery, University Hospital Magdeburg, Otto-Von-Guericke University Magdeburg, Leipziger Str. 44, 39120, Magdeburg, Germany
| |
Collapse
|
50
|
Miles VP, Schroll RW, Beaty JS. Empowering the Resident Learner. JOURNAL OF SURGICAL EDUCATION 2022; 79:909-917. [PMID: 35490137 DOI: 10.1016/j.jsurg.2022.03.009] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/06/2021] [Revised: 03/07/2022] [Accepted: 03/22/2022] [Indexed: 06/14/2023]
Abstract
Surgical resident autonomy is an essential element of the transition from supervised training to independent practice. However, in an age of duty hour restrictions, legislative constraints, increased litigation, and heightened societal expectations, training an autonomous resident proves increasingly difficult. To tackle these barriers to training successful surgeons, the American College of Surgeons Committee on Resident Education initiated the Resident Mentored Autonomy Project. As a subdivision of this project, the Empowered Learner research team here presents a framework for training the self-directed and empowered surgical resident learner. There are many strategies by which surgical faculty, program directors and chairs, and residents themselves may engage to improve resident operative autonomy.
Collapse
Affiliation(s)
- Victoria P Miles
- Department of Surgery, University of Tennessee College of Medicine Chattanooga, Chattanooga, Tennessee.
| | - Rebecca W Schroll
- Department of Surgery, Tulane University School of Medicine, New Orleans, Louisiana
| | | |
Collapse
|