1
|
Hertz P, Rattenborg S, Haug TR, Houlind K, Konge L, Bjerrum F. Training and assessment for colorectal surgery and appendicectomy- a systematic review. Colorectal Dis 2024; 26:597-608. [PMID: 38396135 DOI: 10.1111/codi.16905] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/21/2023] [Revised: 11/07/2023] [Accepted: 12/28/2023] [Indexed: 02/25/2024]
Abstract
AIM There is currently an increased focus on competency-based training, in which training and assessment play a crucial role. The aim of this systematic review is to create an overview of hands-on training methods and assessment tools for appendicectomy and colon and rectal surgery procedures using either an open, laparoscopic or robot-assisted approach. METHOD A systematic review of Medline, Embase, Cochrane and Scopus databases was conducted following the PRISMA guidelines. We conducted the last search on 9 March 2023. All published papers describing hands-on training, evaluation of performance data and development of assessment tools were eligible. The quality of studies and the validity evidence of assessment tools are reported. RESULTS Fifty-one studies were identified. Laparoscopic assessment tools are abundant, but the literature still lacks good-quality assessment tools for open appendicectomy, robotic colectomy and open rectal surgery. Overall, there is a lack of discussion regarding the establishment of pass/fail standards and the consequences of assessment. Virtual reality simulation is used more for appendicectomy than colorectal procedures. Only a few of the studies investigating training were of acceptable quality. There is a need for high-quality studies in open and robotic-assisted colon surgery and all approaches to rectal surgery. CONCLUSION This review provides an overview of current training methods and assessment tools and identifies where more research is needed based on the quality of the studies and the current validity evidence.
Collapse
Affiliation(s)
- Peter Hertz
- Department of Surgery, Hospital Lillebaelt, University of Southern Denmark, Kolding, Denmark
- Department of Regional Health Research, University of Southern Denmark, Kolding, Denmark
- Copenhagen Academy for Medical Education and Simulation (CAMES), Center for HR and Education, Copenhagen, Denmark
| | - Søren Rattenborg
- Department of Regional Health Research, University of Southern Denmark, Kolding, Denmark
- Department of Surgery, Hospital Lillebaelt Vejle, Colorectal Cancer Center South, University of Southern Denmark DK, Kolding, Denmark
| | - Tora R Haug
- Department of Surgery, Gødstrup Hospital, Herning, Denmark
- Aarhus University, Aarhus, Denmark
| | - Kim Houlind
- Department of Regional Health Research, University of Southern Denmark, Kolding, Denmark
- Department of Vascular Surgery, Hospital Lillebaelt, University of Southern Denmark, Kolding, Denmark
| | - Lars Konge
- Copenhagen Academy for Medical Education and Simulation (CAMES), Center for HR and Education, Copenhagen, Denmark
- Department of Clinical Medicine, University of Copenhagen, Copenhagen, Denmark
| | - Flemming Bjerrum
- Copenhagen Academy for Medical Education and Simulation (CAMES), Center for HR and Education, Copenhagen, Denmark
- Department of Clinical Medicine, University of Copenhagen, Copenhagen, Denmark
- Gastrounit, Surgical Section, Copenhagen University Hospital - Amager and Hvidovre, Hvidovre, Denmark
| |
Collapse
|
2
|
Nikolian VC, Camacho D, Earle D, Lehmann R, Nau P, Ramshaw B, Stulberg J. Development and preliminary validation of a new task-based objective procedure-specific assessment of inguinal hernia repair procedural safety. Surg Endosc 2024; 38:1583-1591. [PMID: 38332173 DOI: 10.1007/s00464-024-10677-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2023] [Accepted: 12/30/2023] [Indexed: 02/10/2024]
Abstract
BACKGROUND Surgical videos coupled with structured assessments enable surgical training programs to provide independent competency evaluations and align with the American Board of Surgery's entrustable professional activities initiative. Existing assessment instruments for minimally invasive inguinal hernia repair (IHR) have limitations with regards to reliability, validity, and usability. A cross-sectional study of six surgeons using a novel objective, procedure-specific, 8-item competency assessment for minimally invasive inguinal hernia repair (IHR-OPSA) was performed to assess inter-rater reliability using a "safe" vs. "unsafe" scoring rubric. METHODS The IHR-OPSA was developed by three expert IHR surgeons, field tested with five IHR surgeons, and revised based upon feedback. The final instrument included: (1) incision/port placement; (2) dissection of peritoneal flap (TAPP) or dissection of peritoneal flap (TEP); (3) exposure; (4) reducing the sac; (5) full dissection of the myopectineal orifice; (6) mesh insertion; (7) mesh fixation; and (8) operation flow. The IHR-OPSA was applied by six expert IHR surgeons to 20 IHR surgical videos selected to include a spectrum of hernia procedures (15 laparoscopic, 5 robotic), anatomy (14 indirect, 5 direct, 1 femoral), and Global Case Difficulty (easy, average, hard). Inter-rater reliability was assessed against Gwet's AC2. RESULTS The IHR-OPSA inter-rater reliability was good to excellent, ranging from 0.65 to 0.97 across the eight items. Assessments of robotic procedures had higher reliability with near perfect agreement for 7 of 8 items. In general, assessments of easier cases had higher levels of agreement than harder cases. CONCLUSIONS A novel 8-item minimally invasive IHR assessment tool was developed and tested for inter-rater reliability using a "safe" vs. "unsafe" rating system with promising results. To promote instrument validity the IHR-OPSA was designed and evaluated within the context of intended use with iterative engagement with experts and testing of constructs against real-world operative videos.
Collapse
Affiliation(s)
- Vahagn C Nikolian
- Department of Surgery, Oregon Health & Science University, 3181 S.W. Sam Jackson Park Rd., Portland, OR, 97239, USA.
| | - Diego Camacho
- Minimally Invasive and Endoscopic Surgery at Montefiore Medical Center, New York, NY, USA
| | - David Earle
- New England Hernia Center, Lowell, MA, USA
- Tufts University School of Medicine, Boston, MA, USA
| | - Ryan Lehmann
- Department of Surgery, Section of Bariatric Surgery, University of Iowa Hospitals & Clinics, Iowa City, IA, USA
| | - Peter Nau
- Department of Surgery, Section of Bariatric Surgery, University of Iowa Hospitals & Clinics, Iowa City, IA, USA
| | - Bruce Ramshaw
- CQInsights PBC, Knoxville, TN, USA
- Caresyntax Corporation, Boston, MA, USA
| | - Jonah Stulberg
- Department of Surgery, McGovern Medical School University of Texas Health Science Center at Houston, Houston, TX, USA
| |
Collapse
|
3
|
Li Y, Chan R, Menon MR, Ryan JF, Mador B, Campbell SM, Turner SR. Validity Evidence for Procedure-specific Competence Assessment Tools in Orthopaedic Surgery: A Scoping Review. J Am Acad Orthop Surg Glob Res Rev 2024; 8:01979360-202401000-00005. [PMID: 38237074 PMCID: PMC10796133 DOI: 10.5435/jaaosglobal-d-23-00065] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2023] [Revised: 11/05/2023] [Accepted: 11/07/2023] [Indexed: 01/22/2024]
Abstract
INTRODUCTION Competency-based training requires frequent assessment of residents' skills to determine clinical competence. This study reviews existing literature on procedure-specific competence assessment tools in orthopaedic surgery. METHODS A systematic search of eight databases up to May 2023 was conducted. Two reviewers independently assessed validity evidence and educational utility of each assessment tool and evaluated studies' methodological quality. RESULTS Database searching identified 2,556 unique studies for title and abstract screening. Full texts of 290 studies were reviewed; 17 studies met the inclusion criteria. Bibliography review identified another five studies, totaling 22 studies examining 24 assessment tools included in the analysis. These tools assessed various orthopaedic surgery procedures within trauma, sports medicine, spine, and upper extremity. Overall validity evidence was low across all studies, and was lowest for consequences and highest for content. Methodological quality of studies was moderate. Educational utility assessment was not explicitly done for most tools. DISCUSSION The paucity of current procedure-specific assessment tools in orthopaedic surgery lacks the validity evidence required to be used reliably in high-stake summative assessments. Study strengths include robust methodology and use of an evidence-based validity evidence framework. Poor-quality existing evidence is a limitation and highlights the need for evidence-based tools across more subspecialties.
Collapse
Affiliation(s)
- Yibo Li
- From the Department of Surgery, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, Alberta, Canada (Dr. Li, Dr. Chan, Dr. Menon, Dr. Ryan, Dr. Mador, and Dr. Turner); the Western Upper Limb Facility, Sturgeon Community Hospital, St. Albert, Alberta, Canada (Dr. Chan); and the John W. Scott Health Sciences Library, University of Alberta, Edmonton, Canada (Ms. Campbell)
| | - Robert Chan
- From the Department of Surgery, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, Alberta, Canada (Dr. Li, Dr. Chan, Dr. Menon, Dr. Ryan, Dr. Mador, and Dr. Turner); the Western Upper Limb Facility, Sturgeon Community Hospital, St. Albert, Alberta, Canada (Dr. Chan); and the John W. Scott Health Sciences Library, University of Alberta, Edmonton, Canada (Ms. Campbell)
| | - Matthew R.G. Menon
- From the Department of Surgery, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, Alberta, Canada (Dr. Li, Dr. Chan, Dr. Menon, Dr. Ryan, Dr. Mador, and Dr. Turner); the Western Upper Limb Facility, Sturgeon Community Hospital, St. Albert, Alberta, Canada (Dr. Chan); and the John W. Scott Health Sciences Library, University of Alberta, Edmonton, Canada (Ms. Campbell)
| | - Joanna F. Ryan
- From the Department of Surgery, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, Alberta, Canada (Dr. Li, Dr. Chan, Dr. Menon, Dr. Ryan, Dr. Mador, and Dr. Turner); the Western Upper Limb Facility, Sturgeon Community Hospital, St. Albert, Alberta, Canada (Dr. Chan); and the John W. Scott Health Sciences Library, University of Alberta, Edmonton, Canada (Ms. Campbell)
| | - Brett Mador
- From the Department of Surgery, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, Alberta, Canada (Dr. Li, Dr. Chan, Dr. Menon, Dr. Ryan, Dr. Mador, and Dr. Turner); the Western Upper Limb Facility, Sturgeon Community Hospital, St. Albert, Alberta, Canada (Dr. Chan); and the John W. Scott Health Sciences Library, University of Alberta, Edmonton, Canada (Ms. Campbell)
| | - Sandra M. Campbell
- From the Department of Surgery, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, Alberta, Canada (Dr. Li, Dr. Chan, Dr. Menon, Dr. Ryan, Dr. Mador, and Dr. Turner); the Western Upper Limb Facility, Sturgeon Community Hospital, St. Albert, Alberta, Canada (Dr. Chan); and the John W. Scott Health Sciences Library, University of Alberta, Edmonton, Canada (Ms. Campbell)
| | - Simon R. Turner
- From the Department of Surgery, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, Alberta, Canada (Dr. Li, Dr. Chan, Dr. Menon, Dr. Ryan, Dr. Mador, and Dr. Turner); the Western Upper Limb Facility, Sturgeon Community Hospital, St. Albert, Alberta, Canada (Dr. Chan); and the John W. Scott Health Sciences Library, University of Alberta, Edmonton, Canada (Ms. Campbell)
| |
Collapse
|
4
|
Nau P, Worden E, Lehmann R, Kleppe K, Mancini GJ, Mancini ML, Ramshaw B. Global assessment of surgical skills (GASS): validation of a new instrument to measure global technical safety in surgical procedures. Surg Endosc 2023; 37:7964-7969. [PMID: 37442836 DOI: 10.1007/s00464-023-10116-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2023] [Accepted: 05/08/2023] [Indexed: 07/15/2023]
Abstract
BACKGROUND Broad implementation of the American Board of Surgery's entrustable professional activities initiative will require assessment instruments that are reliable and easy to use. Existing assessment instruments of general laparoscopic surgical skills have limited reliability, efficiency, and validity across the spectrum of formative (low-stakes) and summative (high-stakes) assessments. A novel six-item global assessment of surgical skills (GASS) instrument was developed and evaluated with a focus upon safe versus unsafe surgical practice scoring rubric. METHODS The GASS was developed by iterative engagement with expert laparoscopic surgeons and includes six items (economy of motion, tissue handling, appreciating operative anatomy, bimanual dexterity, achievement of hemostasis, overall performance) with a uniform three-point scoring rubric ("poor-unsafe", "adequate-safe", "good-safe"). To test inter-rater reliability, a cross-sectional study of four bariatric surgeons with experience ranging from 4 to 28 years applied the GASS and the global operative assessment of laparoscopic skills (GOALS) to 30 consecutive Roux-en-Y gastric bypass procedure operative videos. Inter-rater reliability was assessed for a simplified dichotomous "safe" versus "unsafe" scoring rubric using Gwet's AC2. RESULTS The GASS inter-rater reliability was very high across all six domains (0.88-1.00). The GASS performed comparably to the GOALS inter-rater reliability scores (0.96-1.00). The economy of motion and bimanual dexterity items had the highest percentage of unsafe ratings (9.2% and 5.8%, respectively). CONCLUSION The GASS, a novel six-item instrument of general laparoscopic surgical skills, was designed with a simple scoring rubric (poor-safe, adequate-safe, good-safe) to minimize rater burden and focus feedback to trainees and promotion evaluations on safe surgical performance. Initial evaluation of the GASS is promising, demonstrating high inter-rater reliability. Future research will seek to assess the GASS against a broader spectrum of laparoscopic procedures.
Collapse
Affiliation(s)
- Peter Nau
- Department of Surgery, Section of Bariatric Surgery, University of Iowa Hospitals and Clinics, 200 Hawkins Drive, Iowa City, IA, 52242, USA.
| | - Erin Worden
- Department of Surgery, Section of Bariatric Surgery, University of Iowa Hospitals and Clinics, 200 Hawkins Drive, Iowa City, IA, 52242, USA
| | - Ryan Lehmann
- Department of Surgery, Section of Bariatric Surgery, University of Iowa Hospitals and Clinics, 200 Hawkins Drive, Iowa City, IA, 52242, USA
| | - Kyle Kleppe
- Department of Surgery, Section of Foregut Surgery, University of Tennessee, Knoxville, TN, USA
| | - Gregory J Mancini
- Department of Surgery, Section of Foregut Surgery, University of Tennessee, Knoxville, TN, USA
| | - Matt L Mancini
- Department of Surgery, Section of Foregut Surgery, University of Tennessee, Knoxville, TN, USA
| | - Bruce Ramshaw
- CQInsights PBC, Knoxville, TN, USA
- Caresyntax Corporation, Boston, MA, USA
| |
Collapse
|
5
|
Nel D, Jonas E, Burch V, Nel A, Cairncross L, Alseidi A, George B, Ten Cate O. Entrustable professional activities in postgraduate general surgery training: a scoping review protocol. BMJ Open 2023; 13:e067322. [PMID: 36921948 PMCID: PMC10030674 DOI: 10.1136/bmjopen-2022-067322] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 03/17/2023] Open
Abstract
INTRODUCTION The competency-based medical education (CBME) movement continues to gain momentum in postgraduate physician training, resulting in increasing interest among surgical training programmes on how to implement it effectively. Entrustable professional activities (EPAs) were introduced to connect competencies (characteristics/abilities of learners) and the professional activities to be entrusted to them on qualification/graduation. Although reviews related to the field of general surgery have been published on specific aspects of CBME, for example, workplace-based assessment, there is a paucity of published guidance available for surgeon-educators to plan and implement CBME using an EPA framework. This scoping review aims to provide an overview of the current use of EPAs in general surgery. METHODS AND ANALYSIS This scoping review will be based on the original five-stage approach described by Arksey and O'Malley with the addition of protocol-specific items from the Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols extension for scoping reviews. The primary research question focuses on describing how EPAs have been used in general surgery and where potential gaps remain that warrant further study. The review will include peer-reviewed journal publications and an extensive review of grey literature sources. The following databases will be searched for published literature from 1 January 2005 to 31 December 2022: PubMed; CINAHL, Africa-Wide Information, PsycInfo and ERIC (via EBSCOhost); Scopus; and SciELO (via Web of Science). Studies that describe the use of EPAs for curriculum design, teaching and/or assessment of competence in postgraduate general surgery training will be eligible for inclusion. ETHICS AND DISSEMINATION The institutional ethics board of the University of Cape Town has granted a waiver of formal approval requirement. The dissemination strategy includes publication of results in peer-reviewed journals, presentation at international conferences and presentation to relevant stakeholders as deemed appropriate.
Collapse
Affiliation(s)
- Daniel Nel
- Department of Surgery, University of Cape Town, Cape Town, South Africa
| | - Eduard Jonas
- Department of Surgery, University of Cape Town, Cape Town, South Africa
| | - Vanessa Burch
- Department of Medicine, Groote Schuur Hospital and University of Cape Town, and the Colleges of Medicine of South Africa, Cape Town, South Africa
| | - Amy Nel
- Department of Public Health, University of Cape Town, Cape Town, South Africa
| | - Lydia Cairncross
- Department of Surgery, University of Cape Town, Cape Town, South Africa
| | - Adnan Alseidi
- Department of Surgery, University of California San Francisco, San Francisco, California, USA
| | - Brian George
- Department of Surgery, University of Michigan, Ann Arbor, Michigan, USA
| | - Olle Ten Cate
- Center for Research and Development of Education, UMC Utrecht, Utrecht, Netherlands
| |
Collapse
|