1
|
Yang D, Draganov PV, Pohl H, Aihara H, Jeyalingam T, Khashab M, Liu N, Hasan MK, Jawaid S, Othman M, Al-Haddad M, DeWitt JM, Triggs JR, Wang AY, Bechara R, Sethi A, Law R, Aadam AA, Kumta N, Sharma N, Hayat M, Zhang Y, Yi F, Elmunzer BJ. Development and initial validation of a video-based peroral endoscopic myotomy assessment tool. Gastrointest Endosc 2024; 99:177-185. [PMID: 37500019 DOI: 10.1016/j.gie.2023.07.032] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/28/2023] [Revised: 05/18/2023] [Accepted: 07/19/2023] [Indexed: 07/29/2023]
Abstract
BACKGROUND AND AIMS Video analysis has emerged as a potential strategy for performance assessment and improvement. We aimed to develop a video-based skill assessment tool for peroral endoscopic myotomy (POEM). METHODS POEM was deconstructed into basic procedural components through video analysis by an expert panel. A modified Delphi approach and 2 validation exercises were conducted to refine the POEM assessment tool (POEMAT). Twelve assessors used the final POEMAT version to grade 10 videos. Fully crossed generalizability (G) studies investigated the contributions of assessors, endoscopists' performance, and technical elements to reliability. G coefficients below .5 were considered unreliable, between .5 and .7 as modestly reliable, and above .7 as indicative of satisfactory reliability. RESULTS After task deconstruction, discussions, and the modified Delphi process, the final POEMAT comprised 9 technical elements. G analysis showed low variance for endoscopist performance (.8%-24.9%) and high interrater variability (range, 63.2%-90.1%). The G score was moderately reliable (≥.60) for "submucosal tunneling" and "myotomy" and satisfactorily reliable (≥.70) for "active hemostasis" and "mucosal closure." CONCLUSIONS We developed and established initial content and response process validity evidence for the POEMAT. Future steps include appraisal of the tool using a wider range of POEM videos to establish and improve the discriminative validity of this tool.
Collapse
Affiliation(s)
- Dennis Yang
- Center for Interventional Endoscopy, AdventHealth, Orlando, Florida, USA.
| | - Peter V Draganov
- Division of Gastroenterology and Hepatology, University of Florida, Gainesville, Florida, USA
| | - Heiko Pohl
- Veterans Affairs Medical Center, White River Junction, Vermont; Geisel School of Medicine at Dartmouth, Hanover, New Hampshire, USA
| | - Hiroyuki Aihara
- Division of Gastroenterology, Hepatology and Endoscopy, Brigham and Women's Hospital, Boston, Massachusetts, USA
| | - Thurarshen Jeyalingam
- Division of Gastroenterology and Hepatology, University of Toronto, Toronto, Ontario, Canada
| | - Mouen Khashab
- Division of Gastroenterology and Hepatology, Johns Hopkins Hospital, Baltimore, Maryland, USA
| | - Nanlong Liu
- Division of Gastroenterology, University of Louisville, Louisville, Kentucky, USA
| | - Muhammad K Hasan
- Center for Interventional Endoscopy, AdventHealth, Orlando, Florida, USA
| | - Salmaan Jawaid
- Division of Gastroenterology, Baylor College of Medicine, Houston, Texas, USA
| | - Mohamed Othman
- Division of Gastroenterology, Baylor College of Medicine, Houston, Texas, USA
| | - Mohamed Al-Haddad
- Department of Gastroenterology and Hepatology, Indiana University School of Medicine, Indianapolis, Indiana, USA
| | - John M DeWitt
- Department of Gastroenterology and Hepatology, Indiana University School of Medicine, Indianapolis, Indiana, USA
| | - Joseph R Triggs
- Division of Gastroenterology, Fox Chase Cancer Center, Temple Health, Philadelphia, Pennsylvania, USA
| | - Andrew Y Wang
- Division of Gastroenterology and Hepatology, University of Virginia, Charlottesville, Virginia, USA
| | - Robert Bechara
- Division of Gastroenterology and GI Diseases Research Unit, Queen's University, Kingston, Ontario, Canada
| | - Amrita Sethi
- Division of Digestive and Liver Diseases, Columbia University Irving Medical Center, Presbyterian Hospital, New York, New York, USA
| | - Ryan Law
- Division of Gastroenterology and Hepatology, Mayo Clinic, Minneapolis, Minnesota, USA
| | - Aziz A Aadam
- Division of Gastroenterology and Hepatology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, USA
| | - Nikhil Kumta
- Henry D. Janowitz Division of Gastroenterology, Department of Medicine, Icahn School of Medicine at Mount Sinai, New York, New York, USA
| | - Neil Sharma
- Division of Interventional Oncology and Surgical Endoscopy (IOSE), Parkview Cancer Institute, Fort Wayne, Indiana, USA
| | - Maham Hayat
- Center for Interventional Endoscopy, AdventHealth, Orlando, Florida, USA
| | - YiYang Zhang
- Center for Collaborative Research, AdventHealth Research Institute, Orlando, Florida, USA
| | - Fanchao Yi
- Center for Collaborative Research, AdventHealth Research Institute, Orlando, Florida, USA
| | - B Joseph Elmunzer
- Department of Gastroenterology and Hepatology, Medical University of South Carolina, Charleston, South Carolina, USA
| |
Collapse
|
2
|
Jørgensen RJ, Olsen RG, Svendsen MBS, Stadeager M, Konge L, Bjerrum F. Comparing Simulator Metrics and Rater Assessment of Laparoscopic Suturing Skills. JOURNAL OF SURGICAL EDUCATION 2023; 80:302-310. [PMID: 37683093 DOI: 10.1016/j.jsurg.2022.09.020] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/21/2022] [Revised: 08/17/2022] [Accepted: 09/25/2022] [Indexed: 09/10/2023]
Abstract
BACKGROUND Laparoscopic intracorporeal suturing is important to master and competence should be ensured using an optimal method in a simulated environment before proceeding to real operations. The objectives of this study were to gather validity evidence for two tools for assessing laparoscopic intracorporeal knot tying and compare the rater-based assessment of laparoscopic intracorporeal suturing with the assessment based on simulator metrics. METHODS Twenty-eight novices and 19 experienced surgeons performed four laparoscopic sutures on a Simball Box simulator twice. Two surgeons used the Intracorporeal Suturing Assessment Tool (ISAT) for blinded video rating. RESULTS Composite Simulator Score (CSS) had higher test-retest reliability than the ISAT. The correlation between the number performed procedures including suturing and ISAT score was 0.51, p<0.001, and 0.59 p<0.001 for CSS. We found an inter-rater reliability (0.72, p<0.001 for test 1 and 0.53 p<0.001 for test 2). The pass/fail rates for ISAT and CSS were similar. CONCLUSION CSS and ISAT provide similar results for assessing laparoscopic suturing but assess different aspects of performance. Using simulator metrics and raters' assessments in combination should be considered for a more comprehensive evaluation of laparoscopic knot-tying competency.
Collapse
Affiliation(s)
- Rikke Jeong Jørgensen
- Copenhagen Academy for Medical Education and Simulation, Centre for HR and Education, Capital Region, Copenhagen, Denmark.
| | - Rikke Groth Olsen
- Copenhagen Academy for Medical Education and Simulation, Centre for HR and Education, Capital Region, Copenhagen, Denmark
| | - Morten Bo Søndergaard Svendsen
- Copenhagen Academy for Medical Education and Simulation, Centre for HR and Education, Capital Region, Copenhagen, Denmark
| | - Morten Stadeager
- Copenhagen Academy for Medical Education and Simulation, Centre for HR and Education, Capital Region, Copenhagen, Denmark; Department of Surgery, Hvidovre Hospital, Copenhagen University Hospital, Copenhagen, Denmark
| | - Lars Konge
- Copenhagen Academy for Medical Education and Simulation, Centre for HR and Education, Capital Region, Copenhagen, Denmark; University of Copenhagen, Copenhagen, Denmark
| | - Flemming Bjerrum
- Copenhagen Academy for Medical Education and Simulation, Centre for HR and Education, Capital Region, Copenhagen, Denmark; Department of Surgery, Herlev-Gentofte Hospital, Herlev, Denmark
| |
Collapse
|
3
|
Patnaik R, Khan MTA, Oh T, Yamaguchi S, Fritze DM. Technical skills simulation in transplant surgery: a systematic review. GLOBAL SURGICAL EDUCATION : JOURNAL OF THE ASSOCIATION FOR SURGICAL EDUCATION 2022; 1:42. [PMID: 38013707 PMCID: PMC9483372 DOI: 10.1007/s44186-022-00028-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 03/05/2022] [Revised: 05/02/2022] [Accepted: 06/13/2022] [Indexed: 11/29/2023]
Abstract
Purpose Transplant surgery is a demanding field in which the technical skills of the surgeon correlates with patient outcomes. As such, there is potential for simulation-based training to play an important role in technical skill acquisition. This study provides a systematic assessment of the current literature regarding the use of simulation to improve surgeon technical skills in transplantation. Methods Data were collected by performing an electronic search of the PubMed and Scopus database for articles describing simulation in transplant surgery. The abstracts were screened using the preferred reporting items for systematic reviews and meta-analysis (PRISMA) guidelines. Three reviewers analyzed 172 abstracts and agreed upon articles that met the inclusion criteria for the systematic review. Results Simulators can be categorized into virtual reality simulators, cadaveric models, animal models (animate or inanimate) and synthetic physical models. No virtual reality simulators in transplant surgery are described in the literature. Three cadaveric models, seven animal models and eight synthetic physical models specific to transplant surgery are described. A total of 18 publications focusing on technical skills simulation in kidney, liver, lung, pancreas, and cardiac transplantation were found with the majority focusing on kidney transplantation. Conclusions This systematic review identifies currently reported simulation models in transplant surgery. This will serve as a reference for general surgery and transplant surgery professionals interested in using simulation to enhance their technical skills.
Collapse
Affiliation(s)
- R. Patnaik
- Department of Surgery, University of Texas Health San Antonio, 7703 Floyd Curl Drive, San Antonio, TX 78229-3900 USA
| | - M. T. A. Khan
- Department of Surgery, University of Texas Health San Antonio, 7703 Floyd Curl Drive, San Antonio, TX 78229-3900 USA
| | - T. Oh
- Long School of Medicine, University of Texas Health San Antonio, San Antonio, TX USA
| | - S. Yamaguchi
- Department of Transplant Surgery, University of Texas Health San Antonio, San Antonio, TX USA
| | - D. M. Fritze
- Department of Transplant Surgery, University of Texas Health San Antonio, San Antonio, TX USA
| |
Collapse
|
4
|
Milan FB, Grochowalski JH. A resource efficient and reliable standard setting method for OSCEs: Borderline regression method using standardized patients as sole raters in clinical case encounters with medical students. MEDICAL TEACHER 2022; 44:878-885. [PMID: 35234562 DOI: 10.1080/0142159x.2022.2041586] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Finding a reliable, practical and low-cost criterion-referenced standard setting method for performance-based assessments has proved challenging. The borderline regression method of standard setting for OSCEs has been shown to estimate reliable scores in studies using faculty as raters. Standardized patients (SPs) have been shown to be reliable OSCE raters but have not been evaluated as raters using this standard setting method. Our study sought to find whether SPs could be reliably used as sole raters in an OSCE of clinical encounters using the borderline regression standard setting method.SPs were trained for on a five-point global rating scale. In an OSCE for medical students, SPs completed skills checklists and the global rating scale. The borderline regression method was used to create case passing scores. We estimated the dependability of the final pass or fail decisions and the absolute dependability coefficients for global ratings, checklist scores, and case pass-score decisions using generalizability theory.The overall dependability estimate is 0.92 for pass or fail decisions for the complete OSCE. Dependability coefficients (0.70-0.86) of individual case passing scores range demonstrated high dependability.Based on our findings, the borderline regression method of standard setting can be used with SPs as sole raters in a medical student OSCE to produce a dependable passing score. For those already using SPs as raters, this can provide a practical criterion-referenced standard setting method for no additional cost or faculty time.
Collapse
Affiliation(s)
- Felise B Milan
- Clinical Skills Center, Albert Einstein College of Medicine, Bronx, NY, USA
| | | |
Collapse
|
5
|
Elmunzer BJ, Walsh CM, Guiton G, Serrano J, Chak A, Edmundowicz S, Kwon RS, Mullady D, Papachristou GI, Elta G, Baron TH, Yachimski P, Fogel E, Draganov PV, Taylor J, Scheiman J, Singh V, Varadarajulu S, Willingham FF, Cote G, Cotton PB, Simon V, Spitzer R, Keswani R, Wani S. Development and initial validation of an instrument for video-based assessment of technical skill in ERCP. Gastrointest Endosc 2021; 93:914-923. [PMID: 32739484 PMCID: PMC8961206 DOI: 10.1016/j.gie.2020.07.055] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/21/2020] [Accepted: 07/24/2020] [Indexed: 12/11/2022]
Abstract
BACKGROUND AND AIMS The accurate measurement of technical skill in ERCP is essential for endoscopic training, quality assurance, and coaching of this procedure. Hypothesizing that technical skill can be measured by analysis of ERCP videos, we aimed to develop and validate a video-based ERCP skill assessment tool. METHODS Based on review of procedural videos, the task of ERCP was deconstructed into its basic components by an expert panel that developed an initial version of the Bethesda ERCP Skill Assessment Tool (BESAT). Subsequently, 2 modified Delphi panels and 3 validation exercises were conducted with the goal of iteratively refining the tool. Fully crossed generalizability studies investigated the contributions of assessors, ERCP performance, and technical elements to reliability. RESULTS Twenty-nine technical elements were initially generated from task deconstruction. Ultimately, after iterative refinement, the tool comprised 6 technical elements and 11 subelements. The developmental process achieved consistent improvements in the performance characteristics of the tool with every iteration. For the most recent version of the tool, BESAT-v4, the generalizability coefficient (a reliability index) was .67. Most variance in BESAT scores (43.55%) was attributed to differences in endoscopists' skill, indicating that the tool can reliably differentiate between endoscopists based on video analysis. CONCLUSIONS Video-based assessment of ERCP skill appears to be feasible with a novel instrument that demonstrates favorable validity evidence. Future steps include determining whether the tool can discriminate between endoscopists of varying experience levels and predict important outcomes in clinical practice.
Collapse
Affiliation(s)
- B. Joseph Elmunzer
- Division of Gastroenterology and Hepatology, Medical University of South Carolina, Charleston, SC, USA
| | - Catharine M Walsh
- Division of Gastroenterology, Hepatology, and Nutrition, Learning Institute and Research Institute, Hospital for Sick Children, Toronto, Canada
| | - Gretchen Guiton
- Department of Internal Medicine, University of Colorado School of Medicine, Aurora, CO, USA
| | - Jose Serrano
- National Institute of Diabetes and Digestive and Kidney Diseases, National Institutes of Health, Bethesda, Maryland, USA
| | - Amitabh Chak
- Division of Gastroenterology and Liver Disease, Case Western Reserve University, Cleveland, OH, USA
| | - Steven Edmundowicz
- Division of Gastroenterology and Hepatology, University of Colorado Anschutz Medical Campus, Aurora, CO, USA
| | - Richard S. Kwon
- Division of Gastroenterology, University of Michigan, Ann Arbor, MI, USA
| | - Daniel Mullady
- Division of Gastroenterology, Washington University School of Medicine, St Louis, Missouri, USA
| | - Georgios I. Papachristou
- Division of Gastroenterology, Hepatology, and Nutrition, Ohio State University Wexner Medical Center, Columbus, OH, USA
| | - Grace Elta
- Division of Gastroenterology, University of Michigan, Ann Arbor, MI, USA
| | - Todd H. Baron
- Division of Gastroenterology and Hepatology, University of North Carolina, Chapel Hill, NC, USA
| | - Patrick Yachimski
- Division of Gastroenterology, Vanderbilt University, Nashville, TN, USA
| | - Evan Fogel
- Division of Gastroenterology and Hepatology, Indiana University, Indianapolis, IN, USA
| | - Peter V. Draganov
- Division of Gastroenterology, Hepatology, and Nutrition, University of Florida, Gainesville, FL, USA
| | - Jason Taylor
- Division of Gastroenterology and Hepatology, Saint Louis University, Saint Louis, MO, USA
| | - James Scheiman
- Division of Gastroenterology and Hepatology, University of Virginia, Charlottesville, VA, USA
| | - Vikesh Singh
- Division of Gastroenterology, Johns Hopkins Medical Institutions, Baltimore, MD, USA
| | | | | | - Gregory Cote
- Division of Gastroenterology and Hepatology, Medical University of South Carolina, Charleston, SC, USA
| | - Peter B. Cotton
- Division of Gastroenterology and Hepatology, Medical University of South Carolina, Charleston, SC, USA
| | - Violette Simon
- Division of Gastroenterology and Hepatology, University of Colorado Anschutz Medical Campus, Aurora, CO, USA
| | - Rebecca Spitzer
- Division of Gastroenterology and Hepatology, Medical University of South Carolina, Charleston, SC, USA
| | - Rajesh Keswani
- Division of Gastroenterology, Northwestern University, Chicago, IL, USA
| | - Sachin Wani
- Division of Gastroenterology and Hepatology, University of Colorado Anschutz Medical Campus, Aurora, CO, USA
| | | |
Collapse
|
6
|
Welton ML. Invited commentary for "The what? How? And Who? Of video based assessment. Am J Surg 2020; 221:11-12. [PMID: 32778400 DOI: 10.1016/j.amjsurg.2020.07.010] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/09/2020] [Revised: 07/15/2020] [Accepted: 07/15/2020] [Indexed: 02/02/2023]
Affiliation(s)
- Mark Lane Welton
- Fairview Health Services, United States; Section of Colon and Rectal Surgery, University of Minnesota, United States.
| |
Collapse
|