1
|
Gates RS, Krumm AE, Cate OT, Chen X, Marcotte K, Thelen AE, Deal SB, Alseidi A, Swanson D, George BC. How Reliable are Single-Question Workplace-Based Assessments in Surgery? JOURNAL OF SURGICAL EDUCATION 2024; 81:967-972. [PMID: 38816336 DOI: 10.1016/j.jsurg.2024.03.015] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/27/2023] [Revised: 02/29/2024] [Accepted: 03/18/2024] [Indexed: 06/01/2024]
Abstract
OBJECTIVE Workplace-based assessments (WBAs) play an important role in the assessment of surgical trainees. Because these assessment tools are utilized by a multitude of faculty, inter-rater reliability is important to consider when interpreting WBA data. Although there is evidence supporting the validity of many of these tools, inter-reliability evidence is lacking. This study aimed to evaluate the inter-rater reliability of multiple operative WBA tools utilized in general surgery residency. DESIGN General surgery residents and teaching faculty were recorded during 6 general surgery operations. Nine faculty raters each reviewed 6 videos and rated each resident on performance (using the Society for Improving Medical Professional Learning, or SIMPL, Performance Scale as well as the operative performance rating system (OPRS) Scale), entrustment (using the ten Cate Entrustment-Supervision Scale), and autonomy (using the Zwisch Scale). The ratings were reviewed for inter-rater reliability using percent agreement and intraclass correlations. PARTICIPANTS Nine faculty members viewed the videos and assigned ratings for multiple WBAs. RESULTS Absolute intraclass correlation coefficients for each scale ranged from 0.33 to 0.47. CONCLUSIONS All single-item WBA scales had low to moderate inter-rater reliability. While rater training may improve inter-rater reliability for single observations, many observations by many raters are needed to reliably assess trainee performance in the workplace.
Collapse
Affiliation(s)
- Rebecca S Gates
- Center for Surgical Training and Research, Department of Surgery, University of Michigan 010-A193, North Campus Research Complex 2800, Plymouth Road Ann Arbor, Michigan 48109; Department of Surgery, Carilion Clinic, 1906 Belleview Ave, Roanoke, VA 24014.
| | - Andrew E Krumm
- Department of Learning Health Sciences, University of Michigan Medical School, 209 Victor Vaughan Building, 2054, 1111 E. Catherine St., Ann Arbor, MI 48109-2054
| | - Olle Ten Cate
- University Medical Center Utrecht, Heidelberglaan 100, 3584 CX Utrecht, Netherlands
| | - Xilin Chen
- Center for Surgical Training and Research, Department of Surgery, University of Michigan 010-A193, North Campus Research Complex 2800, Plymouth Road Ann Arbor, Michigan 48109
| | - Kayla Marcotte
- Department of Learning Health Sciences, University of Michigan Medical School, 209 Victor Vaughan Building, 2054, 1111 E. Catherine St., Ann Arbor, MI 48109-2054
| | - Angela E Thelen
- Center for Surgical Training and Research, Department of Surgery, University of Michigan 010-A193, North Campus Research Complex 2800, Plymouth Road Ann Arbor, Michigan 48109
| | - Shanley B Deal
- Department of Surgery, Virginia Mason Franciscan Health, 1100 9th Ave., Seattle, WA 98101
| | - Adnan Alseidi
- Department of Surgery, University of California San Francisco, 400 Parnassus Ave, San Francisco, CA 94143
| | - David Swanson
- University of Melbourne School of Medicine, Ground Floor, Medical Building, Cnr Grattan Street & Royal Parade, University of Melbourne, VIC 3052, Australia; University of Queensland School of Medicine, Level 4, Building 69, 288 Herston Road, Herston QLD 4006, Australia
| | - Brian C George
- Center for Surgical Training and Research, Department of Surgery, University of Michigan 010-A193, North Campus Research Complex 2800, Plymouth Road Ann Arbor, Michigan 48109
| |
Collapse
|
2
|
Balvardi S, Kammili A, Hanson M, Mueller C, Vassiliou M, Lee L, Schwartzman K, Fiore JF, Feldman LS. The association between video-based assessment of intraoperative technical performance and patient outcomes: a systematic review. Surg Endosc 2022; 36:7938-7948. [PMID: 35556166 DOI: 10.1007/s00464-022-09296-6] [Citation(s) in RCA: 16] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/18/2021] [Accepted: 04/18/2022] [Indexed: 01/06/2023]
Abstract
BACKGROUND Efforts to improve surgical safety and outcomes have traditionally placed little emphasis on intraoperative performance, partly due to difficulties in measurement. Video-based assessment (VBA) provides an opportunity for blinded and unbiased appraisal of surgeon performance. Therefore, we aimed to systematically review the existing literature on the association between intraoperative technical performance, measured using VBA, and patient outcomes. METHODS Major databases (Medline, Embase, Cochrane Database, and Web of Science) were systematically searched for studies assessing the association of intraoperative technical performance measured by tools supported by validity evidence with short-term (≤ 30 days) and/or long-term postoperative outcomes. Study quality was assessed using the Newcastle-Ottawa Scale. Results were appraised descriptively as study heterogeneity precluded meta-analysis. RESULTS A total of 11 observational studies were identified involving 8 different procedures in foregut/bariatric (n = 4), colorectal (n = 4), urologic (n = 2), and hepatobiliary surgery (n = 1). The number of surgeons assessed ranged from 1 to 34; patient sample size ranged from 47 to 10,242. High risk of bias was present in 5 of 8 studies assessing short-term outcomes and 2 of 6 studies assessing long-term outcomes. Short-term outcomes were reported in 8 studies (i.e., morbidity, mortality, and readmission), while 6 reported long-term outcomes (i.e., cancer outcomes, weight loss, and urinary continence). Better intraoperative performance was associated with fewer postoperative complications (6 of 7 studies), reoperations (3 of 4 studies), and readmissions (1 of 4 studies). Long-term outcomes were less commonly investigated, with mixed results. CONCLUSION Current evidence supports an association between superior intraoperative technical performance measured using surgical videos and improved short-term postoperative outcomes. Intraoperative performance analysis using video-based assessment represents a promising approach to surgical quality-improvement.
Collapse
Affiliation(s)
- Saba Balvardi
- Department of Surgery, McGill University, 1650 Cedar Ave, D6-136, Montreal, QC, H3G 1A4, Canada
- Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, QC, Canada
| | - Anitha Kammili
- Department of Surgery, McGill University, 1650 Cedar Ave, D6-136, Montreal, QC, H3G 1A4, Canada
- Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, QC, Canada
| | - Melissa Hanson
- Department of Surgery, McGill University, 1650 Cedar Ave, D6-136, Montreal, QC, H3G 1A4, Canada
- Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, QC, Canada
| | - Carmen Mueller
- Department of Surgery, McGill University, 1650 Cedar Ave, D6-136, Montreal, QC, H3G 1A4, Canada
- Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, QC, Canada
| | - Melina Vassiliou
- Department of Surgery, McGill University, 1650 Cedar Ave, D6-136, Montreal, QC, H3G 1A4, Canada
- Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, QC, Canada
| | - Lawrence Lee
- Department of Surgery, McGill University, 1650 Cedar Ave, D6-136, Montreal, QC, H3G 1A4, Canada
- Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, QC, Canada
| | - Kevin Schwartzman
- Respiratory Division, Department of Medicine, McGill University, Montreal, QC, Canada
- McGill International Tuberculosis Centre, Research Institute of the McGill University Health Centre, Montreal, QC, Canada
| | - Julio F Fiore
- Department of Surgery, McGill University, 1650 Cedar Ave, D6-136, Montreal, QC, H3G 1A4, Canada
- Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, QC, Canada
| | - Liane S Feldman
- Department of Surgery, McGill University, 1650 Cedar Ave, D6-136, Montreal, QC, H3G 1A4, Canada.
- Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, QC, Canada.
| |
Collapse
|
3
|
Mascagni P, Alapatt D, Laracca GG, Guerriero L, Spota A, Fiorillo C, Vardazaryan A, Quero G, Alfieri S, Baldari L, Cassinotti E, Boni L, Cuccurullo D, Costamagna G, Dallemagne B, Padoy N. Multicentric validation of EndoDigest: a computer vision platform for video documentation of the critical view of safety in laparoscopic cholecystectomy. Surg Endosc 2022; 36:8379-8386. [PMID: 35171336 DOI: 10.1007/s00464-022-09112-1] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/03/2021] [Accepted: 02/07/2022] [Indexed: 01/06/2023]
Abstract
BACKGROUND A computer vision (CV) platform named EndoDigest was recently developed to facilitate the use of surgical videos. Specifically, EndoDigest automatically provides short video clips to effectively document the critical view of safety (CVS) in laparoscopic cholecystectomy (LC). The aim of the present study is to validate EndoDigest on a multicentric dataset of LC videos. METHODS LC videos from 4 centers were manually annotated with the time of the cystic duct division and an assessment of CVS criteria. Incomplete recordings, bailout procedures and procedures with an intraoperative cholangiogram were excluded. EndoDigest leveraged predictions of deep learning models for workflow analysis in a rule-based inference system designed to estimate the time of the cystic duct division. Performance was assessed by computing the error in estimating the manually annotated time of the cystic duct division. To provide concise video documentation of CVS, EndoDigest extracted video clips showing the 2 min preceding and the 30 s following the predicted cystic duct division. The relevance of the documentation was evaluated by assessing CVS in automatically extracted 2.5-min-long video clips. RESULTS 144 of the 174 LC videos from 4 centers were analyzed. EndoDigest located the time of the cystic duct division with a mean error of 124.0 ± 270.6 s despite the use of fluorescent cholangiography in 27 procedures and great variations in surgical workflows across centers. The surgical evaluation found that 108 (75.0%) of the automatically extracted short video clips documented CVS effectively. CONCLUSIONS EndoDigest was robust enough to reliably locate the time of the cystic duct division and efficiently video document CVS despite the highly variable workflows. Training specifically on data from each center could improve results; however, this multicentric validation shows the potential for clinical translation of this surgical data science tool to efficiently document surgical safety.
Collapse
Affiliation(s)
- Pietro Mascagni
- ICube, University of Strasbourg, CNRS, c/o IHU-Strasbourg, 1, place de l'hôpital, 67000, Strasbourg, France.
- Fondazione Policlinico Universitario A. Gemelli IRCCS, Rome, Italy.
| | - Deepak Alapatt
- ICube, University of Strasbourg, CNRS, c/o IHU-Strasbourg, 1, place de l'hôpital, 67000, Strasbourg, France
| | - Giovanni Guglielmo Laracca
- Department of Medical Surgical Science and Translational Medicine, Sant'Andrea Hospital, Sapienza University of Rome, Rome, Italy
| | - Ludovica Guerriero
- Department of Laparoscopic and Robotic General Surgery, Monaldi Hospital, AORN dei Colli, Naples, Italy
| | - Andrea Spota
- Scuola di Specializzazione in Chirurgia Generale, University of Milan, Milan, Italy
| | - Claudio Fiorillo
- Fondazione Policlinico Universitario A. Gemelli IRCCS, Rome, Italy
| | - Armine Vardazaryan
- ICube, University of Strasbourg, CNRS, c/o IHU-Strasbourg, 1, place de l'hôpital, 67000, Strasbourg, France
| | - Giuseppe Quero
- Fondazione Policlinico Universitario A. Gemelli IRCCS, Rome, Italy
| | - Sergio Alfieri
- Fondazione Policlinico Universitario A. Gemelli IRCCS, Rome, Italy
| | - Ludovica Baldari
- Department of Surgery, Fondazione IRCCS Ca' Granda Ospedale Maggiore Policlinico di Milano, University of Milan, Milan, Italy
| | - Elisa Cassinotti
- Department of Surgery, Fondazione IRCCS Ca' Granda Ospedale Maggiore Policlinico di Milano, University of Milan, Milan, Italy
| | - Luigi Boni
- Department of Surgery, Fondazione IRCCS Ca' Granda Ospedale Maggiore Policlinico di Milano, University of Milan, Milan, Italy
| | - Diego Cuccurullo
- Department of Laparoscopic and Robotic General Surgery, Monaldi Hospital, AORN dei Colli, Naples, Italy
| | - Guido Costamagna
- Fondazione Policlinico Universitario A. Gemelli IRCCS, Rome, Italy
| | - Bernard Dallemagne
- Institute for Research Against Digestive Cancer (IRCAD), Strasbourg, France
- IHU-Strasbourg, Institute of Image-Guided Surgery, Strasbourg, France
| | - Nicolas Padoy
- ICube, University of Strasbourg, CNRS, c/o IHU-Strasbourg, 1, place de l'hôpital, 67000, Strasbourg, France
- IHU-Strasbourg, Institute of Image-Guided Surgery, Strasbourg, France
| |
Collapse
|
4
|
Barnhill CW, Kaplan SJ, Alseidi AA, Deal SB. A comparative analysis of video-based surgical assessment: is evaluation of the entire critical portion of the operation necessary? Surg Endosc 2022; 36:6719-6723. [PMID: 35146556 DOI: 10.1007/s00464-021-08945-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/28/2021] [Accepted: 12/06/2021] [Indexed: 10/19/2022]
Abstract
BACKGROUND Previous studies of video-based operative assessments using crowd sourcing have established the efficacy of non-expert evaluations. Our group sought to establish the equivalence of abbreviating video content for operative assessment. METHODS A single institution video repository of six core general surgery operations was submitted for evaluation. Each core surgery included three unique surgical performances, totaling 18 unique operative videos. Each video was edited using four different protocols based on the critical portion of the operation: (1) custom edited critical portion (2) condensed critical portion (3) first 20 s of every minute of the critical portion, and (4) first 10 s of every minute of the critical portion. In total, 72 individually edited operative videos were submitted to the C-SATS (Crowd-Sourced Assessment of Technical Skills) platform (C-SATS) for evaluation. Aggregate score for study protocol was compared using the Kruskal-Wallis test. A multivariable, multilevel mixed-effects model was constructed to predict total skill assessment scores. RESULTS Median video lengths for each protocol were: custom, 6:20 (IQR 5:27-7:28); condensed, 10:35 (8:50-12:06); 10 s, 4:35 (2:11-6:09); and 20 s, 9:09 (4:20-12:14). There was no difference in aggregate median score among the four study protocols: custom, 15.7 (14.4-16.2); condensed, 15.8 (15.2-16.4); 10 s, 15.8 (15.3-16.1); 20 s, 16.0 (15.1-16.3); χ2 = 1.661, p = 0.65. Regression modeling demonstrated a significant, but minimal effect of the 10 s and 20 s editing protocols compared to the custom method on individual video score: condensed, + 0.33 (- 0.05-0.70), p = 0.09; 10 s, + 0.29 (0.04-0.55), p = 0.03; 20 s, + 0.40 (0.15-0.66), p = 0.002. CONCLUSION A standardized protocol for video editing abbreviated surgical performances yields reproducible assessment of surgical aptitude when assessed by non-experts.
Collapse
Affiliation(s)
- Caleb W Barnhill
- Virginia Mason Medical Center, 1100 9th Ave., C6-GS, Seattle, WA, 98101, USA.
| | - Stephen J Kaplan
- Virginia Mason Medical Center, 1100 9th Ave., C6-GS, Seattle, WA, 98101, USA
| | - Adnan A Alseidi
- Virginia Mason Medical Center, 1100 9th Ave., C6-GS, Seattle, WA, 98101, USA
| | - Shanley B Deal
- Virginia Mason Medical Center, 1100 9th Ave., C6-GS, Seattle, WA, 98101, USA
| |
Collapse
|
5
|
Esposito AC, Coppersmith NA, White EM, Yoo PS. Video Coaching in Surgical Education: Utility, Opportunities, and Barriers to Implementation. JOURNAL OF SURGICAL EDUCATION 2022; 79:717-724. [PMID: 34972670 DOI: 10.1016/j.jsurg.2021.12.004] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/01/2021] [Revised: 11/30/2021] [Accepted: 12/04/2021] [Indexed: 06/14/2023]
Abstract
OBJECTIVE This review discusses the literature on Video-Based Coaching (VBC) and explores the barriers to widespread implementation. DESIGN A search was performed on Scopus and PubMed for the terms "operation," "operating room," "surgery," "resident," "house staff," "graduate medical education," "teaching," "coaching," "assessment," "reflection," "camera," and "video" on July 27, 2021, in English. This yielded 828 results. A single author reviewed the titles and abstracts and eliminated any results that did not pertain to operative VBC or assessment. All bibliographies were reviewed, and appropriate manuscripts were included in this study. This resulted in a total of 52 manuscripts included in this review. SETTING/PARTICIPANTS Original, peer-reviewed studies focused on VBC or assessment. RESULTS VBC has been both subjectively and objectively found to be a valuable educational tool. Nearly every study of video recording in the operating room found that subjects, including surgical residents and seasoned surgeons alike, overwhelmingly considered it a useful, non-redundant adjunct to their training. Most studies that evaluated skill acquisition via standardized assessment tools found that surgical residents who underwent a VBC program had significant improvements compared to their counterparts who did not undergo video review. Despite this evidence of effectiveness, fewer than 5% of residency programs employ video recording in the operating room. Barriers to implementation include significant time commitments for proposed coaching curricula and difficulty with integration of video cameras into the operating room. CONCLUSIONS VBC has significant educational benefits, but a scalable curriculum has not been developed. An optimal solution would ensure technical ease and expediency, simple, high-quality cameras, immediate review, and overcoming entrenched surgical norms and culture.
Collapse
Affiliation(s)
- Andrew C Esposito
- Yale School of Medicine, Department of Surgery, New Haven, Connecticut
| | | | - Erin M White
- Yale School of Medicine, Department of Surgery, New Haven, Connecticut
| | - Peter S Yoo
- Yale School of Medicine, Department of Surgery, New Haven, Connecticut.
| |
Collapse
|
6
|
OR black box and surgical control tower: Recording and streaming data and analytics to improve surgical care. J Visc Surg 2021; 158:S18-S25. [PMID: 33712411 DOI: 10.1016/j.jviscsurg.2021.01.004] [Citation(s) in RCA: 15] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/14/2022]
Abstract
Effective and safe surgery results from a complex sociotechnical process prone to human error. Acquiring large amount of data on surgical care and modelling the process of surgery with artificial intelligence's computational methods could shed lights on system strengths and limitations and enable computer-based smart assistance. With this vision in mind, surgeons and computer scientists have joined forces in a novel discipline called Surgical Data Science. In this regard, operating room (OR) black boxes and surgical control towers are being developed to systematically capture comprehensive data on surgical procedures and to oversee and assist during operating rooms activities, respectively. Most of the early Surgical Data Science works have focused on understanding risks and resilience factors affecting surgical safety, the context and workflow of procedures, and team behaviors. These pioneering efforts in sensing and analyzing surgical activities, together with the advent of precise robotic actuators, bring surgery on the verge of a fourth revolution characterized by smart assistance in perceptual, cognitive and physical tasks. Barriers to implement this vision exist, but the surgical-technical partnerships set by ambitious efforts such as the OR black box and the surgical control tower are working to overcome these roadblocks and translate the vision and early works described in the manuscript into value for patients, surgeons and health systems.
Collapse
|
7
|
Jogerst KM, Eurboonyanun C, Park YS, Cassidy D, McKinley SK, Hamdi I, Phitayakorn R, Petrusa E, Gee DW. Implementation of the ACS/ APDS Resident Skills Curriculum reveals a need for rater training: An analysis using generalizability theory. Am J Surg 2021; 222:541-548. [PMID: 33516415 DOI: 10.1016/j.amjsurg.2021.01.018] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2020] [Revised: 12/16/2020] [Accepted: 01/11/2021] [Indexed: 11/25/2022]
Abstract
BACKGROUND The American College of Surgeons (ACS)/Association of Program Directors in Surgery (APDS) Resident Skills Curriculum includes validated task-specific checklists and global rating scales (GRS) for Objective Structured Assessment of Technical Skills (OSATS). However, it does not include instructions on use of these assessment tools. Since consistency of ratings is a key feature of assessment, we explored rater reliability for two skills. METHODS Surgical faculty assessed hand-sewn bowel and vascular anastomoses in real-time using the OSATS GRS. OSATS were video-taped and independently evaluated by a research resident and surgical attending. Rating consistency was estimated using intraclass correlation coefficients (ICC) and generalizability analysis. RESULTS Three-rater ICC coefficients across 24 videos ranged from 0.12 to 0.75. Generalizability reliability coefficients ranged from 0.55 to 0.8. Percent variance attributable to raters ranged from 2.7% to 32.1%. Pairwise agreement showed considerable inconsistency for both tasks. CONCLUSIONS Variability of ratings for these two skills indicate the need for rater training to increase scoring agreement and decrease rater variability for technical skill assessments.
Collapse
Affiliation(s)
- Kristen M Jogerst
- Department of General Surgery, Mayo Clinic Arizona, 5777 E. Mayo Blvd, Phoenix, AZ, 85054, USA; Department of Surgery, Massachusetts General Hospital, 55 Fruit St, Boston, MA, 02114, USA.
| | - Chalerm Eurboonyanun
- Department of Surgery, Massachusetts General Hospital, 55 Fruit St, Boston, MA, 02114, USA; Department of Surgery, Khon Kaen University, 123 Tambon Sila, Mueang Khon Kaen District, Khon Kaen 40002, Thailand.
| | - Yoon Soo Park
- Department of Emergency Medicine, Massachusetts General Hospital, 55 Fruit St, Boston, MA, 02114, USA.
| | - Douglas Cassidy
- Department of Surgery, Massachusetts General Hospital, 55 Fruit St, Boston, MA, 02114, USA.
| | - Sophia K McKinley
- Department of Surgery, Massachusetts General Hospital, 55 Fruit St, Boston, MA, 02114, USA.
| | - Isra Hamdi
- Department of Surgery, Massachusetts General Hospital, 55 Fruit St, Boston, MA, 02114, USA.
| | - Roy Phitayakorn
- Department of Surgery, Massachusetts General Hospital, 55 Fruit St, Boston, MA, 02114, USA.
| | - Emil Petrusa
- Department of Surgery, Massachusetts General Hospital, 55 Fruit St, Boston, MA, 02114, USA.
| | - Denise W Gee
- Department of Surgery, Massachusetts General Hospital, 55 Fruit St, Boston, MA, 02114, USA.
| |
Collapse
|