1
|
Rabil MJ, Jalfon M, Heckscher D, Song Z, Li F, Martin T, Sprenkle PC, Hesse D, Kim IY, Leapman MS, Cavallo JA. Association of Crowd-Sourced Assessment of Technical Skills (CSATS TM) and Outcomes of Robotic Assisted Radical Prostatectomy. Urology 2024:S0090-4295(24)00568-5. [PMID: 39019332 DOI: 10.1016/j.urology.2024.07.014] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/23/2024] [Revised: 06/24/2024] [Accepted: 07/06/2024] [Indexed: 07/19/2024]
Abstract
OBJECTIVE To investigate if use of the Crowd-Sourced Assessment of Technical Skills (CSATSTM) platform and video peer review with constructive feedback is associated with improvement in technical skill and patient outcomes for robotic-assisted laparoscopic prostatectomy (RALP). METHODS Five fellowship-trained urologists voluntarily submitted RALP cases for CSATSTM Global Evaluative Assessment of Robotic Skills (GEARS) scoring and expert narrative review between April 15, 2022 - April 30, 2023. Surgeon-selected and randomly-selected cases were reviewed. Surgeons underwent local peer review of videos with constructive feedback. Change in GEARS scores and frequency of post-operative outcomes over the 12-month periods before and during the study were analyzed in logistic regression models. Bias was assessed with sensitivity analysis comparing surgeon-selected to randomly-selected cases. RESULTS GEARS scores for randomly-selected versus surgeon-selected cases did not differ significantly. Overall GEARS score correlated positively with annual surgical RALP volume (r=0.39, p=0.003) and negatively with years in practice (r=-0.34, p=0.01). After adjusting for confounders, there was no significant improvement in overall GEARS Score (0.01±0.02/month, p=0.48); but likelihood of sepsis (Odds Ratio 0.07, 95% CI 0.01-1.00, p=0.05) and pelvic fluid collection (Odds Ratio 0.09, 95% CI 0.01-0.99, p=0.049) were significantly decreased during the intervention period (n=165) compared to the prior 12 months (n=144). No outcome increased in likelihood (p>0.05). CONCLUSIONS Integration of CSATSTM and local video peer review is associated with significant improvement in patient outcomes after RALP, despite no significant change in surgeons' GEARS scores. This is the first study demonstrating improvement in patient RALP outcomes after implementation of such a paradigm in practicing surgeons.
Collapse
Affiliation(s)
| | - Michael Jalfon
- Yale University School of Medicine, New Haven, Connecticut
| | | | - Zhiqian Song
- Yale Center for Analytical Sciences, New Haven, Connecticut
| | - Fangyong Li
- Yale Center for Analytical Sciences, New Haven, Connecticut
| | - Thomas Martin
- Yale University School of Medicine, New Haven, Connecticut
| | - Preston C Sprenkle
- Yale University School of Medicine, New Haven, Connecticut; Veterans Affairs Connecticut Healthcare System, West Haven and Newington, Connecticut
| | - David Hesse
- Yale University School of Medicine, New Haven, Connecticut
| | - Isaac Y Kim
- Yale University School of Medicine, New Haven, Connecticut
| | - Michael S Leapman
- Yale University School of Medicine, New Haven, Connecticut; Veterans Affairs Connecticut Healthcare System, West Haven and Newington, Connecticut
| | - Jaime A Cavallo
- Yale University School of Medicine, New Haven, Connecticut; Veterans Affairs Connecticut Healthcare System, West Haven and Newington, Connecticut.
| |
Collapse
|
2
|
Lakatta AC, Steppe C, Teotia SS, Haddock NT. Head-to-Head Analysis of Vertical vs Horizontal Incision Patterns in Breast Reconstruction: Surgical Outcomes and Aesthetic Implications. Aesthet Surg J 2024; 44:286-294. [PMID: 37824425 DOI: 10.1093/asj/sjad331] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2023] [Revised: 10/08/2023] [Accepted: 10/09/2023] [Indexed: 10/14/2023] Open
Abstract
BACKGROUND Modern aesthetically optimized breast reconstruction requires collaboration between the patient, breast surgeon, and plastic surgeon. To optimize both surgical outcome and aesthetic results, incision patterns must be carefully planned. OBJECTIVES We aimed to determine whether vertical or horizontal orientation of mastectomy incision was preferred in the general population and to analyze corresponding complication profiles. METHODS A retrospective review was performed of all patients undergoing bilateral mastectomy followed by autologous breast reconstruction utilizing either vertical or horizontal incision from January 2011 to November 2022. Postoperative complications of the 2 incision patterns were analyzed. Additionally, crowdsourcing was utilized to assess aesthetic implications of horizontal and vertical incision patterns on postoperative pictures of completed breast reconstruction. Survey rater demographics were also analyzed to assess differences in scoring based on voter characteristics. RESULTS There were no significant differences in postoperative breast complications between patients with horizontal or vertical incisions when considering wound, infection, seroma, hematoma, fat necrosis, or overall complications (P > .05). Crowdsourcing showed that, regardless of voter demographics, vertical incisions were preferred over horizontal incisions (P < .001). Additionally, voters who knew someone who had undergone breast reconstruction were more likely to rate all incision patterns higher than other voters (P < .001). CONCLUSIONS Although there are no significant differences in complication profiles between vertical and horizontal incisions in autologous breast reconstruction patients, vertical incision patterns are preferred aesthetically by the general population. LEVEL OF EVIDENCE: 4
Collapse
|
3
|
El-Sayed C, Yiu A, Burke J, Vaughan-Shaw P, Todd J, Lin P, Kasmani Z, Munsch C, Rooshenas L, Campbell M, Bach SP. Measures of performance and proficiency in robotic assisted surgery: a systematic review. J Robot Surg 2024; 18:16. [PMID: 38217749 DOI: 10.1007/s11701-023-01756-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/03/2023] [Accepted: 11/07/2023] [Indexed: 01/15/2024]
Abstract
Robotic assisted surgery (RAS) has seen a global rise in adoption. Despite this, there is not a standardised training curricula nor a standardised measure of performance. We performed a systematic review across the surgical specialties in RAS and evaluated tools used to assess surgeons' technical performance. Using the PRISMA 2020 guidelines, Pubmed, Embase and the Cochrane Library were searched systematically for full texts published on or after January 2020-January 2022. Observational studies and RCTs were included; review articles and systematic reviews were excluded. The papers' quality and bias score were assessed using the Newcastle Ottawa Score for the observational studies and Cochrane Risk Tool for the RCTs. The initial search yielded 1189 papers of which 72 fit the eligibility criteria. 27 unique performance metrics were identified. Global assessments were the most common tool of assessment (n = 13); the most used was GEARS (Global Evaluative Assessment of Robotic Skills). 11 metrics (42%) were objective tools of performance. Automated performance metrics (APMs) were the most widely used objective metrics whilst the remaining (n = 15, 58%) were subjective. The results demonstrate variation in tools used to assess technical performance in RAS. A large proportion of the metrics are subjective measures which increases the risk of bias amongst users. A standardised objective metric which measures all domains of technical performance from global to cognitive is required. The metric should be applicable to all RAS procedures and easily implementable. Automated performance metrics (APMs) have demonstrated promise in their wide use of accurate measures.
Collapse
Affiliation(s)
- Charlotte El-Sayed
- RCS England/HEE Robotics Research Fellow, University of Birmingham, Birmingham, United Kingdom.
| | - A Yiu
- RCS England/HEE Robotics Research Fellow, University of Birmingham, Birmingham, United Kingdom
| | - J Burke
- RCS England/HEE Robotics Research Fellow, University of Birmingham, Birmingham, United Kingdom
| | - P Vaughan-Shaw
- RCS England/HEE Robotics Research Fellow, University of Birmingham, Birmingham, United Kingdom
| | - J Todd
- RCS England/HEE Robotics Research Fellow, University of Birmingham, Birmingham, United Kingdom
| | - P Lin
- RCS England/HEE Robotics Research Fellow, University of Birmingham, Birmingham, United Kingdom
| | - Z Kasmani
- RCS England/HEE Robotics Research Fellow, University of Birmingham, Birmingham, United Kingdom
| | - C Munsch
- RCS England/HEE Robotics Research Fellow, University of Birmingham, Birmingham, United Kingdom
| | - L Rooshenas
- RCS England/HEE Robotics Research Fellow, University of Birmingham, Birmingham, United Kingdom
| | - M Campbell
- RCS England/HEE Robotics Research Fellow, University of Birmingham, Birmingham, United Kingdom
| | - S P Bach
- RCS England/HEE Robotics Research Fellow, University of Birmingham, Birmingham, United Kingdom
| |
Collapse
|
4
|
Boal MWE, Anastasiou D, Tesfai F, Ghamrawi W, Mazomenos E, Curtis N, Collins JW, Sridhar A, Kelly J, Stoyanov D, Francis NK. Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review. Br J Surg 2024; 111:znad331. [PMID: 37951600 PMCID: PMC10771126 DOI: 10.1093/bjs/znad331] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2023] [Revised: 09/18/2023] [Accepted: 09/19/2023] [Indexed: 11/14/2023]
Abstract
BACKGROUND There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula. METHODS A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias. RESULTS Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent. CONCLUSIONS Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.
Collapse
Affiliation(s)
- Matthew W E Boal
- The Griffin Institute, Northwick Park & St Marks’ Hospital, London, UK
- Wellcome/ESPRC Centre for Interventional Surgical Sciences (WEISS), University College London (UCL), London, UK
- Division of Surgery and Interventional Science, Research Department of Targeted Intervention, UCL, London, UK
| | - Dimitrios Anastasiou
- Wellcome/ESPRC Centre for Interventional Surgical Sciences (WEISS), University College London (UCL), London, UK
- Medical Physics and Biomedical Engineering, UCL, London, UK
| | - Freweini Tesfai
- The Griffin Institute, Northwick Park & St Marks’ Hospital, London, UK
- Wellcome/ESPRC Centre for Interventional Surgical Sciences (WEISS), University College London (UCL), London, UK
| | - Walaa Ghamrawi
- The Griffin Institute, Northwick Park & St Marks’ Hospital, London, UK
| | - Evangelos Mazomenos
- Wellcome/ESPRC Centre for Interventional Surgical Sciences (WEISS), University College London (UCL), London, UK
- Medical Physics and Biomedical Engineering, UCL, London, UK
| | - Nathan Curtis
- Department of General Surgey, Dorset County Hospital NHS Foundation Trust, Dorchester, UK
| | - Justin W Collins
- Division of Surgery and Interventional Science, Research Department of Targeted Intervention, UCL, London, UK
- University College London Hospitals NHS Foundation Trust, London, UK
| | - Ashwin Sridhar
- Division of Surgery and Interventional Science, Research Department of Targeted Intervention, UCL, London, UK
- University College London Hospitals NHS Foundation Trust, London, UK
| | - John Kelly
- Division of Surgery and Interventional Science, Research Department of Targeted Intervention, UCL, London, UK
- University College London Hospitals NHS Foundation Trust, London, UK
| | - Danail Stoyanov
- Wellcome/ESPRC Centre for Interventional Surgical Sciences (WEISS), University College London (UCL), London, UK
- Computer Science, UCL, London, UK
| | - Nader K Francis
- The Griffin Institute, Northwick Park & St Marks’ Hospital, London, UK
- Division of Surgery and Interventional Science, Research Department of Targeted Intervention, UCL, London, UK
- Yeovil District Hospital, Somerset Foundation NHS Trust, Yeovil, Somerset, UK
| |
Collapse
|
5
|
Addison P, Bitner DP, Addy J, Dechario S, Husk G, Antonacci A, Talamini M, Giangola G, Filicori F. Does Surgeon Experience Correlate with Crowd-Sourced Skill Assessment in Robotic Bariatric Surgery? Am Surg 2023; 89:5253-5262. [PMID: 36454236 DOI: 10.1177/00031348221142586] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/22/2023]
Abstract
BACKGROUND The Global Evaluative Assessment of Robotic Skills (GEARS) rubric provides a measure of skill in robotic surgery. We hypothesize surgery performed by more experienced operators will be associated with higher GEARS scores. METHOD Patients undergoing sleeve gastrectomy from 2016 to 2020 were analyzed. Three groups were defined by time in practice: less than 5, between 5 and 15, and more than 15 years. Continuous variables were compared with ANOVA and multivariable regression was performed. RESULTS Fourteen operators performing 154 cases were included. More experienced surgeons had higher GEARS scores and shorter operative times. On multivariable regression, operative time (P = 0.027), efficiency (P = .022), depth perception (P = 0.033), and bimanual dexterity (P = 0.047) were associated with experience. CONCLUSIONS In our video-based assessment (VBA) model, operative time and several GEARS subcomponent scores were associated with surgical experience. Further studies should determine the association between these metrics and surgical outcomes.
Collapse
Affiliation(s)
- Poppy Addison
- Intraoperative Performance Analytics Laboratory, Department of General Surgery, Lenox Hill Hospital, New York, NY, USA
| | - Daniel P Bitner
- Intraoperative Performance Analytics Laboratory, Department of General Surgery, Lenox Hill Hospital, New York, NY, USA
| | - Jermyn Addy
- Intraoperative Performance Analytics Laboratory, Department of General Surgery, Lenox Hill Hospital, New York, NY, USA
| | | | - Gregg Husk
- Intraoperative Performance Analytics Laboratory, Department of General Surgery, Lenox Hill Hospital, New York, NY, USA
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, NY, USA
| | - Anthony Antonacci
- Intraoperative Performance Analytics Laboratory, Department of General Surgery, Lenox Hill Hospital, New York, NY, USA
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, NY, USA
| | - Mark Talamini
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, NY, USA
- Department of General Surgery, North Shore University Hospital, Manhasset, NY, USA
| | - Gary Giangola
- Intraoperative Performance Analytics Laboratory, Department of General Surgery, Lenox Hill Hospital, New York, NY, USA
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, NY, USA
| | - Filippo Filicori
- Intraoperative Performance Analytics Laboratory, Department of General Surgery, Lenox Hill Hospital, New York, NY, USA
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, NY, USA
| |
Collapse
|
6
|
Olsen RG, Konge L, Hayatzaki K, Mortensen MA, Bube SH, Røder A, Azawi N, Bjerrum F. Laypersons versus experienced surgeons in assessing simulated robot-assisted radical prostatectomy. World J Urol 2023; 41:3745-3751. [PMID: 37882808 PMCID: PMC10693505 DOI: 10.1007/s00345-023-04664-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2023] [Accepted: 09/29/2023] [Indexed: 10/27/2023] Open
Abstract
BACKGROUND Feedback is important for surgical trainees but it can be biased and time-consuming. We examined crowd-sourced assessment as an alternative to experienced surgeons' assessment of robot-assisted radical prostatectomy (RARP). METHODS We used video recordings (n = 45) of three RARP modules on the RobotiX, Simbionix simulator from a previous study in a blinded comparative assessment study. A group of crowd workers (CWs) and two experienced RARP surgeons (ESs) evaluated all videos with the modified Global Evaluative Assessment of Robotic Surgery (mGEARS). RESULTS One hundred forty-nine CWs performed 1490 video ratings. Internal consistency reliability was high (0.94). Inter-rater reliability and test-retest reliability were low for CWs (0.29 and 0.39) and moderate for ESs (0.61 and 0.68). In an Analysis of Variance (ANOVA) test, CWs could not discriminate between the skill level of the surgeons (p = 0.03-0.89), whereas ES could (p = 0.034). CONCLUSION We found very low agreement between the assessments of CWs and ESs when they assessed robot-assisted radical prostatectomies. As opposed to ESs, CWs could not discriminate between surgical experience using the mGEARS ratings or when asked if they wanted the surgeons to perform their robotic surgery.
Collapse
Affiliation(s)
- Rikke Groth Olsen
- Copenhagen Academy for Medical Education and Simulation (CAMES), University Hospital - Rigshospitalet, Ryesgade 53B, 2100, Copenhagen, Denmark.
- Department of Urology, Copenhagen Prostate Cancer Center, Copenhagen University Hospital - Rigshospitalet, Copenhagen, Denmark.
- Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark.
| | - Lars Konge
- Copenhagen Academy for Medical Education and Simulation (CAMES), University Hospital - Rigshospitalet, Ryesgade 53B, 2100, Copenhagen, Denmark
- Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
| | | | - Mike Allan Mortensen
- Department of Urology, Odense University Hospital, Odense, Denmark
- Department of Clinical Research, University of Southern Denmark, Odense, Denmark
| | - Sarah Hjartbro Bube
- Copenhagen Academy for Medical Education and Simulation (CAMES), University Hospital - Rigshospitalet, Ryesgade 53B, 2100, Copenhagen, Denmark
- Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
- Department of Urology, Zealand University Hospital, Roskilde, Denmark
| | - Andreas Røder
- Department of Urology, Copenhagen Prostate Cancer Center, Copenhagen University Hospital - Rigshospitalet, Copenhagen, Denmark
- Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
| | - Nessn Azawi
- Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
- Department of Urology, Zealand University Hospital, Roskilde, Denmark
| | - Flemming Bjerrum
- Copenhagen Academy for Medical Education and Simulation (CAMES), University Hospital - Rigshospitalet, Ryesgade 53B, 2100, Copenhagen, Denmark
- Department of Surgery, Herlev-Gentofte Hospital, Herlev, Denmark
| |
Collapse
|
7
|
Ketel MHM, Klarenbeek BR, Eddahchouri Y, Cuesta MA, van Daele E, Gutschow CA, Hölscher AH, Hubka M, Luyer MDP, Merritt RE, Nieuwenhuijzen GAP, Shen Y, Abma IL, Rosman C, van Workum F. Crowd-sourced and expert video assessment in minimally invasive esophagectomy. Surg Endosc 2023; 37:7819-7828. [PMID: 37605010 PMCID: PMC10520122 DOI: 10.1007/s00464-023-10297-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/24/2023] [Accepted: 07/02/2023] [Indexed: 08/23/2023]
Abstract
BACKGROUND Video-based assessment by experts may structurally measure surgical performance using procedure-specific competency assessment tools (CATs). A CAT for minimally invasive esophagectomy (MIE-CAT) was developed and validated previously. However, surgeon's time is scarce and video assessment is time-consuming and labor intensive. This study investigated non-procedure-specific assessment of MIE video clips by MIE experts and crowdsourcing, collective surgical performance evaluation by anonymous and untrained laypeople, to assist procedure-specific expert review. METHODS Two surgical performance scoring frameworks were used to assess eight MIE videos. First, global performance was assessed with the non-procedure-specific Global Operative Assessment of Laparoscopic Skills (GOALS) of 64 procedural phase-based video clips < 10 min. Each clip was assessed by two MIE experts and > 30 crowd workers. Second, the same experts assessed procedure-specific performance with the MIE-CAT of the corresponding full-length video. Reliability and convergent validity of GOALS for MIE were investigated using hypothesis testing with correlations (experience, blood loss, operative time, and MIE-CAT). RESULTS Less than 75% of hypothesized correlations between GOALS scores and experience of the surgical team (r < 0.3), blood loss (r = - 0.82 to 0.02), operative time (r = - 0.42 to 0.07), and the MIE-CAT scores (r = - 0.04 to 0.76) were met for both crowd workers and experts. Interestingly, experts' GOALS and MIE-CAT scores correlated strongly (r = 0.40 to 0.79), while crowd workers' GOALS and experts' MIE-CAT scores correlations were weak (r = - 0.04 to 0.49). Expert and crowd worker GOALS scores correlated poorly (ICC ≤ 0.42). CONCLUSION GOALS assessments by crowd workers lacked convergent validity and showed poor reliability. It is likely that MIE is technically too difficult to assess for laypeople. Convergent validity of GOALS assessments by experts could also not be established. GOALS might not be comprehensive enough to assess detailed MIE performance. However, expert's GOALS and MIE-CAT scores strongly correlated indicating video clip (instead of full-length video) assessments could be useful to shorten assessment time.
Collapse
Affiliation(s)
- Mirte H M Ketel
- Department of Surgery, Radboud University Medical Center, Nijmegen, The Netherlands.
| | | | - Yassin Eddahchouri
- Department of Surgery, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Miguel A Cuesta
- Department of Surgery, Amsterdam University Medical Centers, Location VUmc, Amsterdam, The Netherlands
| | - Elke van Daele
- Department of Digestive Surgery, Ghent University Hospital, Ghent, Belgium
| | - Christian A Gutschow
- Department of Surgery and Transplantation, University Hospital Zurich, Zurich, Switzerland
| | - Arnulf H Hölscher
- Department for General, Visceral and Trauma Surgery, Elisabeth-Krankenhaus-Essen GmbH, Essen, Germany
| | - Michal Hubka
- Department of Thoracic Surgery, Virginia Mason Medical Center, Seattle, SE, USA
| | - Misha D P Luyer
- Department of Surgery, Catharina Hospital, Eindhoven, The Netherlands
| | - Robert E Merritt
- Department of Surgery, Ohio State University - Wexner Medical Center, Columbus, OH, USA
| | | | - Yaxing Shen
- Department of Thoracic Surgery, Zhongshan Hospital, Fudan University, Shanghai, China
| | - Inger L Abma
- IQ Healthcare, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Camiel Rosman
- Department of Surgery, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Frans van Workum
- Department of Surgery, Radboud University Medical Center, Nijmegen, The Netherlands
- Department of Surgery, Canisius-Wilhelmina Hospital, Nijmegen, The Netherlands
| |
Collapse
|
8
|
Ross SB, Modasi A, Christodoulou M, Sucandy I, Mehran A, Lobe TE, Witkowski E, Satava R. New generation evaluations: video-based surgical assessments : A technology update. Surg Endosc 2023; 37:7401-7411. [PMID: 37608232 DOI: 10.1007/s00464-023-10311-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/06/2023] [Accepted: 07/12/2023] [Indexed: 08/24/2023]
Abstract
BACKGROUND Surgical skill training, assessment, and feedback are the backbone of surgical training. High-quality skills require expert supervision and evaluation throughout a resource-intensive multi-year training process. As technological barriers to internet access and the ability to save and upload surgical videos continue to improve, video-based assessment technology is emerging as a tool that could reshape surgical training for the next generation of surgeons. Video-based assessment platforms have the potential to allow surgeons from across the globe to upload their surgical videos online and receive high-quality, standardized, and unbiased feedback. They combine visual recordings of a surgeon's operative technique, with standardized grading tools that have the potential to significantly impact surgical training and technical skill acquisition across the world. METHOD The platforms included in this review are in various stages of development after a thorough discussion with national experts on the SAGES TAVAC (Technology and Value Assessments) Committee. For each VBA program, a description of its platform was given and a literature review was obtained using a PubMed search performed from inception until December 2021. RESULTS The study reviewed all video-based assessment programs currently available in the market, identified their strengths and weaknesses, and how they can be optimized in future. CONCLUSION The technological platforms will play a key role in the training and technical skill acquisition of the next generation of surgeons and can have an immense impact on patient care across the world. There is immense potential for all these platforms to grow and become incorporated within the framework of an effective surgical training program.
Collapse
Affiliation(s)
- Sharona B Ross
- Digestive Health Institute, AdventHealth Tampa, 3000 Medical Park Dr. Suite 500, Tampa, FL, 33613, USA.
| | | | - Maria Christodoulou
- Digestive Health Institute, AdventHealth Tampa, 3000 Medical Park Dr. Suite 500, Tampa, FL, 33613, USA
| | - Iswanto Sucandy
- Digestive Health Institute, AdventHealth Tampa, 3000 Medical Park Dr. Suite 500, Tampa, FL, 33613, USA
| | - Anvari Mehran
- Division of General Surgery, St. Joseph's Healthcare, Hamilton, ON, Canada
- Department of Surgery, McMaster University, Hamilton, ON, Canada
| | - Thom E Lobe
- Department of Pediatric Surgery, University of Illinois at Chicago, Chicago, IL, USA
| | - Elan Witkowski
- Department of Surgery, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | | |
Collapse
|
9
|
Laverty RB, Khan MT, Patnaik R, Lee CS, Leonardo CD, Krell RW, Stull MC. Intentional enterotomies: validation of a novel robotic surgery training exercise. J Robot Surg 2023; 17:2109-2115. [PMID: 37219784 DOI: 10.1007/s11701-023-01625-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2023] [Accepted: 05/16/2023] [Indexed: 05/24/2023]
Abstract
While laparoscopic simulation-based training is a well-established component of general surgery training, no such requirement or standardized curriculum exists for robotic surgery. Furthermore, there is a lack of high-fidelity electrocautery simulation training exercises in the literature. Using Messick's validity framework, we sought to determine the content, response process, internal content and construct validity of a novel inanimate tissue model that utilizes electrocautery for potential incorporation in such curricula. A multi-institutional, prospective study involving medical students (MS) and general surgery residents (PGY1-3) was conducted. Participants performed an exercise using a biotissue bowel model on the da Vinci Xi robotic console during which they created an enterotomy using electrocautery, followed by approximation with interrupted sutures. Participant performance was recorded and then scored by crowd-sourced assessors of technical skill, along with three of the authors. Construct validity was determined via difference in Global Evaluative Assessment of Robotic Skills (GEARS) score, time to completion, and total number of errors between the two cohorts. Upon completion of the exercise, participants were surveyed on their perception of the exercise and its impact on their robotic training to determine content validity. 31 participants were enrolled and separated into two cohorts: MS + PGY1 vs. PGY2-3. Time spent on the robotic trainer (0.8 vs. 8.13 h, p = 0.002), number of bedside robotic assists (5.7 vs. 14.8, p < 0.001), and number of robotic cases as primary surgeon (0.3 vs. 13.1, p < 0.001) were statistically significant between the two groups. Differences in GEARS scores (18.5 vs. 19.9, p = 0.001), time to completion (26.1 vs. 14.4 min, p < 0.001), and total errors (21.5 vs. 11.9, p = 0.018) between the groups were statistically significant as well. Of the 23 participants that completed the post-exercise survey, 87% and 91.3% reported improvement in robotic surgical ability and confidence, respectively. On a 10-point Likert scale, respondents rated the realism of the exercise 7.5, educational benefit 9.1, and effectiveness in teaching robotic skills 8.7. Controlling for the upfront investment of certain training materials, each exercise iteration cost ~ $30. This study confirmed the content, response process, internal structure and construct validity of a novel, high-fidelity and cost-effective inanimate tissue exercise which successfully incorporates electrocautery. Consideration should be given to its addition to robotic surgery training programs.
Collapse
Affiliation(s)
- Robert B Laverty
- Department of Surgery, Brooke Army Medical Center, San Antonio, TX, USA
| | - Mustafa T Khan
- Department of General Surgery, Long School of Medicine, University of Texas Health San Antonio, 7703 Floyd Curl Drive, San Antonio, TX, 78229-3900, USA.
| | - Ronit Patnaik
- Department of General Surgery, Long School of Medicine, University of Texas Health San Antonio, 7703 Floyd Curl Drive, San Antonio, TX, 78229-3900, USA
| | - Christina S Lee
- Department of Surgery, Brooke Army Medical Center, San Antonio, TX, USA
| | - Cassandra D Leonardo
- Department of General Surgery, Long School of Medicine, University of Texas Health San Antonio, 7703 Floyd Curl Drive, San Antonio, TX, 78229-3900, USA
| | - Robert W Krell
- Department of Surgery, Brooke Army Medical Center, San Antonio, TX, USA
| | - Mamie C Stull
- Department of Surgery, Brooke Army Medical Center, San Antonio, TX, USA
| |
Collapse
|
10
|
Geoghegan L, Papadopoulos D, Petrie N, Teo I, Papavasiliou T. Utilization of a 3D Printed Simulation Training Model to Improve Microsurgical Training. PLASTIC AND RECONSTRUCTIVE SURGERY-GLOBAL OPEN 2023; 11:e4898. [PMID: 37020985 PMCID: PMC10069837 DOI: 10.1097/gox.0000000000004898] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2022] [Accepted: 01/24/2023] [Indexed: 04/05/2023]
Abstract
Simulation is integral to the development and maintenance of micro- surgical skills. Several simulation models have been described ranging from bench- top to live animal models. High fidelity models are often burdened by cost and ethical issues limiting widespread implementation. This study aims to determine the feasibility of a microsurgical training platform using the Konjac noodle model.
Collapse
Affiliation(s)
- Luke Geoghegan
- From the Department of Plastic Surgery, Oxford University Hospitals, John Radcliffe Hospital, Headington, Oxford, UK
- Section of Vascular Surgery, Department of Surgery and Cancer, Imperial College London, London, UK
| | - Dimitrios Papadopoulos
- Department of Plastic and Reconstructive Surgery, Queen Victoria Hospital NHS Foundation Trust, East Grinstead, UK
| | - Nicola Petrie
- From the Department of Plastic Surgery, Oxford University Hospitals, John Radcliffe Hospital, Headington, Oxford, UK
| | - Isabel Teo
- From the Department of Plastic Surgery, Oxford University Hospitals, John Radcliffe Hospital, Headington, Oxford, UK
| | - Theodora Papavasiliou
- From the Department of Plastic Surgery, Oxford University Hospitals, John Radcliffe Hospital, Headington, Oxford, UK
| |
Collapse
|
11
|
Brown JD, Kuchenbecker KJ. Effects of automated skill assessment on robotic surgery training. Int J Med Robot 2023; 19:e2492. [PMID: 36524325 DOI: 10.1002/rcs.2492] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2022] [Revised: 12/07/2022] [Accepted: 12/08/2022] [Indexed: 12/23/2022]
Abstract
BACKGROUND Several automated skill-assessment approaches have been proposed for robotic surgery, but their utility is not well understood. This article investigates the effects of one machine-learning-based skill-assessment approach on psychomotor skill development in robotic surgery training. METHODS N = 29 trainees (medical students and residents) with no robotic surgery experience performed five trials of inanimate peg transfer with an Intuitive Surgical da Vinci Standard robot. Half of the participants received no post-trial feedback. The other half received automatically calculated scores from five Global Evaluative Assessment of Robotic Skill domains post-trial. RESULTS There were no significant differences between the groups regarding overall improvement or skill improvement rate. However, participants who received post-trial feedback rated their overall performance improvement significantly lower than participants who did not receive feedback. CONCLUSIONS These findings indicate that automated skill evaluation systems might improve trainee self-awareness but not accelerate early stage psychomotor skill development in robotic surgery training.
Collapse
Affiliation(s)
- Jeremy D Brown
- Department of Mechanical Engineering, Johns Hopkins University, Baltimore, Maryland, USA
| | - Katherine J Kuchenbecker
- Haptic Intelligence Department, Max Planck Institute for Intelligent Systems, Stuttgart, Germany
| |
Collapse
|
12
|
Khan MTA, Patnaik R, Lee CS, Willson CM, Demario VK, Krell RW, Laverty RB. Systematic review of academic robotic surgery curricula. J Robot Surg 2022; 17:719-743. [DOI: 10.1007/s11701-022-01500-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2022] [Accepted: 11/14/2022] [Indexed: 11/23/2022]
|
13
|
Zheng Y, Ershad M, Fey AM. Toward Correcting Anxious Movements Using Haptic Cues on the Da Vinci Surgical Robot. PROCEEDINGS OF THE ... IEEE/RAS-EMBS INTERNATIONAL CONFERENCE ON BIOMEDICAL ROBOTICS AND BIOMECHATRONICS. IEEE/RAS-EMBS INTERNATIONAL CONFERENCE ON BIOMEDICAL ROBOTICS AND BIOMECHATRONICS 2022; 2022:10.1109/biorob52689.2022.9925380. [PMID: 37408769 PMCID: PMC10321328 DOI: 10.1109/biorob52689.2022.9925380] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/07/2023]
Abstract
Surgical movements have an important stylistic quality that individuals without formal surgical training can use to identify expertise. In our prior work, we sought to characterize quantitative metrics associated with surgical style and developed a near-real-time detection framework for stylistic deficiencies using a commercial haptic device. In this paper, we implement bimanual stylistic detection on the da Vinci Research Kit (dVRK) and focus on one stylistic deficiency, "Anxious", which may describe movements under stressful conditions. Our goal is to potentially correct these "Anxious" movements by exploring the effects of three different types of haptic cues (time-variant spring, damper, and spring-damper feedback) on performance during a basic surgical training task using the da Vinci Research Kit (dVRK). Eight subjects were recruited to complete peg transfer tasks using a randomized order of haptic cues and with baseline trials between each task. Overall, all cues lead to a significant improvement over baseline economy of volume and time-variant spring haptic cues lead to significant improvements in reducing the classified "Anxious" movements and also corresponded with significantly lower path length and economy of volume for the non-dominant hand. This work is the first step in evaluating our stylistic detection model on a surgical robot and could lay the groundwork for future methods to actively and adaptively reduce the negative effect of stress in the operating room.
Collapse
Affiliation(s)
- Yi Zheng
- Department of Mechanical Engineering, the University of Texas at Austin, 204 East Dean Keeton Street, Austin, TX 78712, USA
| | - Marzieh Ershad
- Intuitive Surgical, Inc., 1020 Kifer Road Sunnyvale, CA 94086
| | - Ann Majewicz Fey
- Department of Mechanical Engineering, the University of Texas at Austin, 204 East Dean Keeton Street, Austin, TX 78712, USA
- Department of Surgery, UT South-western Medical Center, 5323 Harry Hines Blvd, Dallas, TX 75390, USA
| |
Collapse
|
14
|
Soleymani A, Li X, Tavakoli M. A Domain-Adapted Machine Learning Approach for Visual Evaluation and Interpretation of Robot-Assisted Surgery Skills. IEEE Robot Autom Lett 2022. [DOI: 10.1109/lra.2022.3186769] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Affiliation(s)
- Abed Soleymani
- Electrical and Computer Engineering Department, University of Alberta, Edmonton, AB, Canada
| | - Xingyu Li
- Electrical and Computer Engineering Department, University of Alberta, Edmonton, AB, Canada
| | - Mahdi Tavakoli
- Electrical and Computer Engineering Department, University of Alberta, Edmonton, AB, Canada
| |
Collapse
|
15
|
Karani R, Tapiero S, Jefferson FA, Vernez S, Xie L, Larson KN, Osann K, Okhunov Z, Patel RM, Landman J, Clayman RV, Stephany HA. Crowd-Sourced Assessment of Surgical Skills of Urology Resident Applicants: Four-Year Experience. JOURNAL OF SURGICAL EDUCATION 2021; 78:2030-2037. [PMID: 34147416 DOI: 10.1016/j.jsurg.2021.05.005] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/16/2020] [Revised: 04/21/2021] [Accepted: 05/16/2021] [Indexed: 06/12/2023]
Abstract
OBJECTIVE To determine a) if surgical skills among urology resident applicants could be reliably assessed via crowdsourcing and b) to what extent surgical skills testing impacts resident selection. DESIGN Interviewees completed the following surgical skills tasks during their interview day: open knot tying (OKT), laparoscopic peg transfer (LPT), and robotic suturing (RS). Urology faculty and crowd-workers evaluated each applicant's video-recorded performance using validated scoring and were assessed for agreement using Cronbach's alpha. Applicants' USMLE scores, interview scores, and Jefferson Scale of Physician Empathy (JSPE-S) scores were assessed for correlation with skills testing scores and match rank. Additionally, a survey was distributed to interviewees assessing match outcomes. SETTING University of California Irvine Department of Urology, Surgical Skills Laboratory PARTICIPANTS: All 94 urology residency interviewees at the University of California Irvine Department of Urology from 2015-2018 were invited to complete the three surgical skills tasks on their interview day. RESULTS Survey responses were received from all 94 interviewees (100%). Crowd and expert agreement was good (α=0.88), fair (α=0.67), and poor (α=0.32) for LPT, RS, and OKT scores, respectively. The skills testing scores did not correlate with match rank, USMLE score, or JSPE-S score. On multivariate analysis, only interview score (r= -0.723; p<0.001) and faculty LPT score (r=-0.262; p=0.001) were significant predictors of match rank. Interviewees who reported matching into a top 3 residency choice had significantly higher faculty LPT scores than those who did not (11.9 vs. 9.7, p=0.03). CONCLUSIONS Surgical skills overall did not significantly impact match rank. Expert assessment of laparoscopic peg transfer skills and interview performance among urology resident applicants correlated with match rank.
Collapse
Affiliation(s)
- Rajiv Karani
- Department of Urology, University of California, Irvine, Orange, California
| | - Shlomi Tapiero
- Department of Urology, University of California, Irvine, Orange, California
| | | | - Simone Vernez
- Department of Urology, University of California, Irvine, Orange, California
| | - Lillian Xie
- Department of Urology, University of California, Irvine, Orange, California
| | - Krista N Larson
- Department of Urology, University of California, Irvine, Orange, California
| | - Kathryn Osann
- Department of Urology, University of California, Irvine, Orange, California
| | - Zhamshid Okhunov
- Department of Urology, University of California, Irvine, Orange, California
| | - Roshan M Patel
- Department of Urology, University of California, Irvine, Orange, California
| | - Jaime Landman
- Department of Urology, University of California, Irvine, Orange, California
| | - Ralph V Clayman
- Department of Urology, University of California, Irvine, Orange, California
| | - Heidi A Stephany
- Department of Urology, University of California, Irvine, Orange, California.
| |
Collapse
|
16
|
Nair D, Wells JM, Yi M, Beasley S. Is Technical Expertise Necessary to Assess Technical Expertise? Let's Ask the Kids. J Laparoendosc Adv Surg Tech A 2021; 31:1363-1366. [PMID: 34677094 DOI: 10.1089/lap.2021.0321] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022] Open
Abstract
Background: Simulation has an increasing role in surgical training; however, using validated tools such as the Objective Structured Assessment of Technical Skills (OSATS) is time-consuming, which may be a potential barrier to simulation-based training. This study tests the hypothesis that assessors with technical expertise are necessary to objectively score a technical task. Methods: Three tasks, ring transfer, needle pass, and atresia cut, were performed using a synthetic thoracoscopic simulator for esophageal atresia/tracheoesophageal fistula. Three pediatric surgeons, 6 novice adults, and 3 children aged 9-13 years scored each attempt using the overall global OSATS rating from 1 to 5 (1 "repeated or awkward movements" to 5 "fluid movement. No awkwardness"). Results: For the ring transfer, all assessors scored novice participants significantly less than expert and intermediate scores (surgeon P = .0004, nonsurgeon adults P = .0009 and children P = .0003). For the needle pass, all assessors gave significantly different scores between novices and experts (surgeon P = .0007, nonsurgeon adult P = .0008, and children P = .0040). For the atresia cut, surgeon assessors gave significantly higher scores for experts and intermediate and novice (P = .0004). Nonsurgeon assessors gave experts and intermediates significantly higher scores than novices (P = .0001). Surgeon assessors achieved good reliability for ring transfer (0.8252) and needle pass (0.7769) compared with nonsurgeon assessors who showed poor reliability for the ring transfer (0.3959) and moderate for the needle pass (0.6551). Conclusions: Expertise in performing these procedures is not a prerequisite for an assessor to evaluate the technical skill, hence assessors of skill acquisition can be nonexpert, a nonsurgeon, or even a child. The variability in all groups suggests that reliability overall is increased with multiple assessors. Although nonsurgeon assessors may be appropriate for formative assessments, they lack the reliability to provide assessment of competence for high stakes complex tasks. Summative assessment will likely require at least 1 surgeon/expert assessor to provide reliability.
Collapse
Affiliation(s)
- David Nair
- Canterbury District Health Board, Department of Paediatric Surgery, Christchurch, New Zealand
| | - Jonathan M Wells
- Canterbury District Health Board, Department of Paediatric Surgery, Christchurch, New Zealand.,University of Otago, Department of Surgery, Christchurch, New Zealand
| | - Ma Yi
- Canterbury District Health Board, Department of Biostatistics, Christchurch, New Zealand
| | - Spencer Beasley
- Canterbury District Health Board, Department of Paediatric Surgery, Christchurch, New Zealand.,University of Otago, Department of Surgery, Christchurch, New Zealand
| |
Collapse
|
17
|
Lovasik BP, Fay KT, Patel A, Stetler J, Papandria D, Santore MT, Davis SS, Lin E, Srinivasan JK. Development of a laparoscopic surgical skills simulation curriculum: Enhancing resident training through directed coaching and closed-loop feedback. Surgery 2021; 171:897-903. [PMID: 34521515 DOI: 10.1016/j.surg.2021.08.020] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2021] [Revised: 08/03/2021] [Accepted: 08/11/2021] [Indexed: 01/22/2023]
Abstract
BACKGROUND Performance feedback through peer coaching and rigorous self-assessment is a critical part of technical skills improvement. However, formal collaborative programs using operative video-based skills assessments to generate peer coaching feedback have only been validated among attending surgeons. In this study, we developed a unique longitudinal, simulation video-based laparoscopic skills resident curriculum using video-based peer coaching and evaluated its association with skills acquisition among surgical trainees. METHODS The laparoscopic simulation curriculum consists of a pre-practice laparoscopic skill video recording, followed by receipt of directed coaching and feedback on performance from a faculty coach, a peer coach, and self-coaching. Residents then completed 6 weeks of feedback-directed practice and submitted a second post-practice laparoscopic skill video recording of the same skill, which was evaluated by a minimally invasive surgery expert grader. All general surgery residents in a single institution were enrolled, with 107 residents completing the curriculum in its initial 2 years. RESULTS Overall, more than two-thirds of residents achieved skills proficiency on their expert assessments, with similar rates of residents achieving skills proficiency at all postgraduate year levels. Significant improvements between the pre-practice assessments and post-practice assessments were most frequently seen in the instrument handling, precision, and motion & flow categories (P < .05 each). Faculty provided the highest number and proportion of closed-loop comments; residents' self-coaching feedback had the lowest number of closed-loop comments, with 83% of self-assessments containing none. CONCLUSION In this study, we describe the successful implementation of a longitudinal laparoscopic skills video-based coaching curriculum designed to improve residents' laparoscopic technical abilities through iterative directed practice supplemented by formative closed-loop feedback. This feasible, reproducible, and low-cost simulation curriculum can be adapted to other training programs and skills acquisition endeavors. This program also prepares trainees for ongoing performance feedback after completion of residency through rigorous self-assessment and peer-to-peer coaching.
Collapse
Affiliation(s)
- Brendan P Lovasik
- Department of Surgery, Emory University School of Medicine, Atlanta, GA. https://twitter.com/blovasik
| | - Katherine T Fay
- Department of Surgery, Emory University School of Medicine, Atlanta, GA
| | - Ankit Patel
- Department of Surgery, Emory University School of Medicine, Atlanta, GA. https://twitter.com/ankitpatelmd
| | - Jamil Stetler
- Department of Surgery, Emory University School of Medicine, Atlanta, GA
| | - Dominic Papandria
- Department of Surgery, Emory University School of Medicine, Atlanta, GA
| | - Matthew T Santore
- Department of Surgery, Emory University School of Medicine, Atlanta, GA
| | - S Scott Davis
- Department of Surgery, Emory University School of Medicine, Atlanta, GA
| | - Edward Lin
- Department of Surgery, Emory University School of Medicine, Atlanta, GA. https://twitter.com/edlinemory
| | | |
Collapse
|
18
|
Video-Based Coaching: Current Status and Role in Surgical Practice (Part 1) From the Society for Surgery of the Alimentary Tract, Health Care Quality and Outcomes Committee. J Gastrointest Surg 2021; 25:2439-2446. [PMID: 34355331 DOI: 10.1007/s11605-021-05102-1] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/14/2021] [Accepted: 07/21/2021] [Indexed: 01/31/2023]
Abstract
Patient safety and outcomes are directly related to surgical performance. Surgical training emphasizes the importance of the surgeon in determining these outcomes. After training is complete, there is a lack of structured programs for surgeons to audit their skills and continue their individual development. There is a significant linear relationship between surgeon technical skill and surgical outcomes; however, measuring technical performance is difficult. Video-based coaching matches an individual surgeon in practice with a surgical colleague who has been trained in the core principles of coaching for individualizing instruction. It can provide objective assessment for teaching higher-level concepts, such as technical skills, cognitive skills, and decision-making. There are many benefits to video-based coaching. While the concept is gaining acceptance as a method of surgical education, it is still novel in clinical practice. As more surgeons look towards video-based coaching for quality improvement, a consistent definition of the program, goals, and metrics for assessment will be critical. This paper is a review on the status of the video-based coaching as it applies to practicing surgeons.
Collapse
|
19
|
Rai A, Beland L, Aro T, Jarrett M, Kavoussi L. Patient Safety in the Operating Room During Urologic Surgery: The OR Black Box Experience. World J Surg 2021; 45:3306-3312. [PMID: 34351487 DOI: 10.1007/s00268-021-06251-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 07/03/2021] [Indexed: 10/20/2022]
Abstract
PURPOSE To mitigate intraoperative adverse events, it is important to understand the context in which these errors occur. The purpose of this study is to characterize the IAEs and potential distractions that occur in minimally invasive urologic procedures. METHODS We conducted a prospective cohort study in patients undergoing laparoscopic urologic surgery at an academic health center. The OR Black Box, a unique technology system which captures video and audio recordings of the operating room as well as the operative field, was used to collect data regarding procedure type, critical step, IAEs, and distractions. RESULTS Of a total of 80 cases analyzed, the majority of these cases were partial nephrectomy (n = 36; 45%), radical nephrectomy (n = 20; 25%), and adrenalectomy (n = 4; 5%). Across all cases, there were a total of 138 clinically significant IAEs, 10 of which (14%) were of the highest severity (five on the SEVerity of intraoperative Events and Rectification Tool (SEVERE) matrix). Of these, 70 (51%) occurred during an a priori defined critical step of the operation. Distractions were common across all cases. The median rate of external communication per case was 16 events (IQR 11-22); and per critical step was 4 (IQR 2.75-8), while median room traffic per case was 65 entries/exits (IQR 42-76); and per critical step was 17 (IQR 10-65). CONCLUSION Our data demonstrate that IAEs occur frequently during all phases of the operation at hand. Future study will be required to examine the role of distractions and IAE as well as IAE and their relationship to post-operative clinical outcomes.
Collapse
Affiliation(s)
- A Rai
- Smith Institute for Urology, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, 450 Lakeville Road, New Hyde Park, Hempstead, NY, 11042, USA.
| | - L Beland
- Smith Institute for Urology, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, 450 Lakeville Road, New Hyde Park, Hempstead, NY, 11042, USA
| | - T Aro
- Smith Institute for Urology, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, 450 Lakeville Road, New Hyde Park, Hempstead, NY, 11042, USA
| | - M Jarrett
- Department of Medicine, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, New Hyde Park, Hempstead, NY, 11042, USA
| | - L Kavoussi
- Smith Institute for Urology, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Northwell Health, 450 Lakeville Road, New Hyde Park, Hempstead, NY, 11042, USA
| |
Collapse
|
20
|
Virtual Education in Pediatric Surgery during the COVID-19 Era: Facing and Overcoming Current Challenges. Eur J Pediatr Surg 2021; 31:319-325. [PMID: 34176106 DOI: 10.1055/s-0041-1731297] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
Abstract
The novel coronavirus disease 2019 (COVID-19) pandemic has impacted our way of living in an unprecedented manner. Medical professionals at all levels have been forced to adapt to the novel virus. The delivery of surgical services and the subsequent learning opportunities for surgical residents have especially been disrupted and the pediatric surgical community has not been exempted by this. This article highlights the challenges imposed by the pandemic and outlines the various learning modalities that can be implemented to ensure continued learning opportunities throughout the pandemic and beyond. Furthermore, it aims to show how the utilization and expansion of technologies maintain and further increase the communication, as well as the exchange of and access to knowledge among peers. Virtual education-, application-, and simulation-based learning and social media, as well as telemedicine and online conferences, will play a considerable role in the future of surgical specialties and surgical education.
Collapse
|
21
|
Paley GL, Grove R, Sekhar TC, Pruett J, Stock MV, Pira TN, Shields SM, Waxman EL, Wilson BS, Gordon MO, Culican SM. Crowdsourced Assessment of Surgical Skill Proficiency in Cataract Surgery. JOURNAL OF SURGICAL EDUCATION 2021; 78:1077-1088. [PMID: 33640326 PMCID: PMC8217126 DOI: 10.1016/j.jsurg.2021.02.004] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/02/2021] [Accepted: 02/13/2021] [Indexed: 05/09/2023]
Abstract
OBJECTIVE To test whether crowdsourced lay raters can accurately assess cataract surgical skills. DESIGN Two-armed study: independent cross-sectional and longitudinal cohorts. SETTING Washington University Department of Ophthalmology. PARTICIPANTS AND METHODS Sixteen cataract surgeons with varying experience levels submitted cataract surgery videos to be graded by 5 experts and 300+ crowdworkers masked to surgeon experience. Cross-sectional study: 50 videos from surgeons ranging from first-year resident to attending physician, pooled by years of training. Longitudinal study: 28 videos obtained at regular intervals as residents progressed through 180 cases. Surgical skill was graded using the modified Objective Structured Assessment of Technical Skill (mOSATS). Main outcome measures were overall technical performance, reliability indices, and correlation between expert and crowd mean scores. RESULTS Experts demonstrated high interrater reliability and accurately predicted training level, establishing construct validity for the modified OSATS. Crowd scores were correlated with (r = 0.865, p < 0.0001) but consistently higher than expert scores for first, second, and third-year residents (p < 0.0001, paired t-test). Longer surgery duration negatively correlated with training level (r = -0.855, p < 0.0001) and expert score (r = -0.927, p < 0.0001). The longitudinal dataset reproduced cross-sectional study findings for crowd and expert comparisons. A regression equation transforming crowd score plus video length into expert score was derived from the cross-sectional dataset (r2 = 0.92) and demonstrated excellent predictive modeling when applied to the independent longitudinal dataset (r2 = 0.80). A group of student raters who had edited the cataract videos also graded them, producing scores that more closely approximated experts than the crowd. CONCLUSIONS Crowdsourced rankings correlated with expert scores, but were not equivalent; crowd scores overestimated technical competency, especially for novice surgeons. A novel approach of adjusting crowd scores with surgery duration generated a more accurate predictive model for surgical skill. More studies are needed before crowdsourcing can be reliably used for assessing surgical proficiency.
Collapse
Affiliation(s)
- Grace L Paley
- Department of Ophthalmology and Visual Sciences, Washington University School of Medicine, Saint Louis, Missouri
| | - Rebecca Grove
- Graduate Medical Education, University of Minnesota, Minneapolis, Minnesota
| | - Tejas C Sekhar
- Department of Ophthalmology and Visual Sciences, Washington University School of Medicine, Saint Louis, Missouri
| | - Jack Pruett
- Department of Ophthalmology and Visual Sciences, Washington University School of Medicine, Saint Louis, Missouri
| | - Michael V Stock
- Department of Ophthalmology and Visual Sciences, Washington University School of Medicine, Saint Louis, Missouri
| | - Tony N Pira
- Department of Ophthalmology, Boston University School of Medicine, Boston, Massachusetts
| | - Steven M Shields
- Department of Ophthalmology and Visual Sciences, Washington University School of Medicine, Saint Louis, Missouri
| | - Evan L Waxman
- Department of Ophthalmology, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania
| | - Bradley S Wilson
- Department of Ophthalmology and Visual Sciences, Washington University School of Medicine, Saint Louis, Missouri
| | - Mae O Gordon
- Department of Ophthalmology and Visual Sciences, Washington University School of Medicine, Saint Louis, Missouri
| | - Susan M Culican
- Department of Ophthalmology and Visual Sciences, Washington University School of Medicine, Saint Louis, Missouri; Graduate Medical Education, University of Minnesota, Minneapolis, Minnesota; Department of Ophthalmology and Visual Neurosciences, University of Minnesota, Minneapolis, Minnesota.
| |
Collapse
|
22
|
Fukuoka K, Teishima J, Inoue S, Hayashi T, Matsubara A. The influence of reviewer's occupation on the skill assessment of urethrovesical anastomosis in robot-assisted radical prostatectomy. Asian J Endosc Surg 2021; 14:451-457. [PMID: 33145920 DOI: 10.1111/ases.12892] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/04/2020] [Revised: 09/25/2020] [Accepted: 10/19/2020] [Indexed: 01/15/2023]
Abstract
INTRODUCTION In recent years, crowd-sourced assessments have been reported as a timesaving, cost-efficient, and practicable method of surgical skill evaluation. However, the differences in the assessment of surgical skills by the individual reviewers cannot be further examined in terms of characteristics of the reviewers because they are usually anonymously and randomly selected. This study aimed to reveal the effects of reviewers' occupations on their assessment of a surgeon's skill. METHODS In total, 42 urologists, 19 paramedics, 73 medical students, and 28 non-medical personnel used the Global Evaluative Assessment of Robotic Skills (GEARS) validated robotic surgery rating tool to assess the surgical skill of surgeons in nine edited video clips of complete urethrovesical anastomosis during a robot-assisted radical prostatectomy. The total GEARS scores of the four groups of reviewers were compared, and the similarities and the differences between the ratings of the urologists group and those of the other three groups were subsequently investigated. RESULTS The rankings of video clips in the order of GEARS scores were very similar in each group, and a strong positive correlation (R2 values >0.8) was observed between the scores assigned by the urologists group and those assigned by the other three groups. CONCLUSION Our findings indicate that the crude evaluation of robot-assisted urethrovesical anastomosis is not affected by the reviewers' occupations. Non-medical personnel may be able to provide a rudimentary screening evaluation of surgical skill.
Collapse
Affiliation(s)
- Kenichiro Fukuoka
- Department of Urology, Graduate School of Biomedical and Health Sciences, Hiroshima University, Hiroshima, Japan
| | - Jun Teishima
- Department of Urology, Graduate School of Biomedical and Health Sciences, Hiroshima University, Hiroshima, Japan
| | - Shogo Inoue
- Department of Urology, Graduate School of Biomedical and Health Sciences, Hiroshima University, Hiroshima, Japan
| | - Tetsutaro Hayashi
- Department of Urology, Graduate School of Biomedical and Health Sciences, Hiroshima University, Hiroshima, Japan
| | - Akio Matsubara
- Department of Urology, Hiroshima General Hospital, Hatsukaichi, Japan
| |
Collapse
|
23
|
The Use of Crowdsourcing Technology to Evaluate Preoperative Severity in Patients With Unilateral Cleft Lip in a Multiethnic Population. J Craniofac Surg 2021; 32:482-485. [PMID: 33704965 DOI: 10.1097/scs.0000000000006917] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022] Open
Abstract
ABSTRACT Crowd sourcing has been used in multiple disciplines to quickly generate large amounts of diverse data. The objective of this study was to use crowdsourcing to grade preoperative severity of unilateral cleft lip phenotype in a multiethnic cohort with the hypothesis that crowdsourcing could efficiently achieve similar rankings compared to expert surgeons. Deidentified preoperative photos were collected for patients with primary, unilateral cleft lip with or without cleft palate (CL ± P). A platform was developed with C-SATS for pairwise comparisons utilizing Elo rankings by crowdsource workers through Amazon Mechanical Turk. Images were independently ranked by 2 senior surgeons for comparison. Seventy-six patients with varying severity of unilateral (CL ± P) phenotype were chosen from Operation Smile missions in Bolivia, Madagascar, Vietnam, and Morocco. Patients were an average of 1.2 years' old, ranging from 3 months to 3.3 years. Each image was compared with 10 others, for a total of 380 unique pairwise comparisons. A total of 4627 total raters participated with a median of 12 raters per pair. Data collection was completed in <20 hours. The crowdsourcing ranking and expert surgeon rankings were highly correlated with Pearson correlation coefficient of R = 0.77 (P = 0.0001). Crowdsourcing provides a rapid and convenient method of obtaining preoperative severity ratings, comparable to expert surgeon assessment, across multiple ethnicities. The method serves as a potential solution to the current lack of rating systems for preoperative severity and overcomes the difficulty of acquiring large-scale assessment from expert surgeons.
Collapse
|
24
|
Ward TM, Fer DM, Ban Y, Rosman G, Meireles OR, Hashimoto DA. Challenges in surgical video annotation. Comput Assist Surg (Abingdon) 2021; 26:58-68. [PMID: 34126014 DOI: 10.1080/24699322.2021.1937320] [Citation(s) in RCA: 30] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022] Open
Abstract
Annotation of surgical video is important for establishing ground truth in surgical data science endeavors that involve computer vision. With the growth of the field over the last decade, several challenges have been identified in annotating spatial, temporal, and clinical elements of surgical video as well as challenges in selecting annotators. In reviewing current challenges, we provide suggestions on opportunities for improvement and possible next steps to enable translation of surgical data science efforts in surgical video analysis to clinical research and practice.
Collapse
Affiliation(s)
- Thomas M Ward
- Surgical AI & Innovation Laboratory, Department of Surgery, Massachusetts General Hospital, Boston, MA, USA
| | - Danyal M Fer
- Department of Surgery, University of California San Francisco East Bay, Hayward, CA, USA
| | - Yutong Ban
- Surgical AI & Innovation Laboratory, Department of Surgery, Massachusetts General Hospital, Boston, MA, USA.,Distributed Robotics Laboratory, Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Guy Rosman
- Surgical AI & Innovation Laboratory, Department of Surgery, Massachusetts General Hospital, Boston, MA, USA.,Distributed Robotics Laboratory, Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Ozanan R Meireles
- Surgical AI & Innovation Laboratory, Department of Surgery, Massachusetts General Hospital, Boston, MA, USA
| | - Daniel A Hashimoto
- Surgical AI & Innovation Laboratory, Department of Surgery, Massachusetts General Hospital, Boston, MA, USA
| |
Collapse
|
25
|
Haras C, Calhoun A, Olson APJ, Rosenberg M. Mindful Medical Education Online. MEDICAL SCIENCE EDUCATOR 2021; 31:863-872. [PMID: 33688449 PMCID: PMC7932687 DOI: 10.1007/s40670-021-01253-7] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Accepted: 02/16/2021] [Indexed: 06/12/2023]
Abstract
The COVID-19 pandemic disrupted medical education. In-person classes and clinical rotations were urgently canceled, followed by a historic and unprecedented migration to online teaching. Most medical school courses were not designed to be fully online, and faculty and students are novices in the process. The purpose of this article is to provide recommendations for educators to optimize their approach to online curricular transformation. Mindful teaching online creates presences that set climate and support discourse, establish routines that build practice, model professional expectations, and challenge but support learners.
Collapse
Affiliation(s)
- Catherine Haras
- California State University Los Angeles, California Los Angeles, USA
| | - Austin Calhoun
- University of Minnesota Medical School, Minnesota Minneapolis, USA
| | - Andrew PJ Olson
- University of Minnesota Medical School, Minnesota Minneapolis, USA
| | - Mark Rosenberg
- University of Minnesota Medical School, Minnesota Minneapolis, USA
| |
Collapse
|
26
|
Teotia SS, Alford JA, Kadakia Y, Haddock NT. Crowdsourced Assessment of Aesthetic Outcomes after Breast Reconstruction. Plast Reconstr Surg 2021; 147:570-577. [PMID: 33620921 DOI: 10.1097/prs.0000000000007637] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Abstract
BACKGROUND Evaluating the aesthetic success of breast reconstruction can be difficult. Patients, surgeons, and the general population may differ in what constitutes a successful outcome. Recently, crowdsourcing has emerged to accumulate and analyze data on a massive scale. The authors propose that crowdsourcing can be a useful tool to reliably rate aesthetic outcomes of breast reconstruction. METHODS One hundred one deidentified photographs of patients at various stages of breast reconstruction were gathered. Assessment tools included a five-point Likert scale and the transverse rectus abdominis myocutaneous (TRAM) visual assessment scale. Anonymous crowd workers and a group of expert reconstructive surgeons rated an identical set of photographs on the Likert scale. Crowd workers also rated the set of photographs on the TRAM scale. RESULTS The authors obtained 901 anonymous, layperson evaluations on both Likert and TRAM scales. Crowdsourced assessment data collection took 28.6 hours. Expert assessment took 15 months. Expert and crowdsourced scores were equivalent on the Likert scale (overall interrater reliability, κ = 0.99; 95 percent CI, 0.98 to 0.99). Intrarater reliability among each subcomponent was highly reproducible for the crowd (r = 0.98; 95 percent CI, 0.97 to 0.99) and experts (r = 0.82; 95 percent CI, 0.77 to 0.87). Breast contour and positioning were most predictive of overall aesthetic result. Skin patch and scar were least predictive of overall aesthetic appearance. CONCLUSIONS Aesthetic outcomes rated by crowds were reliable and correlated closely with those by expert surgeons. Crowdsourcing can be a rapid, reliable, and valid way to assess aesthetic outcomes in the breast reconstruction patient.
Collapse
Affiliation(s)
- Sumeet S Teotia
- From the Department of Plastic Surgery, University of Texas, Southwestern Medical Center
| | - Jake A Alford
- From the Department of Plastic Surgery, University of Texas, Southwestern Medical Center
| | - Yash Kadakia
- From the Department of Plastic Surgery, University of Texas, Southwestern Medical Center
| | - Nicholas T Haddock
- From the Department of Plastic Surgery, University of Texas, Southwestern Medical Center
| |
Collapse
|
27
|
Alnafisee N, Zafar S, Vedula SS, Sikder S. Current methods for assessing technical skill in cataract surgery. J Cataract Refract Surg 2021; 47:256-264. [PMID: 32675650 DOI: 10.1097/j.jcrs.0000000000000322] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/05/2020] [Accepted: 06/19/2020] [Indexed: 12/18/2022]
Abstract
Surgery is a major source of errors in patient care. Preventing complications from surgical errors in the operating room is estimated to lead to reduction of up to 41 846 readmissions and save $620.3 million per year. It is now established that poor technical skill is associated with an increased risk of severe adverse events postoperatively and traditional models to train surgeons are being challenged by rapid advances in technology, an intensified patient-safety culture, and a need for value-driven health systems. This review discusses the current methods available for evaluating technical skills in cataract surgery and the recent technological advancements that have enabled capture and analysis of large amounts of complex surgical data for more automated objective skills assessment.
Collapse
Affiliation(s)
- Nouf Alnafisee
- From the The Wilmer Eye Institute, Johns Hopkins University School of Medicine (Alnafisee, Zafar, Sikder), Baltimore, and the Department of Computer Science, Malone Center for Engineering in Healthcare, The Johns Hopkins University Whiting School of Engineering (Vedula), Baltimore, Maryland, USA
| | | | | | | |
Collapse
|
28
|
Jogerst KM, Eurboonyanun C, Park YS, Cassidy D, McKinley SK, Hamdi I, Phitayakorn R, Petrusa E, Gee DW. Implementation of the ACS/ APDS Resident Skills Curriculum reveals a need for rater training: An analysis using generalizability theory. Am J Surg 2021; 222:541-548. [PMID: 33516415 DOI: 10.1016/j.amjsurg.2021.01.018] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2020] [Revised: 12/16/2020] [Accepted: 01/11/2021] [Indexed: 11/25/2022]
Abstract
BACKGROUND The American College of Surgeons (ACS)/Association of Program Directors in Surgery (APDS) Resident Skills Curriculum includes validated task-specific checklists and global rating scales (GRS) for Objective Structured Assessment of Technical Skills (OSATS). However, it does not include instructions on use of these assessment tools. Since consistency of ratings is a key feature of assessment, we explored rater reliability for two skills. METHODS Surgical faculty assessed hand-sewn bowel and vascular anastomoses in real-time using the OSATS GRS. OSATS were video-taped and independently evaluated by a research resident and surgical attending. Rating consistency was estimated using intraclass correlation coefficients (ICC) and generalizability analysis. RESULTS Three-rater ICC coefficients across 24 videos ranged from 0.12 to 0.75. Generalizability reliability coefficients ranged from 0.55 to 0.8. Percent variance attributable to raters ranged from 2.7% to 32.1%. Pairwise agreement showed considerable inconsistency for both tasks. CONCLUSIONS Variability of ratings for these two skills indicate the need for rater training to increase scoring agreement and decrease rater variability for technical skill assessments.
Collapse
Affiliation(s)
- Kristen M Jogerst
- Department of General Surgery, Mayo Clinic Arizona, 5777 E. Mayo Blvd, Phoenix, AZ, 85054, USA; Department of Surgery, Massachusetts General Hospital, 55 Fruit St, Boston, MA, 02114, USA.
| | - Chalerm Eurboonyanun
- Department of Surgery, Massachusetts General Hospital, 55 Fruit St, Boston, MA, 02114, USA; Department of Surgery, Khon Kaen University, 123 Tambon Sila, Mueang Khon Kaen District, Khon Kaen 40002, Thailand.
| | - Yoon Soo Park
- Department of Emergency Medicine, Massachusetts General Hospital, 55 Fruit St, Boston, MA, 02114, USA.
| | - Douglas Cassidy
- Department of Surgery, Massachusetts General Hospital, 55 Fruit St, Boston, MA, 02114, USA.
| | - Sophia K McKinley
- Department of Surgery, Massachusetts General Hospital, 55 Fruit St, Boston, MA, 02114, USA.
| | - Isra Hamdi
- Department of Surgery, Massachusetts General Hospital, 55 Fruit St, Boston, MA, 02114, USA.
| | - Roy Phitayakorn
- Department of Surgery, Massachusetts General Hospital, 55 Fruit St, Boston, MA, 02114, USA.
| | - Emil Petrusa
- Department of Surgery, Massachusetts General Hospital, 55 Fruit St, Boston, MA, 02114, USA.
| | - Denise W Gee
- Department of Surgery, Massachusetts General Hospital, 55 Fruit St, Boston, MA, 02114, USA.
| |
Collapse
|
29
|
Azadi S, Green IC, Arnold A, Truong M, Potts J, Martino MA. Robotic Surgery: The Impact of Simulation and Other Innovative Platforms on Performance and Training. J Minim Invasive Gynecol 2020; 28:490-495. [PMID: 33310145 DOI: 10.1016/j.jmig.2020.12.001] [Citation(s) in RCA: 33] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/27/2020] [Revised: 11/25/2020] [Accepted: 12/02/2020] [Indexed: 01/30/2023]
Abstract
OBJECTIVE To review the current status of robotic training and the impact of various training platforms on the performance of robotic surgical trainees. DATA SOURCES Literature review of Google Scholar and PubMed. The search terms included a combination of the following: "robotic training," "simulation," "robotic curriculum," "obgyn residency robotic training," "virtual reality robotic training," "DaVinci training," "surgical simulation," "gyn surgical training." The sources considered for inclusion included peer-reviewed articles, literature reviews, textbook chapters, and statements from various institutions involved in resident training. METHODS OF STUDY SELECTION A literature search of Google Scholar and PubMed using terms related to robotic surgery and robotics training, as mentioned in the "Data Sources" section. RESULTS Multiple novel platforms that use machine learning and real-time video feedback to teach and evaluate robotic surgical skills have been developed over recent years. Various training curricula, virtual reality simulators, and other robotic training tools have been shown to enhance robotic surgical education and improve surgical skills. The integration of didactic learning, simulation, and intraoperative teaching into more comprehensive training curricula shows positive effects on robotic skills proficiency. Few robotic surgery training curricula have been validated through peer-reviewed study, and there is more work to be completed in this area. In addition, there is a lack of information about how the skills obtained through robotics curricula and simulation translate into operating room performance and patient outcomes. CONCLUSION Data collected to date show promising advances in the training of robotic surgeons. A diverse array of curricula for training robotic surgeons continue to emerge, and existing teaching modalities are evolving to keep up with the rapidly growing demand for proficient robotic surgeons. Futures areas of growth include establishing competency benchmarks for existing training tools, validating existing curricula, and determining how to translate the acquired skills in simulation into performance in the operating room and patient outcomes. Many surgical training platforms are beginning to expand beyond discrete robotic skills training to procedure-specific and team training. There is still a wealth of research to be done to understand how to create an effective training experience for gynecologic surgical trainees and robotics teams.
Collapse
Affiliation(s)
- Shirin Azadi
- Department of Obstetrics and Gynecology, Lehigh Valley Health Network, Allentown, Pennsylvania (Drs. Azadi, Potts, and Martino)
| | - Isabel C Green
- Department of Gynecology and Obstetric, Mayo Clinic, Rochester, Minnesota (Dr. Green)
| | - Anne Arnold
- American College of Obstetricians and Gynecologists, University of Pennsylvania Graduate School of Education, Philadelphia, PA (Ms. Arnold)
| | - Mireille Truong
- Division of Minimally Invasive Gynecologic Surgery, Department of Obstetrics and Gynecology, Cedars-Sinai Medical Center, Los Angeles, California (Dr. Truong)
| | - Jacqueline Potts
- Department of Obstetrics and Gynecology, Lehigh Valley Health Network, Allentown, Pennsylvania (Drs. Azadi, Potts, and Martino)
| | - Martin A Martino
- Department of Obstetrics and Gynecology, Lehigh Valley Health Network, Allentown, Pennsylvania (Drs. Azadi, Potts, and Martino); Department of Obstetrics and Gynecology, University of South Florida, Tampa, Florida (Dr. Martino).
| |
Collapse
|
30
|
Grall P, Ferri J, Nicot R. Surgical Training 2.0: A systematic approach reviewing the literature focusing on oral maxillofacial surgery - Part II. JOURNAL OF STOMATOLOGY, ORAL AND MAXILLOFACIAL SURGERY 2020; 122:423-433. [PMID: 33301948 DOI: 10.1016/j.jormas.2020.11.010] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/18/2020] [Accepted: 11/25/2020] [Indexed: 11/18/2022]
Abstract
PURPOSE Many technologies are emerging in the medical field. Having an overview of the technological arsenal available to train new surgeons seems very interesting to guide subsequent surgical training protocols. METHODS This article is a systematic approach reviewing new technologies in surgical training, in particular in oral and maxillofacial surgery. This review explores what new technologies can do compared to traditional methods in the field of surgical education. A structured literature search of PubMed was performed in adherence to PRISMA guidelines. The articles were selected when they fell within predefined inclusion criteria while respecting the key objectives of this systematic review. We looked at medical students and more specifically in surgery and analysed whether exposure to new technologies improved their surgical skills compared to traditional methods. Each technology is reviewed by highlighting its advantages and disadvantages and studying the feasibility of integration into current practice. RESULTS The results are encouraging. Indeed, all of these technologies make it possible to reduce the learning time, the operating times, the operating complications and increase the enthusiasm of the students compared to more conventional methods. The start-up cost, the complexity to develop new models, and the openness of mind necessary for the integration of these technologies are all obstacles to immediate development. The main limitations of this review are that many of the studies have been carried out on small numbers, they are not interested in acquiring knowledge or skills over the long term and obviously there is a publication bias. CONCLUSION Surgical education methods will probably change in the years to come, integrating these new technologies into the curriculum seems essential so as not to remain on the side. This second part therefore reviews, social networks, serious games and virtual reality. This Systematic review is registered on PROSPERO (CRD42020181376).
Collapse
Affiliation(s)
- Patrick Grall
- Univ. Lille, CHU Lille, Department of Oral and Maxillofacial Surgery, F-59000 Lille, France.
| | - Joël Ferri
- Univ. Lille, CHU Lille, INSERM, Department of Oral and Maxillofacial Surgery, U1008 - Controlled Drug Delivery Systems and Biomaterials, F-59000 Lille, France.
| | - Romain Nicot
- Univ. Lille, CHU Lille, INSERM, Department of Oral and Maxillofacial Surgery, U1008 - Controlled Drug Delivery Systems and Biomaterials, F-59000 Lille, France.
| |
Collapse
|
31
|
Abstract
The use of robotic surgery in urology has grown exponentially in the past 2 decades, but robotic surgery training has lagged behind. Most graduating residents report a lack of comfort independently performing common robotic urologic surgeries, despite an abundance of available resources. There is a general consensus on the key components of a comprehensive robotics curriculum, and well-validated tools have been developed to assess trainee competency. However, no single curriculum has emerged as the gold standard on which individual programs can build their own robotics curricula.
Collapse
Affiliation(s)
- Robert S Wang
- Department of Urology, Michigan Medicine, 1500 East Medical Center Drive, TC 3875, Ann Arbor, MI 48109, USA
| | - Sapan N Ambani
- Department of Urology, Michigan Medicine, 1500 East Medical Center Drive, TC 3875, Ann Arbor, MI 48109, USA.
| |
Collapse
|
32
|
Deal SB, Scully RE, Wnuk G, George BC, Alseidi AA. Crowd-Sourced and Attending Assessment of General Surgery Resident Operative Performance Using Global Ratings Scales. JOURNAL OF SURGICAL EDUCATION 2020; 77:e214-e219. [PMID: 33041252 DOI: 10.1016/j.jsurg.2020.07.011] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/22/2020] [Revised: 06/08/2020] [Accepted: 07/12/2020] [Indexed: 06/11/2023]
Abstract
OBJECTIVE We sought to assess the extent to which both crowd and intraoperative attending ratings using objective structured assessment of technical skill (OSATS) or global objective assessment of laparoscopic skills (GOALS) would correlate with the system for improving procedural learning (SIMPL) Zwisch and Performance scales. DESIGN Comparison of directly observed versus crowd sourced review of operative video. SETTING Operative video captured at 2 institutions. PARTICIPANTS Six (6) core general surgery procedures, 3 open and 3 laparoscopic, were selected from the American Board of Surgery's Resident Assessments list. Thirty-two cases performed by General Surgery residents across all training levels at 2 institutions were filmed. Videos were condensed using a standardized protocol to include the critical portion of the procedure. Condensed videos were then submitted to crowd-sourced assessment of technical skills (C-SATS), an online crowd source-driven assessment service, for assessment using the appropriate resident assessment form (GOALS or OSATS) as well as with the SIMPL Zwisch and Performance scales. Crowd workers watched an educational tutorial on how to use the Zwisch and SIMPL Performance rating scales prior to participating. Attendings scored residents using the same tools immediately after the shared operative experience. Statistical analysis was performed using Pearson's correlation coefficient. RESULTS Crowd raters evaluated 32 procedures using GOALS/OSATS, Zwisch and Performance (35-50 ratings per video). Attendings also evaluated all 32 procedures using GOALS/OSATS and 26 of the procedures using SIMPL Zwisch and Performance. Pearson correlation coefficients with 95% confidence intervals for crowd ratings were: GOALS and Zwisch -0.40 [-0.73 to 0.10], OSATS and Zwisch 0.11 [-0.41 to 0.57], GOALS and Performance -0.06 [-0.44 to 0.35], and OSATS and Performance 0.22 [-0.46 to 0.20]. Pearson correlation coefficients for attendings were: GOALS and Zwisch (0.77), OSATS and Zwisch (0.65), GOALS and Performance (0.93), and OSATS and Performance (0.59). CONCLUSIONS Overall, correlations between crowd-sourced ratings using GOALS/OSATS and SIMPL global operative performance ratings tools were weak, yet for attendings, they were strong. Direct attending assessment may be required for evaluation of global performance while crowd sourcing may be more suitable for technical assessment. Further studies are needed to see if more extensive crowd training would result in improved ability for global performance evaluation.
Collapse
Affiliation(s)
| | | | - Gregory Wnuk
- Consulting for Statistics, Computing, and Analytics, University of Michigan, Ann Arbor, Michigan
| | - Brian C George
- Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, Michigan
| | - Adnan A Alseidi
- University of California San Francisco, Department of Surgery, San Francisco, California
| |
Collapse
|
33
|
Bidirectional long short-term memory for surgical skill classification of temporally segmented tasks. Int J Comput Assist Radiol Surg 2020; 15:2079-2088. [PMID: 33000365 DOI: 10.1007/s11548-020-02269-x] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/13/2020] [Accepted: 09/23/2020] [Indexed: 10/23/2022]
Abstract
PURPOSE The majority of historical surgical skill research typically analyzes holistic summary task-level metrics to create a skill classification for a performance. Recent advances in machine learning allow time series classification at the sub-task level, allowing predictions on segments of tasks, which could improve task-level technical skill assessment. METHODS A bidirectional long short-term memory (LSTM) network was used with 8-s windows of multidimensional time-series data from the Basic Laparoscopic Urologic Skills dataset. The network was trained on experts and novices from four common surgical tasks. Stratified cross-validation with regularization was used to avoid overfitting. The misclassified cases were re-submitted for surgical technical skill assessment to crowds using Amazon Mechanical Turk to re-evaluate and to analyze the level of agreement with previous scores. RESULTS Performance was best for the suturing task, with 96.88% accuracy at predicting whether a performance was an expert or novice, with 1 misclassification, when compared to previously obtained crowd evaluations. When compared with expert surgeon ratings, the LSTM predictions resulted in a Spearman coefficient of 0.89 for suturing tasks. When crowds re-evaluated misclassified performances, it was found that for all 5 misclassified cases from peg transfer and suturing tasks, the crowds agreed more with our LSTM model than with the previously obtained crowd scores. CONCLUSION The technique presented shows results not incomparable with labels which would be obtained from crowd-sourced labels of surgical tasks. However, these results bring about questions of the reliability of crowd sourced labels in videos of surgical tasks. We, as a research community, should take a closer look at crowd labeling with higher scrutiny, systematically look at biases, and quantify label noise.
Collapse
|
34
|
Correlation between operative time and crowd-sourced skills assessment for robotic bariatric surgery. Surg Endosc 2020; 35:5303-5309. [PMID: 32970207 DOI: 10.1007/s00464-020-08019-z] [Citation(s) in RCA: 13] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/13/2020] [Accepted: 09/16/2020] [Indexed: 12/16/2022]
Abstract
INTRODUCTION Operative time has been traditionally used as a proxy for surgical skill and is commonly utilized to measure the learning curve, assuming that faster operations indicate a more skilled surgeon. The Global Evaluative Assessment of Robotic Skills (GEARS) rubric is a validated Likert scale for evaluating technical skill. We hypothesize that operative time will not correlate with the GEARS score. METHODS Patients undergoing elective robotic sleeve gastrectomy at a single bariatric center of excellence hospital from January 2019 to March 2020 were captured in a prospectively maintained database. For step-specific scoring, videos were broken down into three steps: ligation of short gastric vessels, gastric transection, and oversewing the staple line. Overall and step-specific GEARS scores were assigned by crowd-sourced evaluators. Correlation between operative time and GEARS score was assessed with linear regression and calculation of the R2 statistic. RESULTS Sixty-eight patients were included in the study, with a mean operative time of 112 ± 27.4 min. The mean GEARS score was 20.1 ± 0.81. Mean scores for the GEARS subcomponents were: bimanual dexterity 4.06 ± 0.17; depth perception 3.96 ± 0.24; efficiency 3.82 ± 0.19; force sensitivity 4.06 ± 0.20; robotic control 4.16 ± 0.21. Operative time and overall score showed no correlation (R2 = 0.0146, p = 0.326). Step-specific times and scores showed weak correlation for gastric transection (R2 = 0.0737, p = 0.028) and no correlation for ligation of short gastric vessels (R2 = 0.0262, p = 0.209) or oversewing the staple line (R2 = 0.0142, p = 0.344). CONCLUSIONS Operative time and crowd-sourced GEARS score were not correlated. Operative time and GEARS scores measure different performance characteristics, and future studies should consider using both a validated skills assessment tool and operative time for a more complete evaluation of skill.
Collapse
|
35
|
Richetelli N, Hammer L, Speir JA. Forensic Footwear Reliability: Part II-Range of Conclusions, Accuracy, and Consensus. J Forensic Sci 2020; 65:1871-1882. [PMID: 32940930 DOI: 10.1111/1556-4029.14551] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2020] [Revised: 06/15/2020] [Accepted: 07/31/2020] [Indexed: 12/01/2022]
Abstract
Between February 2017 and August 2018, West Virginia University conducted a reliability study to determine expert performance among forensic footwear examiners in the United States. Throughout the study's duration, 70 examiners each performed 12 comparisons and reported a total of 840 conclusions. In order to assess the accuracy of conclusions, the similarities and differences between mated and nonmated pairs were evaluated according to three criteria: (i) inherent agreement/disagreement in class, wear, and randomly acquired features, (ii) limitations as a function of questioned impression quality, clarity, and totality, and (iii) adherence to the Scientific Working Group for Shoeprint and Tire Tread Evidence (SWGTREAD) 2013 conclusion standard. Using these criteria, acceptable/expected categorical conclusions were defined. Preliminary results from this study are divided into a series of three summaries. This manuscript (Part II) reports accuracy and reproducibility. For mated pairs, accuracy equals 76.3% ± 13.0% (median of 78.6% and a 90% confidence interval between 72.2% and 80.0%). For nonmated pairs, accuracy equals 87.4% ± 9.24% (median of 91.4% and a 90% confidence interval between 84.7% and 89.8%). In addition, the community assessed agreement (denoted by IQR) of reported results equals the research team's accepted/expected conclusions for 10 out of 12 comparisons. In terms of reproducibility, the 90% confidence interval for consensus was computed and found to equal 0.71-0.86 (median of 0.77) for the combined dataset. Although based on a limited sample size, these results provide a baseline estimate of accuracy and consensus/reproducibility as a function of the existing seven-point SWGTREAD 2013 conclusion standard.
Collapse
Affiliation(s)
- Nicole Richetelli
- West Virginia University, 208 Oglebay Hall, PO Box 6121, Morgantown, WV, 26506
| | - Lesley Hammer
- Hammer Forensics, LLC, 10601 Prospect Drive, Anchorage, AK, 99507
| | - Jacqueline A Speir
- West Virginia University, 208 Oglebay Hall, PO Box 6121, Morgantown, WV, 26506
| |
Collapse
|
36
|
Abstract
Orthopaedic residency training has and will continue to evolve with a wide variety of changes. Hands-on surgical simulation "boot camps," computerized simulation of surgical process, and even virtual reality simulators, all can help trainees acquire surgical experience without compromising patient care. Low-cost training modules help remedy the rising costs associated with teaching complex orthopaedic surgery skills. Motion tracking and checklists help refine standardization of assessment. As technology and healthcare systems continue to grow, we encourage training programs to keep pace by considering engagement of these tools.
Collapse
|
37
|
Kelly JD, Nash M, Heller N, Lendvay TS, Kowalewski TM. Temporal variability of surgical technical skill perception in real robotic surgery. Int J Comput Assist Radiol Surg 2020; 15:2101-2107. [PMID: 32860549 DOI: 10.1007/s11548-020-02253-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2020] [Accepted: 08/19/2020] [Indexed: 11/26/2022]
Abstract
PURPOSE Summary score metrics, either from crowds of non-experts, faculty surgeons or from automated performance metrics, have been trusted as the prevailing method of reporting surgeon technical skill. The aim of this paper is to learn whether there exist significant fluctuations in the technical skill assessments of a surgeon throughout long durations of surgical footage. METHODS A set of 12 videos of robotic surgery cases from common human patient robotic surgeries were used to evaluate the perceived technical skill at each individual minute of the surgical videos, which were originally 12-15 min in length. A linear mixed-effects model for each video was used to compare the ratings of each minute to those from every other minute in order to learn whether a change in scores over time can be detected and reliably measured apart from inter- and intrarater variation. RESULTS Modeling the change over time of the global evaluative assessment of robotic skills scores significantly contributed to the prediction models for 11 of the 12 surgeons. This demonstrates that measurable changes in technical skill occur over time during robotic surgery. CONCLUSION The findings from this research raise questions about the optimal duration of footage needed to be evaluated to arrive at an accurate rating of surgical technical skill for longer procedures. This may imply non-negligible label noise for supervised machine learning approaches. In the future, it may be necessary to report a surgeon's skill variability in addition to their mean score to have proper knowledge of a surgeon's overall skill level.
Collapse
Affiliation(s)
- Jason D Kelly
- Department of Mechanical Engineering, University of Minnesota, Minneapolis, MN, USA.
| | - Michael Nash
- Department of Biostatistics, University of Washington, Seattle, WA, USA
| | - Nicholas Heller
- Department of Computer Science and Engineering, University of Minnesota, Minneapolis, MN, USA
| | - Thomas S Lendvay
- Department of Urology, University of Washington, Seattle, WA, USA
| | - Timothy M Kowalewski
- Department of Mechanical Engineering, University of Minnesota, Minneapolis, MN, USA
| |
Collapse
|
38
|
Affiliation(s)
- Elif Bilgic
- Department of Surgery, Division of Surgical Education, McGill University, McGill University Health Centre, 1650 Cedar Avenue, #D6.136, Montreal, Quebec H3G 1A4, Canada
| | - Sofia Valanci-Aroesty
- Department of Surgery, Division of Experimental Surgery, McGill University, McGill University Health Centre, 1650 Cedar Avenue, #D6.136, Montreal, Quebec H3G 1A4, Canada
| | - Gerald M Fried
- Department of Surgery, McGill University, McGill University Health Centre, 1650 Cedar Avenue, #D6.136, Montreal, Quebec H3G 1A4, Canada.
| |
Collapse
|
39
|
St John-Matthews J, Robinson L, Martin F, Newton PM, Grant AJ. Crowdsourcing: A novel tool to elicit the student voice in the curriculum design process for an undergraduate diagnostic radiography degree programme. Radiography (Lond) 2020; 26 Suppl 2:S54-S61. [PMID: 32507591 DOI: 10.1016/j.radi.2020.04.019] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/20/2020] [Revised: 04/13/2020] [Accepted: 04/30/2020] [Indexed: 11/26/2022]
Abstract
INTRODUCTION Stakeholder participation in healthcare curriculum design is an important aspect of higher education with stakeholders including students, staff members, clinical partners, healthcare organisations, patients and members of the public. Significantly, student co-creation, of the curriculum, has become increasingly important. Yet there is limited research which addresses how to engage this group in design processes. METHODS This paper represents the first phase of a three stage action research spiral whereby the authors evaluated the use of a novel tool for curriculum design processes, anonymised crowdsourcing. This initial phase was open to all students enrolled on an undergraduate diagnostic radiography programme in the UK. To confirm the reliability of the crowdsource design an established eight point crowdsourcing verification tool was applied. RESULTS Twenty-three unique ideas were generated by participants, 40 comments made and 173 votes cast. Inductive analysis of the comments generated five themes. These included: the role of technology enhanced learning; simulation activities; patient focused curriculum; mental wealth (resilience) authentic assessment approaches. An evaluation of those who had and had not engaged highlighted areas of improvement for the administration of the second and third iterations which will include a wider pool of participants. CONCLUSION This study from a single programme offers lessons for others wishing to adopt and develop this approach elsewhere. IMPLICATIONS FOR PRACTICE Several ideas elicited by the crowdsource have been considered by the curriculum design team and will be implemented in the 2020 curriculum thus demonstrating the impact on local education practice of this research approach.
Collapse
Affiliation(s)
- J St John-Matthews
- Department of Allied Health Professions, Faculty of Health and Applied Sciences, University of the West of England, Bristol, BS16 1DD, UK.
| | - L Robinson
- Research in Health Professions Education, Swansea University Medical School, Swansea, Wales, SA2 8PP, UK
| | | | - P M Newton
- Research in Health Professions Education, Swansea University Medical School, Swansea, Wales, SA2 8PP, UK
| | - A J Grant
- Research in Health Professions Education, Swansea University Medical School, Swansea, Wales, SA2 8PP, UK
| |
Collapse
|
40
|
Chau JK, Bilgic E, Hada T, Trepanier M, Naghawi H, Kaneva P, Mueller C, Lee L. Development and validation of a transanal endoscopic rectal purse string simulator. Tech Coloproctol 2020; 24:863-871. [PMID: 32504373 DOI: 10.1007/s10151-020-02251-4] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/04/2019] [Accepted: 05/25/2020] [Indexed: 01/02/2023]
Abstract
BACKGROUND Rectal purse string placement for transanal mesorectal excision is challenging, and practice is difficult. The objective of this study is to build an endoscopic rectal purse string simulator and provide evidence for the validity of its use. METHODS A low-cost transanal endoscopic rectal purse string simulator was created and used to measure the performance of participants. Participants included general surgery residents, fellows, and staff surgeons from several Canadian university-affiliated institutions across Canada. The performance of the rectal purse string placement was measured by the time to completion of the task as well scoring with a modified objective structured assessment of technical skills (OSATS) score and a modified advanced laparoscopic suturing (ALS) score. RESULTS Thirty-nine participants were recruited into the study. Participants were split into three groups, based on prior experience with laparoscopic suturing, for the analysis of suturing a rectal purse string on the simulator based on three performance measures. There was a significant difference found in all three measures of performance in the three groups (time to completion p = 0.014, mean blinded OSATS score p = 0.007, mean blinded ALS score p = 0.020). Participants with previous laparoscopic suturing had significantly faster times to completion and higher skills scores when compared to residents (time: 5.1 ± 1.4 min vs 9.0 ± 4.8 min, p = 0.005; OSATS: 19.7 ± 2.8 vs 13.0 ± 5.8, p = 0.00398, ALS: 27.4 ± 4.0 vs 18.9 ± 8.5, p = 0.0151). CONCLUSIONS A transanal endoscopic purse string simulator was constructed and preliminary testing has shown variable performance based on prior laparoscopic suturing experience.
Collapse
Affiliation(s)
- J K Chau
- Department of Surgery, Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, 1001 Decarie Boulevard, DS1-3310, Montreal, QC, H4A 3J1, Canada.,Department of Surgery, Centre for Colon and Rectal Surgery, McGill University Health Centre, Montreal, QC, Canada
| | - E Bilgic
- Department of Surgery, Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, 1001 Decarie Boulevard, DS1-3310, Montreal, QC, H4A 3J1, Canada.,Steinberg Centre for Simulation and Interactive Learning, McGill University, Montreal, QC, Canada
| | - T Hada
- Department of Surgery, Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, 1001 Decarie Boulevard, DS1-3310, Montreal, QC, H4A 3J1, Canada
| | - M Trepanier
- Department of Surgery, Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, 1001 Decarie Boulevard, DS1-3310, Montreal, QC, H4A 3J1, Canada
| | - H Naghawi
- Department of Surgery, Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, 1001 Decarie Boulevard, DS1-3310, Montreal, QC, H4A 3J1, Canada
| | - P Kaneva
- Department of Surgery, Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, 1001 Decarie Boulevard, DS1-3310, Montreal, QC, H4A 3J1, Canada
| | - C Mueller
- Department of Surgery, Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, 1001 Decarie Boulevard, DS1-3310, Montreal, QC, H4A 3J1, Canada
| | - L Lee
- Department of Surgery, Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, 1001 Decarie Boulevard, DS1-3310, Montreal, QC, H4A 3J1, Canada. .,Department of Surgery, Centre for Colon and Rectal Surgery, McGill University Health Centre, Montreal, QC, Canada.
| |
Collapse
|
41
|
Quality control for crowdsourcing citation screening: the importance of assessment number and qualification set size. J Clin Epidemiol 2020; 122:160-162. [DOI: 10.1016/j.jclinepi.2020.02.009] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/09/2019] [Revised: 12/14/2019] [Accepted: 02/27/2020] [Indexed: 11/22/2022]
|
42
|
Kelly JD, Petersen A, Lendvay TS, Kowalewski TM. The effect of video playback speed on surgeon technical skill perception. Int J Comput Assist Radiol Surg 2020; 15:739-747. [PMID: 32297088 DOI: 10.1007/s11548-020-02134-x] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/16/2019] [Accepted: 03/10/2020] [Indexed: 11/26/2022]
Abstract
PURPOSE Finding effective methods of discriminating surgeon technical skill has proved a complex problem to solve computationally. Previous research has shown that obtaining non-expert crowd evaluations of surgical performances is as accurate as the gold standard, expert surgeon review. The aim of this research is: (1) to learn whether crowdsourced evaluators give higher ratings of technical skill to video of performances with increased playback speed, (2) its effect in discriminating skill levels, and (3) whether this increase is related to the evaluator consciously being aware that the video is manually manipulated. METHODS A set of ten peg transfer videos (five novices, five experts) were used to evaluate the perceived technical skill of the performers at each video playback speed used ([Formula: see text]). Objective metrics used for measuring technical skill were also computed for comparison by manipulating the corresponding kinematic data of each performance. Two videos of an expert and novice performing dry laboratory laparoscopic trials of peg transfer tasks were used to obtain evaluations at each playback speed ([Formula: see text]) of perception of whether a video is played at real-time playback speed or not. RESULTS We found that while both novices and experts had increased perceived technical skill as the video playback was increased, the amount of increase was significantly greater for experts. Each increase in the playback speed by [Formula: see text] was associated with, on average, a 0.72-point increase in the GOALS score (95% CI 0.60-0.84 point increase; [Formula: see text]) for expert videos and only a 0.24-point increase in the GOALS score (95% CI 0.13-0.36 point increase; [Formula: see text]) for novice videos. CONCLUSION Due to the differential increase in perceived technical skill due to increased playback speed for experts, the difference between novice and expert skill levels of surgical performances may be more easily discerned by manually increasing the video playback speed.
Collapse
Affiliation(s)
- Jason D Kelly
- Department of Mechanical Engineering, University of Minnesota, Minneapolis, MN, USA.
| | - Ashley Petersen
- Division of Biostatistics, University of Minnesota, Minneapolis, MN, USA
| | - Thomas S Lendvay
- Department of Urology, Seattle Children's Hospital, Seattle, WA, USA
| | - Timothy M Kowalewski
- Department of Mechanical Engineering, University of Minnesota, Minneapolis, MN, USA
| |
Collapse
|
43
|
Ljuhar D, Gibbons AT, Ponsky TA, Nataraja RM. Emerging technology and their application to paediatric surgical training. Semin Pediatr Surg 2020; 29:150909. [PMID: 32423598 DOI: 10.1016/j.sempedsurg.2020.150909] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
Affiliation(s)
- Damir Ljuhar
- Department of Paediatrics, School of Clinical Sciences, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne Australia; Department of Surgical Simulation, Monash Children's Hospital, Melbourne Australia
| | - Alexander T Gibbons
- Department of Pediatric Surgery, Akron Children's Hospital, Akron, OH United States
| | - Todd A Ponsky
- Department of Pediatric Surgery, Akron Children's Hospital, Akron, OH United States; Department of Pediatric Surgery, Cincinnati Children's Hospital Medical Centre, Cincinnati, OH United States
| | - Ramesh M Nataraja
- Department of Paediatrics, School of Clinical Sciences, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne Australia; Department of Surgical Simulation, Monash Children's Hospital, Melbourne Australia; Department of Surgery, School of Clinical Sciences, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne Australia.
| |
Collapse
|
44
|
Farcas MA, Azzie G. Performance assessment - The knowledge, skills and attitudes of surgical performance. Semin Pediatr Surg 2020; 29:150903. [PMID: 32423592 DOI: 10.1016/j.sempedsurg.2020.150903] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/22/2023]
Abstract
Pediatric surgical education has traditionally focused on the hard cognitive and psychomotor (technical) skills. While more and more attention is being paid to softer skills such as communication, collaboration, leadership, health advocacy, professionalism and scholarship, the bulk of curricula remain focused on the hard skills. An intricate part of education is the assessment of performance. This article reviews the current literature on the assessment of cognitive and psychomotor skills, focusing particularly on the assessment of technical skills in the realm of simulation.
Collapse
Affiliation(s)
- Monica A Farcas
- Department of Surgery, University of Toronto, Toronto, Canada; Division of Urology, St. Michael's Hospital, Toronto, Canada; Ki Ka Shing Knowledge Institute, Toronto, Canada; Institute for Biomedical Engineering, Science and Technology, Toronto, Canada.
| | - Georges Azzie
- Department of Surgery, University of Toronto, Toronto, Canada; Department of Pediatric Surgery, Hopitaux Pédiatriques de Nice CHU-Lenval, Canada; Division of General and Thoracic Surgery, Hospital for Sick Children, Toronto, Canada
| |
Collapse
|
45
|
Patnaik R, Anton NE, Stefanidis D. A video anchored rating scale leads to high inter-rater reliability of inexperienced and expert raters in the absence of rater training. Am J Surg 2020; 219:221-226. [PMID: 31918843 PMCID: PMC10495932 DOI: 10.1016/j.amjsurg.2019.12.026] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2019] [Revised: 11/01/2019] [Accepted: 12/21/2019] [Indexed: 11/26/2022]
Abstract
BACKGROUND Our objective was to assess the impact of incorporating videos in a behaviorally anchored performance rating scale on the inter-rater reliability (IRR) of expert, intermediate and novice raters. METHODS The Intra-corporeal Suturing Assessment Tool (ISAT) was modified to include short video clips demonstrating poor, average, and expert performances. Blinded raters used this tool to assess videos of trainees performing suturing on a porcine model. Three attending surgeons, 4 residents, and 4 novice raters participated; no rater training was provided. The IRR was then compared among rater groups. RESULTS The IRR using the modified ISAT was high at 0.80 (p < 0.001). Ratings were significantly correlated with trainee objective suturing scores for all rater groups (experts: R = 0.84, residents: R = 0.81, and novices: R = 0.69; p < 0.001). CONCLUSIONS Incorporating video anchors (to define performance) in the ISAT led to high IRR and enabled novices to achieve similar consistency in their ratings as experts.
Collapse
Affiliation(s)
- Ronit Patnaik
- Indiana University School of Medicine, Department of Surgery, 545 Barnhill Dr. Emerson Hall, Indianapolis, IN, 46202, USA.
| | - Nicholas E Anton
- Indiana University School of Medicine, Department of Surgery, 545 Barnhill Dr. Emerson Hall, Indianapolis, IN, 46202, USA.
| | - Dimitrios Stefanidis
- Indiana University School of Medicine, Department of Surgery, 545 Barnhill Dr. Emerson Hall, Indianapolis, IN, 46202, USA.
| |
Collapse
|
46
|
Prebay ZJ, Peabody JO, Miller DC, Ghani KR. Video review for measuring and improving skill in urological surgery. Nat Rev Urol 2020; 16:261-267. [PMID: 30622365 DOI: 10.1038/s41585-018-0138-2] [Citation(s) in RCA: 23] [Impact Index Per Article: 5.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Interest is growing within the urological surgery community for objective assessments of technical skill. Surgical video review relies on the use of objective assessment tools to evaluate both global and procedure-specific skill. These evaluations provide structured feedback to surgeons with the aim of improving technique, which has been associated with patient outcomes. Currently, skill assessments can be performed by using expert peer-review, crowdsourcing or computer-based methods. Given the relationship between skill and patient outcomes, surgeons might be required in the future to provide empirical evidence of their technical skill for certification, employment, credentialing and quality improvement. Interventions such as coaching and skills workshops incorporating video review might help surgeons improve their skill, with the ultimate goal of improving patient outcomes.
Collapse
Affiliation(s)
- Zachary J Prebay
- School of Medicine, Medical College of Wisconsin, Milwaukee, WI, USA
| | - James O Peabody
- Center for Outcomes Research, Analytics and Evaluation, Vattikuti Urology Institute, Henry Ford Hospital, Detroit, MI, USA
| | - David C Miller
- Department of Urology, University of Michigan, Ann Arbor, MI, USA
| | - Khurshid R Ghani
- Department of Urology, University of Michigan, Ann Arbor, MI, USA.
| |
Collapse
|
47
|
Crowdsourcing in health and medical research: a systematic review. Infect Dis Poverty 2020; 9:8. [PMID: 31959234 PMCID: PMC6971908 DOI: 10.1186/s40249-020-0622-9] [Citation(s) in RCA: 60] [Impact Index Per Article: 15.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/17/2019] [Accepted: 01/07/2020] [Indexed: 12/31/2022] Open
Abstract
Background Crowdsourcing is used increasingly in health and medical research. Crowdsourcing is the process of aggregating crowd wisdom to solve a problem. The purpose of this systematic review is to summarize quantitative evidence on crowdsourcing to improve health. Methods We followed Cochrane systematic review guidance and systematically searched seven databases up to September 4th 2019. Studies were included if they reported on crowdsourcing and related to health or medicine. Studies were excluded if recruitment was the only use of crowdsourcing. We determined the level of evidence associated with review findings using the GRADE approach. Results We screened 3508 citations, accessed 362 articles, and included 188 studies. Ninety-six studies examined effectiveness, 127 examined feasibility, and 37 examined cost. The most common purposes were to evaluate surgical skills (17 studies), to create sexual health messages (seven studies), and to provide layperson cardio-pulmonary resuscitation (CPR) out-of-hospital (six studies). Seventeen observational studies used crowdsourcing to evaluate surgical skills, finding that crowdsourcing evaluation was as effective as expert evaluation (low quality). Four studies used a challenge contest to solicit human immunodeficiency virus (HIV) testing promotion materials and increase HIV testing rates (moderate quality), and two of the four studies found this approach saved money. Three studies suggested that an interactive technology system increased rates of layperson initiated CPR out-of-hospital (moderate quality). However, studies analyzing crowdsourcing to evaluate surgical skills and layperson-initiated CPR were only from high-income countries. Five studies examined crowdsourcing to inform artificial intelligence projects, most often related to annotation of medical data. Crowdsourcing was evaluated using different outcomes, limiting the extent to which studies could be pooled. Conclusions Crowdsourcing has been used to improve health in many settings. Although crowdsourcing is effective at improving behavioral outcomes, more research is needed to understand effects on clinical outcomes and costs. More research is needed on crowdsourcing as a tool to develop artificial intelligence systems in medicine. Trial registration PROSPERO: CRD42017052835. December 27, 2016.
Collapse
|
48
|
Global Interprofessional Therapeutic Communication Scale© (GITCS©): Development and Validation. Clin Simul Nurs 2019. [DOI: 10.1016/j.ecns.2019.05.006] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/23/2022]
|
49
|
Rice MK, Zenati MS, Novak SM, Al Abbas AI, Zureikat AH, Zeh HJ, Hogg ME. Crowdsourced Assessment of Inanimate Biotissue Drills: A Valid and Cost-Effective Way to Evaluate Surgical Trainees. JOURNAL OF SURGICAL EDUCATION 2019; 76:814-823. [PMID: 30472061 DOI: 10.1016/j.jsurg.2018.10.007] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/15/2018] [Revised: 08/28/2018] [Accepted: 10/07/2018] [Indexed: 06/09/2023]
Abstract
OBJECTIVE Providing feedback to surgical trainees is a critical component for assessment of technical skills, yet remains costly and time consuming. We hypothesize that statistical selection can identify a homogenous group of nonexpert crowdworkers capable of accurately grading inanimate surgical video. DESIGN Applicants auditioned by grading 9 training videos using the Objective Structured Assessment of Technical Skills (OSATS) tool and an error-based checklist. The summed OSATS, summed errors, and OSATS summary score were tested for outliers using Cronbach's Alpha and single measure intraclass correlation. Accepted crowdworkers then submitted grades for videos in 3 different compositions: full video 1× speed, full video 2× speed, and critical section segmented video. Graders were blinded to this study and a similar statistical analysis was performed. SETTING The study was conducted at the University of Pittsburgh Medical Center (Pittsburgh, PA), a tertiary care academic teaching hospital. PARTICIPANTS Thirty-six premedical students participated as crowdworker applicants and 2 surgery experts were compared as the gold-standard. RESULTS The selected hire intraclass correlation was 0.717 for Total Errors and 0.794 for Total OSATS for the first hire group and 0.800 for Total OSATS and 0.654 for Total Errors for the second hire group. There was very good correlation between full videos at 1× and 2× speed with an interitem statistic of 0.817 for errors and 0.86 for OSATS. Only moderate correlation was found with critical section segments. In 1 year 275hours of inanimate video was graded costing $22.27/video or $1.03/minute. CONCLUSIONS Statistical selection can be used to identify a homogenous cohort of crowdworkers used for grading trainees' inanimate drills. Crowdworkers can distinguish OSATS metrics and errors in full videos at 2× speed but were less consistent with segmented videos. The program is a comparatively cost-effective way to provide feedback to surgical trainees.
Collapse
Affiliation(s)
- MaryJoe K Rice
- University of Maryland School of Medicine, Baltimore, Maryland
| | - Mazen S Zenati
- Department of Surgery, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania
| | - Stephanie M Novak
- Department of Surgery, Northshore University HealthSystem, Chicago, Illinois
| | - Amr I Al Abbas
- Division of Surgical Oncology, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania
| | - Amer H Zureikat
- Division of Surgical Oncology, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania
| | - Herbert J Zeh
- Department of Surgery, University of Texas Southwestern Medical Center, Dallas, Texas
| | - Melissa E Hogg
- Department of Surgery, Northshore University HealthSystem, Chicago, Illinois.
| |
Collapse
|
50
|
St John-Matthews J, Newton PM, Grant AJ, Robinson L. Crowdsourcing in health professions education: What radiography educators can learn from other disciplines. Radiography (Lond) 2019; 25:164-169. [PMID: 30955690 DOI: 10.1016/j.radi.2018.11.006] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/04/2018] [Revised: 11/15/2018] [Accepted: 11/20/2018] [Indexed: 10/27/2022]
Abstract
OBJECTIVES Crowdsourcing works through an institution outsourcing a function normally performed by an employee or group of individuals. Within a crowdsource users, known as the crowd, form a community who voluntarily undertake a task which involves the pooling of knowledge resources. A literature review was undertaken to identify how the tool is being used in health professions education, and potential for use in radiography education. KEY FINDINGS 17 papers were returned. Literature identified was assessed against an established crowdsourcing definition. Reviewing these yielded four themes for discussion: student selection procedures, lesson planning, teaching materials and assessment. CONCLUSION Crowdsourcing is associated with innovative activities through collective solution seeking via a large network of users. It is increasingly being adopted in healthcare training and maybe transferable to educational activities within the field of radiography education.
Collapse
Affiliation(s)
- J St John-Matthews
- Department of Allied Health Professions, Faculty of Health and Applied Sciences, University of the West of England, Bristol, BS16 1DD, UK.
| | - P M Newton
- Research in Health Professions Education, Swansea University Medical School, Swansea, Wales, SA2 8PP, UK
| | - A J Grant
- Research in Health Professions Education, Swansea University Medical School, Swansea, Wales, SA2 8PP, UK
| | - L Robinson
- School of Health Science, Frederick Road Campus, University of Salford, Allerton Building, M6 6PU, UK
| |
Collapse
|