1
|
Olsen RG, Konge L, Hayatzaki K, Mortensen MA, Røder A, Bjerrum F. Medical Students Cannot Assess Robotic Surgeons Performing Radical Prostatectomy. Simul Healthc 2024; 19:213-219. [PMID: 37279115 DOI: 10.1097/sih.0000000000000733] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
INTRODUCTION Medical students have previously been shown to be just as effective for video rating as experts. We want to compare medical students to experienced surgeons as video assessors of simulated robot-assisted radical prostatectomy (RARP) performance. MATERIALS AND METHODS Video recordings of three RARP modules on the RobotiX (formerly Simbionix) simulator from a previous study were used. Five novice surgeons, five experienced robotic surgeons, and five experienced robotic surgeons in RARP performed a total of 45 video-recorded procedures. The videos were assessed with the modified Global Evaluative Assessment of Robotic Skills tool as both full-length and an edited edition that only included the first 5 minutes of the procedure. RESULTS Fifty medical students and two experienced RARP surgeons (ES) performed a total of 680 video ratings of full-length videos and 5-minute videos (2-9 ratings per video). Medical students and ES showed poor agreement for both full-length videos and 5-minute videos (0.29 and -0.13, respectively). Medical students could not discriminate between the skill level of the surgeons in either full-length videos or 5-minute videos ( P = 0.053-0.36 and P = 0.21-0.82), whereas ES could discriminate between novice surgeons and experienced surgeons (full-length, P < 0.001, and 5 minutes, P = 0.007) and intermediate and experienced surgeons (full-length, P = 0.001, and 5 minutes, P = 0.01) in both full-length videos and 5-minute videos. CONCLUSION We found that medical students cannot be used to assess RARP because they showed poor agreement with the ES rating for both full-length videos and 5-minute videos. Medical students could not discriminate between surgical skill levels.
Collapse
Affiliation(s)
- Rikke Groth Olsen
- From the Copenhagen Academy for Medical Education and Simulation (CAMES) (R.G.O., L.K., F.B.); Department of Urology (R.G.O., A.R.), Copenhagen Prostate Cancer Center, Copenhagen University Hospital-Rigshospitalet; Faculty of Health and Medical Sciences (L.K., A.R.), University of Copenhagen, Copenhagen; Department of Urology (K.H.), Zealand University Hospital, Roskilde; Department of Urology (M.A.M.), Odense University Hospital; Department of Clinical Research (M.A.M.), University of Southern Denmark, Odense; and Department of Surgery, Herlev-Gentofte Hospital (F.B.), Herlev, Denmark
| | | | | | | | | | | |
Collapse
|
2
|
Olsen RG, Konge L, Hayatzaki K, Mortensen MA, Bube SH, Røder A, Azawi N, Bjerrum F. Laypersons versus experienced surgeons in assessing simulated robot-assisted radical prostatectomy. World J Urol 2023; 41:3745-3751. [PMID: 37882808 PMCID: PMC10693505 DOI: 10.1007/s00345-023-04664-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2023] [Accepted: 09/29/2023] [Indexed: 10/27/2023] Open
Abstract
BACKGROUND Feedback is important for surgical trainees but it can be biased and time-consuming. We examined crowd-sourced assessment as an alternative to experienced surgeons' assessment of robot-assisted radical prostatectomy (RARP). METHODS We used video recordings (n = 45) of three RARP modules on the RobotiX, Simbionix simulator from a previous study in a blinded comparative assessment study. A group of crowd workers (CWs) and two experienced RARP surgeons (ESs) evaluated all videos with the modified Global Evaluative Assessment of Robotic Surgery (mGEARS). RESULTS One hundred forty-nine CWs performed 1490 video ratings. Internal consistency reliability was high (0.94). Inter-rater reliability and test-retest reliability were low for CWs (0.29 and 0.39) and moderate for ESs (0.61 and 0.68). In an Analysis of Variance (ANOVA) test, CWs could not discriminate between the skill level of the surgeons (p = 0.03-0.89), whereas ES could (p = 0.034). CONCLUSION We found very low agreement between the assessments of CWs and ESs when they assessed robot-assisted radical prostatectomies. As opposed to ESs, CWs could not discriminate between surgical experience using the mGEARS ratings or when asked if they wanted the surgeons to perform their robotic surgery.
Collapse
Affiliation(s)
- Rikke Groth Olsen
- Copenhagen Academy for Medical Education and Simulation (CAMES), University Hospital - Rigshospitalet, Ryesgade 53B, 2100, Copenhagen, Denmark.
- Department of Urology, Copenhagen Prostate Cancer Center, Copenhagen University Hospital - Rigshospitalet, Copenhagen, Denmark.
- Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark.
| | - Lars Konge
- Copenhagen Academy for Medical Education and Simulation (CAMES), University Hospital - Rigshospitalet, Ryesgade 53B, 2100, Copenhagen, Denmark
- Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
| | | | - Mike Allan Mortensen
- Department of Urology, Odense University Hospital, Odense, Denmark
- Department of Clinical Research, University of Southern Denmark, Odense, Denmark
| | - Sarah Hjartbro Bube
- Copenhagen Academy for Medical Education and Simulation (CAMES), University Hospital - Rigshospitalet, Ryesgade 53B, 2100, Copenhagen, Denmark
- Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
- Department of Urology, Zealand University Hospital, Roskilde, Denmark
| | - Andreas Røder
- Department of Urology, Copenhagen Prostate Cancer Center, Copenhagen University Hospital - Rigshospitalet, Copenhagen, Denmark
- Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
| | - Nessn Azawi
- Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
- Department of Urology, Zealand University Hospital, Roskilde, Denmark
| | - Flemming Bjerrum
- Copenhagen Academy for Medical Education and Simulation (CAMES), University Hospital - Rigshospitalet, Ryesgade 53B, 2100, Copenhagen, Denmark
- Department of Surgery, Herlev-Gentofte Hospital, Herlev, Denmark
| |
Collapse
|
3
|
Ross SB, Modasi A, Christodoulou M, Sucandy I, Mehran A, Lobe TE, Witkowski E, Satava R. New generation evaluations: video-based surgical assessments : A technology update. Surg Endosc 2023; 37:7401-7411. [PMID: 37608232 DOI: 10.1007/s00464-023-10311-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/06/2023] [Accepted: 07/12/2023] [Indexed: 08/24/2023]
Abstract
BACKGROUND Surgical skill training, assessment, and feedback are the backbone of surgical training. High-quality skills require expert supervision and evaluation throughout a resource-intensive multi-year training process. As technological barriers to internet access and the ability to save and upload surgical videos continue to improve, video-based assessment technology is emerging as a tool that could reshape surgical training for the next generation of surgeons. Video-based assessment platforms have the potential to allow surgeons from across the globe to upload their surgical videos online and receive high-quality, standardized, and unbiased feedback. They combine visual recordings of a surgeon's operative technique, with standardized grading tools that have the potential to significantly impact surgical training and technical skill acquisition across the world. METHOD The platforms included in this review are in various stages of development after a thorough discussion with national experts on the SAGES TAVAC (Technology and Value Assessments) Committee. For each VBA program, a description of its platform was given and a literature review was obtained using a PubMed search performed from inception until December 2021. RESULTS The study reviewed all video-based assessment programs currently available in the market, identified their strengths and weaknesses, and how they can be optimized in future. CONCLUSION The technological platforms will play a key role in the training and technical skill acquisition of the next generation of surgeons and can have an immense impact on patient care across the world. There is immense potential for all these platforms to grow and become incorporated within the framework of an effective surgical training program.
Collapse
Affiliation(s)
- Sharona B Ross
- Digestive Health Institute, AdventHealth Tampa, 3000 Medical Park Dr. Suite 500, Tampa, FL, 33613, USA.
| | | | - Maria Christodoulou
- Digestive Health Institute, AdventHealth Tampa, 3000 Medical Park Dr. Suite 500, Tampa, FL, 33613, USA
| | - Iswanto Sucandy
- Digestive Health Institute, AdventHealth Tampa, 3000 Medical Park Dr. Suite 500, Tampa, FL, 33613, USA
| | - Anvari Mehran
- Division of General Surgery, St. Joseph's Healthcare, Hamilton, ON, Canada
- Department of Surgery, McMaster University, Hamilton, ON, Canada
| | - Thom E Lobe
- Department of Pediatric Surgery, University of Illinois at Chicago, Chicago, IL, USA
| | - Elan Witkowski
- Department of Surgery, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | | |
Collapse
|
4
|
Basile G, Pradere B, Bellmunt J, Rouprêt M, Kamat AM, Seisen T. A Plea for the Use of Video-recorded Cystoscopy for Blinded Independent Central Review of Tumor Response in Open-label Non-muscle invasive Bladder Cancer Trials. Eur Urol Oncol 2023; 6:453-455. [PMID: 37604762 DOI: 10.1016/j.euo.2023.07.005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/04/2023] [Accepted: 07/07/2023] [Indexed: 08/23/2023]
Abstract
Given that most of non-muscle invasive bladder cancer trials available in the literature share the same limitation of an open-label design, they are likely prone to detection bias. Indeed, tumor response is currently assessed by unblinded investigators, mainly via follow-up cystoscopy, which has well-known intraobserver and interobserver variability. Thus, we believe that the use of video-recorded cystoscopy for blinded independent central review of tumor response could help to minimize detection bias in open-label NMIBC trials.
Collapse
Affiliation(s)
- Giuseppe Basile
- Department of Urology, IRCCS San Raffaele Hospital, Milan, Italy
| | - Benjamin Pradere
- Department of Urology UROSUD, La Croix Du Sud Hospital, Quint Fonsegrives, France; Department of Urology, Comprehensive Cancer Center, Medical University of Vienna, Vienna, Austria
| | - Joaquim Bellmunt
- Department of Medical Oncology, Dana-Farber Cancer Center and IMIM Research Lab, Harvard Medical School, Boston, MA, USA
| | - Morgan Rouprêt
- Department of Urology, Pitié-Salpêtrière Hospital, AP-HP, GRC 5 Predictive Onco-Urology, Sorbonne University, Paris, France
| | - Ashish M Kamat
- Department of Urology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - Thomas Seisen
- Department of Urology, Pitié-Salpêtrière Hospital, AP-HP, GRC 5 Predictive Onco-Urology, Sorbonne University, Paris, France.
| |
Collapse
|
5
|
Chen X, Zhu H, Mei L, Shu Q, Cheng X, Luo F, Zhao Y, Chen S, Pan Y. Video-Based versus On-Site Neonatal Pain Assessment in Neonatal Intensive Care Units: The Impact of Video-Based Neonatal Pain Assessment in Real-World Scenario on Pain Diagnosis and Its Artificial Intelligence Application. Diagnostics (Basel) 2023; 13:2661. [PMID: 37627921 PMCID: PMC10453072 DOI: 10.3390/diagnostics13162661] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/05/2023] [Revised: 08/02/2023] [Accepted: 08/11/2023] [Indexed: 08/27/2023] Open
Abstract
BACKGROUND Neonatal pain assessment (NPA) represents a huge global problem of essential importance, as a timely and accurate assessment of neonatal pain is indispensable for implementing pain management. PURPOSE To investigate the consistency of pain scores derived through video-based NPA (VB-NPA) and on-site NPA (OS-NPA), providing the scientific foundation and feasibility of adopting VB-NPA results in a real-world scenario as the gold standard for neonatal pain in clinical studies and labels for artificial intelligence (AI)-based NPA (AI-NPA) applications. SETTING A total of 598 neonates were recruited from a pediatric hospital in China. METHODS This observational study recorded 598 neonates who underwent one of 10 painful procedures, including arterial blood sampling, heel blood sampling, fingertip blood sampling, intravenous injection, subcutaneous injection, peripheral intravenous cannulation, nasopharyngeal suctioning, retention enema, adhesive removal, and wound dressing. Two experienced nurses performed OS-NPA and VB-NPA at a 10-day interval through double-blind scoring using the Neonatal Infant Pain Scale to evaluate the pain level of the neonates. Intra-rater and inter-rater reliability were calculated and analyzed, and a paired samples t-test was used to explore the bias and consistency of the assessors' pain scores derived through OS-NPA and VB-NPA. The impact of different label sources was evaluated using three state-of-the-art AI methods trained with labels given by OS-NPA and VB-NPA, respectively. RESULTS The intra-rater reliability of the same assessor was 0.976-0.983 across different times, as measured by the intraclass correlation coefficient. The inter-rater reliability was 0.983 for single measures and 0.992 for average measures. No significant differences were observed between the OS-NPA scores and the assessment of an independent VB-NPA assessor. The different label sources only caused a limited accuracy loss of 0.022-0.044 for the three AI methods. CONCLUSION VB-NPA in a real-world scenario is an effective way to assess neonatal pain due to its high intra-rater and inter-rater reliability compared to OS-NPA and could be used for the labeling of large-scale NPA video databases for clinical studies and AI training.
Collapse
Affiliation(s)
- Xiaofei Chen
- Gastroenterology Department, The Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou 310052, China;
| | - Huaiyu Zhu
- College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, China; (H.Z.); (Y.Z.)
| | - Linli Mei
- Administration Department of Nosocomial Infection, The Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou 310052, China; (L.M.); (Q.S.)
| | - Qi Shu
- Administration Department of Nosocomial Infection, The Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou 310052, China; (L.M.); (Q.S.)
| | - Xiaoying Cheng
- Quality Improvement Office, The Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou 310052, China;
| | - Feixiang Luo
- Neonatal Intensive Care Unit, The Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou 310052, China;
| | - Yisheng Zhao
- College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, China; (H.Z.); (Y.Z.)
| | - Shuohui Chen
- Administration Department of Nosocomial Infection, The Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou 310052, China; (L.M.); (Q.S.)
| | - Yun Pan
- College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, China; (H.Z.); (Y.Z.)
| |
Collapse
|
6
|
Frithioff A, Frendø M, Foghsgaard S, Sørensen MS, Andersen SAW. Are Video Recordings Reliable for Assessing Surgical Performance? A Prospective Reliability Study Using Generalizability Theory. Simul Healthc 2023; 18:219-225. [PMID: 36260767 DOI: 10.1097/sih.0000000000000672] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
INTRODUCTION Reliability is pivotal in surgical skills assessment. Video-based assessment can be used for objective assessment without physical presence of assessors. However, its reliability for surgical assessments remains largely unexplored. In this study, we evaluated the reliability of video-based versus physical assessments of novices' surgical performances on human cadavers and 3D-printed models-an emerging simulation modality. METHODS Eighteen otorhinolaryngology residents performed 2 to 3 mastoidectomies on a 3D-printed model and 1 procedure on a human cadaver. Performances were rated by 3 experts evaluating the final surgical result using a well-known assessment tool. Performances were rated both hands-on/physically and by video recordings. Interrater reliability and intrarater reliability were explored using κ statistics and the optimal number of raters and performances required in either assessment modality was determined using generalizability theory. RESULTS Interrater reliability was moderate with a mean κ score of 0.58 (range 0.53-0.62) for video-based assessment and 0.60 (range, 0.55-0.69) for physical assessment. Video-based and physical assessments were equally reliable (G coefficient 0.85 vs. 0.80 for 3D-printed models and 0.86 vs 0.87 for cadaver dissections). The interaction between rater and assessment modality contributed to 8.1% to 9.1% of the estimated variance. For the 3D-printed models, 2 raters evaluating 2 video-recorded performances or 3 raters physically assessing 2 performances yielded sufficient reliability for high-stakes assessment (G coefficient >0.8). CONCLUSIONS Video-based and physical assessments were equally reliable. Some raters were affected by changing from physical to video-based assessment; consequently, assessment should be either physical or video based, not a combination.
Collapse
Affiliation(s)
- Andreas Frithioff
- From the Copenhagen Hearing and Balance Center, Department of Otorhinolaryngology-Head & Neck Surgery and Audiology (A.F., M.F., S.F., M.S., S.A.W.A.), Rigshospitalet, Copenhagen; and Copenhagen Academy for Medical Education and Simulation (CAMES; A.F., M.F., S.A.W.A.), Center for HR & Education, Copenhagen, Denmark
| | | | | | | | | |
Collapse
|
7
|
Pradere B, Basile G, Seisen T. Re: Maria S. Lindgren, Eric Hansen, Nessn Azawi, Anna M. Nielsen, Lars Dyrskjøt, Jørgen B. Jensen. DaBlaCa-13 Study: Oncological Outcome of Short-term, Intensive Chemoresection with Mitomycin in Nonmuscle Invasive Bladder Cancer: Primary Outcome of a Randomized Controlled Trial. J Clin Oncol. In press. https://doi.org/10.1200/JCO.22.00470. Eur Urol 2023; 83:e105-e106. [PMID: 36717285 DOI: 10.1016/j.eururo.2022.12.036] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/11/2022] [Accepted: 12/21/2022] [Indexed: 01/29/2023]
Affiliation(s)
- Benjamin Pradere
- Department of Urology UROSUD, La Croix Du Sud Hospital, Quint Fonsegrives, France; Department of Urology, Comprehensive Cancer Center, Medical University of Vienna, Vienna, Austria
| | - Giuseppe Basile
- Department of Urology, IRCCS San Raffaele Hospital, Milan, Italy
| | - Thomas Seisen
- Department of Urology, Pitié-Salpêtrière Hospital, APHP, Sorbonne University, GRC 5, Predictive Onco-Urology, Paris, France.
| |
Collapse
|
8
|
Spatiotemporal Modeling of Grip Forces Captures Proficiency in Manual Robot Control. BIOENGINEERING (BASEL, SWITZERLAND) 2023; 10:bioengineering10010059. [PMID: 36671631 PMCID: PMC9854605 DOI: 10.3390/bioengineering10010059] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/10/2022] [Revised: 12/19/2022] [Accepted: 12/27/2022] [Indexed: 01/06/2023]
Abstract
New technologies for monitoring grip forces during hand and finger movements in non-standard task contexts have provided unprecedented functional insights into somatosensory cognition. Somatosensory cognition is the basis of our ability to manipulate and transform objects of the physical world and to grasp them with the right amount of force. In previous work, the wireless tracking of grip-force signals recorded from biosensors in the palm of the human hand has permitted us to unravel some of the functional synergies that underlie perceptual and motor learning under conditions of non-standard and essentially unreliable sensory input. This paper builds on this previous work and discusses further, functionally motivated, analyses of individual grip-force data in manual robot control. Grip forces were recorded from various loci in the dominant and non-dominant hands of individuals with wearable wireless sensor technology. Statistical analyses bring to the fore skill-specific temporal variations in thousands of grip forces of a complete novice and a highly proficient expert in manual robot control. A brain-inspired neural network model that uses the output metric of a self-organizing pap with unsupervised winner-take-all learning was run on the sensor output from both hands of each user. The neural network metric expresses the difference between an input representation and its model representation at any given moment in time and reliably captures the differences between novice and expert performance in terms of grip-force variability.Functionally motivated spatiotemporal analysis of individual average grip forces, computed for time windows of constant size in the output of a restricted amount of task-relevant sensors in the dominant (preferred) hand, reveal finger-specific synergies reflecting robotic task skill. The analyses lead the way towards grip-force monitoring in real time. This will permit tracking task skill evolution in trainees, or identify individual proficiency levels in human robot-interaction, which represents unprecedented challenges for perceptual and motor adaptation in environmental contexts of high sensory uncertainty. Cross-disciplinary insights from systems neuroscience and cognitive behavioral science, and the predictive modeling of operator skills using parsimonious Artificial Intelligence (AI), will contribute towards improving the outcome of new types of surgery, in particular the single-port approaches such as NOTES (Natural Orifice Transluminal Endoscopic Surgery) and SILS (Single-Incision Laparoscopic Surgery).
Collapse
|
9
|
Bube SH, Kingo PS, Madsen MG, Vásquez JL, Norus T, Olsen RG, Dahl C, Hansen RB, Konge L, Azawi N. National Implementation of Simulator Training Improves Transurethral Resection of Bladder Tumours in Patients. EUR UROL SUPPL 2022; 39:29-35. [PMID: 35528788 PMCID: PMC9068726 DOI: 10.1016/j.euros.2022.03.003] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 03/09/2022] [Indexed: 11/02/2022] Open
Abstract
Background Objective Design, setting, and participants Outcome measurements and statistical analyses Results and limitations Conclusions Patient summary
Collapse
|
10
|
Borg S. Video-based observation in impact evaluation. EVALUATION AND PROGRAM PLANNING 2021; 89:102007. [PMID: 34547597 DOI: 10.1016/j.evalprogplan.2021.102007] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/08/2021] [Revised: 09/13/2021] [Accepted: 09/14/2021] [Indexed: 06/13/2023]
Abstract
This paper reviews the literature on the use of video-based observation (VBO) with particular attention to monitoring and evaluation (M&E) on development projects. While the use of video both as a research tool and as a strategy for supporting professional development is well-documented across several disciplines, the extent to which VBO has been utilized in M&E contexts is less clearly defined. In order to provide theoretically-grounded recommendations for the development and implementation of one organisation's innovative VBO impact evaluation scheme, this review examines recent evidence of VBO in M&E contexts and draws on VBO literature more generally to identify its advantages and challenges together with advice for enhancing its effectiveness. Based on this analysis, the paper highlights a number of practical issues that should be considered when VBO is being developed for M&E in development contexts, particularly where videos are being made by participants themselves. The value of VBO in responding to COVID-19 and reducing carbon emissions is also noted.
Collapse
Affiliation(s)
- Simon Borg
- Department of Language, Literature, Mathematics and Interpretation, Western Norway University of Applied Sciences, Bergen, Norway.
| |
Collapse
|
11
|
Naples R, French JC, Han AY, Lipman JM, Awad MM. The Impact of Simulation Training on Operative Performance in General Surgery: Lessons Learned from a Prospective Randomized Trial. J Surg Res 2021; 270:513-521. [PMID: 34801802 DOI: 10.1016/j.jss.2021.10.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/19/2021] [Revised: 09/26/2021] [Accepted: 10/10/2021] [Indexed: 10/19/2022]
Abstract
BACKGROUND Practice in the simulated environment can improve surgical skills. However, the transfer of open complex surgical skills to the operating room is unclear. This study evaluated the effect of resident operative performance following a simulation experience on a hand-sewn small bowel anastomosis and determined the impact of utilizing proficiency-based training. METHODS Nine categorical interns performed a hand-sewn small bowel anastomosis in the operating room prior to (pre-test) and following (post-test) a 3-h simulation training session with an assessment at the end. Participants were randomly assigned to 1of 2 simulation training groups: proficiency-based or standard. Operative performance was videotaped. 2 independent, blinded faculty surgeons assessed performances by a global rating scale. Pre- and post-confidence levels were obtained on a 5-point Likert scale. RESULTS Overall, pre-test and post-test operative performance was similar (3 [IQR, 2.5 -3.5] versus 3 [IQR, 3 -3], P = 0.59). Furthermore, no difference was observed in the post-test performance with proficiency-based or standard training (3 [IQR, 3 -3] versus 3 [IQR, 3 -3], P = 0.73). Self-reported confidence with the skills, however, significantly improved (median 1 versus 4, P = 0.007). CONCLUSIONS In this prospective, randomized study, we did not observe an improvement in operative performance following simulation instruction and assessment, with both training groups. Overcoming barriers to skills transfer will be paramount in the future to optimize simulation training in general surgery. These findings highlight the importance of continued study for the ideal conditions and timing of technical skills training.
Collapse
Affiliation(s)
- Robert Naples
- Department of General Surgery, Cleveland Clinic, Cleveland, Ohio; Cleveland Clinic Lerner College of Medicine of Case Western Reserve University, Cleveland, Ohio.
| | - Judith C French
- Department of General Surgery, Cleveland Clinic, Cleveland, Ohio; Cleveland Clinic Lerner College of Medicine of Case Western Reserve University, Cleveland, Ohio
| | - Amy Y Han
- Department of General Surgery, Cleveland Clinic, Cleveland, Ohio
| | - Jeremy M Lipman
- Department of General Surgery, Cleveland Clinic, Cleveland, Ohio; Cleveland Clinic Lerner College of Medicine of Case Western Reserve University, Cleveland, Ohio
| | - Michael M Awad
- Department of Surgery, Washington University School of Medicine in St Louis, St Louis, Missouri
| |
Collapse
|
12
|
Bube SH, Kingo PS, Madsen MG, Vásquez JL, Norus TP, Olsen RG, Dahl C, Hansen RB, Konge L, Azawi NH. Validation of a novel assessment tool identifying proficiency in Transurethral Bladder Tumour Resection: The OSATURBS assessment tool. J Endourol 2021; 36:572-579. [PMID: 34731011 DOI: 10.1089/end.2021.0768] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022] Open
Abstract
BACKGROUND Competence in transurethral bladder tumour resection (TURB) is critical in bladder cancer management and should be ensured before independent practice. OBJECTIVE Develop an assessment tool for TURB and explore validity evidence in a clinical context. DESIGN, SETTING, AND PARTICIPANTS July 2019-March 2021, a total of 33 volunteer doctors from three hospitals were included. Participants performed two TURB procedures on patients with bladder tumours. A newly developed assessment tool (OSATURBS) was used for direct observation assessment, self-assessment, and blinded video-assessment. Outcome measurements and statistical analysis: Cronbach's alpha and Pearson's r were calculated for across items internal consistency reliability, inter-rater reliability, and test-retest reliability. Correlation between OSATURBS scores and the operative experience was calculated with Pearson's r and a pass/fail score was established. Differences in assessment scores were explored with paired t-test and independent samples t-test. RESULTS AND LIMITATIONS The internal consistency reliability across items Cronbach's alpha was 0.94 (n = 260, p < 0.001). Inter-rater reliability = 0.80 (n = 64, p < 0.001). Test-retest correlation was high, r = 0.71 (n = 32, p < 0.001). Relation to TURB experience was high, r = 0.71 (n = 32, p < 0.001). Pass/fail score = 19 points. Direct observation assessments were strongly correlated with video ratings (r = 0.85, p < 0.001) but with a significant social bias with lower scores for inexperienced and higher scores for experienced participants. Participants tended to overestimate their own performances. CONCLUSIONS OSATURBS assessment tool for TURB can be used for assessment of surgical proficiency in the clinical setting. Direct observation assessment and self-assessment are biased, and blinded video-assessment of TURB performances is advised.
Collapse
Affiliation(s)
- Sarah Hjartbro Bube
- Zealand University Hospital Roskilde, 53140, Department of Urology, Roskilde, Zealand, Denmark.,University of Copenhagen, 4321, Faculty of Health and Medical Science, Copenhagen, Denmark;
| | | | - Mia Gebauer Madsen
- Aarhus Universitetshospital, 11297, Department of Urology, Aarhus, Denmark;
| | - Juan Luis Vásquez
- Zealand University Hospital Roskilde, 53140, Department of Urology, Roskilde, Zealand, Denmark.,University of Copenhagen, 4321, Faculty of Health and Medical Science, Copenhagen, Zealand, Denmark;
| | - Thomas Peter Norus
- Zealand University Hospital Roskilde, 53140, Department of Urology, Roskilde, Sjaelland, Denmark;
| | - Rikke Groth Olsen
- National Hospital of the Faroe Islands, 112892, Surgical Department, Torshavn, Faroe Islands.,Rigshospitalet, 53146, CAMES - Copenhagen Academy for Medical Education and Simulation, Copenhagen, Denmark;
| | - Claus Dahl
- Capio Ramsay Santé, Department of Urology, Hellerup, Denmark;
| | - Rikke Bølling Hansen
- Herlev Hospital, 53176, Department of Urology, Gentofte, Denmark.,Rigshospitalet, 53146, CAMES - Copenhagen Academy for Medical Education and Simulation, Copenhagen, Denmark;
| | - Lars Konge
- Rigshospitalet, 53146, CAMES - Copenhagen Academy for Medical Education and Simulation, Copenhagen, Denmark.,University of Copenhagen, 4321, Faculty of Health and Medical Science, Copenhagen, Denmark;
| | - Nessn H Azawi
- Zealand University Hospital Roskilde, 53140, Department of Urology, Roskilde, Zealand, Denmark.,University of Copenhagen, 4321, Faculty of Health and Medical Science, Copenhagen, Denmark;
| |
Collapse
|
13
|
Navrazhina K, Tung R, Cressey BD, Decker A, Surprenant D, Parra CE, Poon E, Kim W, Minkis K. A blinded, multirater and multi-institutional study evaluating the Objective Structured Assessment of Technical Skills (OSATS) tool in dermatology education. J Am Acad Dermatol 2021; 85:1346-1348. [PMID: 34158162 DOI: 10.1016/j.jaad.2020.10.053] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/28/2020] [Revised: 10/05/2020] [Accepted: 10/20/2020] [Indexed: 11/19/2022]
Affiliation(s)
- Kristina Navrazhina
- Department of Dermatology, Weill Cornell/New York Presbyterian, New York, NY; Weill Cornell/Rockefeller/Sloan-Kettering Tri-Institutional MD-PhD Program, New York, NY
| | - Rebecca Tung
- Florida Dermatology and Skin Cancer Centers, Winter Haven, FL
| | | | | | - David Surprenant
- Division of Dermatology, Loyola University Medical Center, Maywood, IL
| | - Cindy E Parra
- Department of Dermatology, Weill Cornell/New York Presbyterian, New York, NY
| | - Emily Poon
- Department of Dermatology, Feinberg School of Medicine, Northwestern University, Chicago, IL
| | - Wendy Kim
- Division of Dermatology, Loyola University Medical Center, Maywood, IL
| | - Kira Minkis
- Department of Dermatology, Weill Cornell/New York Presbyterian, New York, NY.
| |
Collapse
|
14
|
Dagnaes-Hansen J, Konge L, Andreassen KH, Hansen RB. Virtual reality simulator metrics cannot be used to assess competence in ureteronephroscopy and stone removal - a validation study. Scand J Urol 2021; 55:399-403. [PMID: 34338581 DOI: 10.1080/21681805.2021.1960599] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/20/2022]
Abstract
OBJECTIVE The growing use of simulation-based training makes it necessary to develop efficient training programs in order to ensure optimal use of time and resources. Our aim was to develop and gather validity evidence for a simulation-based test in ureteronephroscopy and set a pass/fail standard for the test that will allow future mastery learning. DESIGN This study is a validation study. A test in ureteronephroscopy and stone removal on the URO Mentor™ virtual reality simulator (3D Systems, USA) was developed by two experienced urologists in order to ensure content. Participants with different experience completed three standardized tasks on the simulator and simulator-generated metrics were used as outcome parameters to minimize bias and ensure a fair response process. RESULTS Twenty novices, 15 intermediates, and 8 experienced urologists were included in the study. Validity evidence for internal structure and relationship to other variables was questionable with weak and mostly insignificant correlations across all four metrics (Cronbach's alpha = 0.14, p = 0.15) and across the three modules (Cronbach's alpha = 0.41 (p = 0.02), 0.35 (p = 0.06), 0.10 (p = 0.35), and 0.30 (p = 0.09) for each metric, respectively). It was not possible to establish a pass/fail score for the simulation test with meaningful consequences. CONCLUSION Our study showed that automatically generated simulator metrics cannot be used as a valid way of assessing competence in ureteronephroscopy. Virtual-reality simulator training could still be a valuable and patient-safe way to practice these skills, but an experienced supervisor is needed to determine when the trainee is ready to continue to supervised practice on patients.
Collapse
Affiliation(s)
- Julia Dagnaes-Hansen
- Copenhagen Academy for Medical Education and Simulation (CAMES), Rigshospitalet, Copenhagen, Denmark.,Urological Research Unit, Copenhagen University Hospital - Rigshospitalet, Copenhagen, Denmark.,Department of Clinical Medicine, University of Copenhagen, Copenhagen, Denmark
| | - Lars Konge
- Copenhagen Academy for Medical Education and Simulation (CAMES), Rigshospitalet, Copenhagen, Denmark
| | - Kim Hovgaard Andreassen
- Department of Urology, Copenhagen University Hospital - Herlev and Gentofte, Copenhagen, Denmark
| | - Rikke Bølling Hansen
- Copenhagen Academy for Medical Education and Simulation (CAMES), Rigshospitalet, Copenhagen, Denmark.,Department of Clinical Medicine, University of Copenhagen, Copenhagen, Denmark.,Department of Urology, Copenhagen University Hospital - Herlev and Gentofte, Copenhagen, Denmark
| |
Collapse
|
15
|
Weinstein JL, El-Gabalawy F, Sarwar A, DeBacker SS, Faintuch S, Berkowitz SJ, Bulman JC, Palmer MR, Matyal R, Mahmood F, Ahmed M. Analysis of Kinematic Differences in Hand Motion between Novice and Experienced Operators in IR: A Pilot Study. J Vasc Interv Radiol 2020; 32:226-234. [PMID: 33339678 DOI: 10.1016/j.jvir.2020.10.010] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/05/2020] [Revised: 09/18/2020] [Accepted: 10/12/2020] [Indexed: 11/28/2022] Open
Abstract
PURPOSE To prospectively validate electromagnetic hand motion tracking in interventional radiology to detect differences in operator experience using simulation. METHODS Sheath task: Six attending interventional radiologists (experts) and 6 radiology trainees (trainees) placed a wire through a sheath and performed a "pin-pull" maneuver, while an electromagnetic motion detection system recorded the hand motion. Radial task: Eight experts and 12 trainees performed palpatory radial artery access task on a radial access simulator. The trainees repeated the task with the nondominant hand. The experts were classified by their most frequent radial artery access technique as having either palpatory, ultrasound, or overall limited experience. The time, path length, and number of movements were calculated. Mann-Whitney U tests were used to compare the groups, and P < .05 was considered significant. RESULTS Sheath task: The experts took less time, had shorter path lengths, and used fewer movements than the trainees (11.7 seconds ± 3.3 vs 19.7 seconds ± 6.5, P < .01; 1.1 m ± 0.3 vs 1.4 m ± 0.4, P < .01; and 19.5 movements ± 8.5 vs 31.0 movements ± 8.0, P < .01, respectively). Radial task: The experts took less time, had shorter path lengths, and used fewer movements than the trainees (24.2 seconds ± 10.6 vs 33.1 seconds ± 16.9, P < .01; 2.0 m ± 0.5 vs 3.0 m ± 1.9, P < .001; and 36.5 movements ± 15.0 vs 54.5 movements ± 28.0, P < .001, respectively). The trainees had a shorter path length for their dominant hand than their nondominant hand (3.0 m ± 1.9 vs 3.5 m ± 1.9, P < .05). The expert palpatory group had a shorter path length than the ultrasound and limited experience groups (1.8 m ± 0.4 vs 2.0 m ± 0.4 and 2.3 m ± 1.2, respectively, P < .05). CONCLUSIONS Electromagnetic hand motion tracking can differentiate between the expert and trainee operators for simulated interventional tasks.
Collapse
Affiliation(s)
- Jeffrey L Weinstein
- Division of Vascular and Interventional Radiology, Beth Israel Deaconess Medical Center, Harvard Medical School, 1 Deaconess Rd, Boston, MA 02215
| | - Fady El-Gabalawy
- Division of Vascular and Interventional Radiology, Beth Israel Deaconess Medical Center, Harvard Medical School, 1 Deaconess Rd, Boston, MA 02215
| | - Ammar Sarwar
- Division of Vascular and Interventional Radiology, Beth Israel Deaconess Medical Center, Harvard Medical School, 1 Deaconess Rd, Boston, MA 02215
| | - Sarah Schroeppel DeBacker
- Division of Vascular and Interventional Radiology, Beth Israel Deaconess Medical Center, Harvard Medical School, 1 Deaconess Rd, Boston, MA 02215
| | - Salomao Faintuch
- Division of Vascular and Interventional Radiology, Beth Israel Deaconess Medical Center, Harvard Medical School, 1 Deaconess Rd, Boston, MA 02215
| | - Seth J Berkowitz
- Division of Vascular and Interventional Radiology, Beth Israel Deaconess Medical Center, Harvard Medical School, 1 Deaconess Rd, Boston, MA 02215
| | - Julie C Bulman
- Division of Vascular and Interventional Radiology, Beth Israel Deaconess Medical Center, Harvard Medical School, 1 Deaconess Rd, Boston, MA 02215
| | - Matthew R Palmer
- Division of Medical Physics, Beth Israel Deaconess Medical Center, Harvard Medical School, 1 Deaconess Rd, Boston, MA 02215
| | - Robina Matyal
- Department of Anesthesia, Beth Israel Deaconess Medical Center, Harvard Medical School, 1 Deaconess Rd, Boston, MA 02215
| | - Feroze Mahmood
- Department of Anesthesia, Beth Israel Deaconess Medical Center, Harvard Medical School, 1 Deaconess Rd, Boston, MA 02215
| | - Muneeb Ahmed
- Division of Vascular and Interventional Radiology, Beth Israel Deaconess Medical Center, Harvard Medical School, 1 Deaconess Rd, Boston, MA 02215.
| |
Collapse
|
16
|
Mantica G, Balzarini F, Dotta F, Rodriguez-Socarras M, Proietti S, Giusti G, Oneto F, Di Pierro M, Traverso P, Terrone C. Development of a photographic handbook to improve cystoscopy findings during resident's training: A randomised prospective study. Arab J Urol 2019; 17:243-248. [PMID: 31489242 PMCID: PMC6711137 DOI: 10.1080/2090598x.2019.1596400] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/04/2018] [Accepted: 03/06/2019] [Indexed: 11/10/2022] Open
Abstract
Objectives: To evaluate if the use of a photographic handbook (PH) can be a useful tool to improve the detection of disorders during cystoscopy training, as several hands-on tools have been proposed to improve technical skills but very few aim to improve specificity and sensitivity. Subjects and methods: Eight junior residents (JRs) were divided into two groups: Group A, comprised four JRs with previous limited experience of performing cystoscopies; and Group B, including four inexperienced JRs who were asked to study a specific PH before performing cystoscopies. The findings of the two groups were compared using the chi-squared test. Results: A total of 401 consecutive cystoscopies, of which 214 (53.4%) were performed by Group A and 187 (46.6%) by Group B, were considered. Group B showed superior ability in detecting uncommon findings (i.e., carcinoma in situ, bullous oedema, interstitial cystitis, etc.) with 24/46 (52.2%) detected vs eight of 32 (25%) in Group A (P = 0.016). Conclusions: The PH was a useful tool for improving identification of pathological conditions, which could be used to enhance hands-on simulator and practical tutored training. Abbreviations: CIS: carcinoma in situ; JR: junior resident; PH: photographic handbook; VR: virtual reality Classification: Stones/Endourology
Collapse
Affiliation(s)
- Guglielmo Mantica
- Department of Urology, Policlinico San Martino Hospital, University of Genova, Genova, Italy.,Department of Urology, San Raffaele Turro Hospital, San Raffaele University, Milan, Italy
| | - Federica Balzarini
- Department of Urology, Policlinico San Martino Hospital, University of Genova, Genova, Italy
| | - Federico Dotta
- Department of Urology, Policlinico San Martino Hospital, University of Genova, Genova, Italy
| | - Moises Rodriguez-Socarras
- Department of Urology, San Raffaele Turro Hospital, San Raffaele University, Milan, Italy.,ETCE - European Training Center in Endourology, San Raffaele Turro Hospital, Milan, Italy
| | - Silvia Proietti
- Department of Urology, San Raffaele Turro Hospital, San Raffaele University, Milan, Italy.,ETCE - European Training Center in Endourology, San Raffaele Turro Hospital, Milan, Italy
| | - Guido Giusti
- Department of Urology, San Raffaele Turro Hospital, San Raffaele University, Milan, Italy.,ETCE - European Training Center in Endourology, San Raffaele Turro Hospital, Milan, Italy
| | - Francesco Oneto
- Department of Urology, Policlinico San Martino Hospital, University of Genova, Genova, Italy
| | - Marco Di Pierro
- Department of Urology, Policlinico San Martino Hospital, University of Genova, Genova, Italy
| | - Paolo Traverso
- Department of Urology, Policlinico San Martino Hospital, University of Genova, Genova, Italy
| | - Carlo Terrone
- Department of Urology, Policlinico San Martino Hospital, University of Genova, Genova, Italy
| |
Collapse
|
17
|
|