1
|
Howie EE, Harari R, Dias RD, Wigmore SJ, Skipworth RJE, Yule S. Feasibility of Wearable Sensors to Assess Cognitive Load During Clinical Performance: Lessons Learned and Blueprint for Success. J Surg Res 2024; 302:222-231. [PMID: 39106733 DOI: 10.1016/j.jss.2024.07.009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2024] [Revised: 05/23/2024] [Accepted: 07/02/2024] [Indexed: 08/09/2024]
Abstract
INTRODUCTION Cognitive load (CogL) is increasingly recognized as an important resource underlying operative performance. Current innovations in surgery aim to develop objective performance metrics via physiological monitoring from wearable digital sensors. Surgeons have access to consumer technology that could measure CogL but need guidance regarding device selection and implementation. To realize the benefits of surgical performance improvement these methods must be feasible, incorporating human factors usability and design principles. This paper aims to evaluate the feasibility of using wearable sensors to assess CogL, identify the benefits and challenges of implementing devices, and develop guidance for surgeons planning to implement wearable devices in their research or practice. METHODS We examined the feasibility of wearable sensors from a series of empirical studies that measured aspects of clinical performance relating to CogL. Across four studies, 84 participants and five sensors were involved in the following clinical settings: (i) real intraoperative surgery; (ii) simulated laparoscopic surgery; and (iii) medical team performance outside the hospital. RESULTS Wearable devices worn on the wrist and chest were found to be comfortable. After a learning curve, electrodermal activity data were easily and reliably collected. Devices using photoplethysmography to determine heart rate variability were significantly limited by movement artifact. There was variable success with electroencephalography devices regarding connectivity, comfort, and usability. CONCLUSIONS It is feasible to use wearable sensors across various clinical settings, including surgery. There are some limitations, and their implementation is context and device dependent. To scale sensor use in clinical research, surgeons must embrace human factors principles to optimize wearability, usability, reliability, and data security.
Collapse
Affiliation(s)
- Emma E Howie
- Clinical Surgery, University of Edinburgh & Royal Infirmary of Edinburgh, Edinburgh, Scotland; Surgical Sabermetrics Laboratory, Usher Institute, University of Edinburgh, Edinburgh, Scotland.
| | - Ryan Harari
- Surgical Sabermetrics Laboratory, Usher Institute, University of Edinburgh, Edinburgh, Scotland; STRATUS Centre for Medical Simulation, Brigham & Women's Hospital, Boston, Massachusetts; Department of Emergency Medicine, Harvard Medical School, Boston, Massachusetts
| | - Roger D Dias
- Surgical Sabermetrics Laboratory, Usher Institute, University of Edinburgh, Edinburgh, Scotland; STRATUS Centre for Medical Simulation, Brigham & Women's Hospital, Boston, Massachusetts; Department of Emergency Medicine, Harvard Medical School, Boston, Massachusetts
| | - Stephen J Wigmore
- Clinical Surgery, University of Edinburgh & Royal Infirmary of Edinburgh, Edinburgh, Scotland; Surgical Sabermetrics Laboratory, Usher Institute, University of Edinburgh, Edinburgh, Scotland
| | - Richard J E Skipworth
- Clinical Surgery, University of Edinburgh & Royal Infirmary of Edinburgh, Edinburgh, Scotland; Surgical Sabermetrics Laboratory, Usher Institute, University of Edinburgh, Edinburgh, Scotland
| | - Steven Yule
- Clinical Surgery, University of Edinburgh & Royal Infirmary of Edinburgh, Edinburgh, Scotland; Surgical Sabermetrics Laboratory, Usher Institute, University of Edinburgh, Edinburgh, Scotland.
| |
Collapse
|
2
|
Chou R, Naz H, Boahene KDO, Maxwell JH, Wanamaker JR, Byrne PJ, Papel ID, Kontis TC, Hager GD, Ishii LE, Malekzadeh S, Vedula SS, Ishii M. Correcting for Rater Effects in Operating Room Surgical Skills Assessment. Laryngoscope 2024; 134:3548-3554. [PMID: 38470307 PMCID: PMC11245371 DOI: 10.1002/lary.31391] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/20/2023] [Revised: 02/05/2024] [Accepted: 02/26/2024] [Indexed: 03/13/2024]
Abstract
OBJECTIVE To estimate and adjust for rater effects in operating room surgical skills assessment performed using a structured rating scale for nasal septoplasty. METHODS We analyzed survey responses from attending surgeons (raters) who supervised residents and fellows (trainees) performing nasal septoplasty in a prospective cohort study. We fit a structural equation model with the rubric item scores regressed on a latent component of skill and then fit a second model including the rating surgeon as a random effect to model a rater-effects-adjusted latent surgical skill. We validated this model against conventional measures including the level of expertise and post-graduation year (PGY) commensurate with the trainee's performance, the actual PGY of the trainee, and whether the surgical goals were achieved. RESULTS Our dataset included 188 assessments by 7 raters and 41 trainees. The model with one latent construct for surgical skill and the rater as a random effect was the best. Rubric scores depended on how severe or lenient the rater was, sometimes almost as much as they depended on trainee skill. Rater-adjusted latent skill scores increased with attending-estimated skill levels and PGY of trainees, increased with the actual PGY, and appeared constant over different levels of achievement of surgical goals. CONCLUSION Our work provides a method to obtain rater effect adjusted surgical skill assessments in the operating room using structured rating scales. Our method allows for the creation of standardized (i.e., rater-effects-adjusted) quantitative surgical skill benchmarks using national-level databases on trainee assessments. LEVEL OF EVIDENCE N/A Laryngoscope, 134:3548-3554, 2024.
Collapse
Affiliation(s)
- Ryan Chou
- Department of Biomedical Engineering, Whiting School of Engineering, Johns Hopkins University, Baltimore, Maryland, U.S.A
| | - Hajira Naz
- Dugoni School of Dentistry, University of Pacific, San Francisco, California, U.S.A
| | - Kofi D O Boahene
- Department of Otolaryngology-Head and Neck Surgery, Johns Hopkins University School of Medicine, Baltimore, Maryland, U.S.A
- Division of Facial Plastic and Reconstructive Surgery, Department of Otolaryngology-Head and Neck Surgery, Johns Hopkins University School of Medicine, Baltimore, Maryland, U.S.A
| | - Jessica H Maxwell
- Department of Otolaryngology-Head and Neck Surgery, MedStar Georgetown University Hospital, Washington, DC, U.S.A
- ENT Section, Veterans Affairs Medical Center, Washington, DC, U.S.A
| | - John R Wanamaker
- Department of Otolaryngology-Head and Neck Surgery, MedStar Georgetown University Hospital, Washington, DC, U.S.A
- ENT Section, Veterans Affairs Medical Center, Washington, DC, U.S.A
| | - Patrick J Byrne
- Head and Neck Institute, Cleveland Clinic, Cleveland, Ohio, U.S.A
| | - Ira D Papel
- Department of Otolaryngology-Head and Neck Surgery, Johns Hopkins University School of Medicine, Baltimore, Maryland, U.S.A
- Aesthetic Center at Woodholme, Baltimore, Maryland, U.S.A
| | - Theda C Kontis
- Department of Otolaryngology-Head and Neck Surgery, Johns Hopkins University School of Medicine, Baltimore, Maryland, U.S.A
- Aesthetic Center at Woodholme, Baltimore, Maryland, U.S.A
| | - Gregory D Hager
- Malone Center for Engineering in Healthcare, Whiting School of Engineering, Johns Hopkins University, Baltimore, Maryland, U.S.A
- Department of Computer Science, Whiting School of Engineering, Johns Hopkins University, Baltimore, Maryland, U.S.A
| | - Lisa E Ishii
- Department of Otolaryngology-Head and Neck Surgery, Johns Hopkins University School of Medicine, Baltimore, Maryland, U.S.A
- Division of Facial Plastic and Reconstructive Surgery, Department of Otolaryngology-Head and Neck Surgery, Johns Hopkins University School of Medicine, Baltimore, Maryland, U.S.A
| | - Sonya Malekzadeh
- Department of Otolaryngology-Head and Neck Surgery, MedStar Georgetown University Hospital, Washington, DC, U.S.A
- ENT Section, Veterans Affairs Medical Center, Washington, DC, U.S.A
| | - S Swaroop Vedula
- Malone Center for Engineering in Healthcare, Whiting School of Engineering, Johns Hopkins University, Baltimore, Maryland, U.S.A
| | - Masaru Ishii
- Department of Otolaryngology-Head and Neck Surgery, Johns Hopkins University School of Medicine, Baltimore, Maryland, U.S.A
| |
Collapse
|
3
|
Silverii H, Cain MP, Ahn J, Fernandez N, Lendvay T, Gupta A, Joyner B, Kieran K, Shnorhavorian M, Merguerian P. Putting the coach in the game: Current and future state of surgical coaching in pediatric urology. J Pediatr Urol 2024; 20:608.e1-608.e8. [PMID: 38508978 DOI: 10.1016/j.jpurol.2024.03.004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/22/2023] [Revised: 03/04/2024] [Accepted: 03/05/2024] [Indexed: 03/22/2024]
Abstract
BACKGROUND Surgical coaching has been proposed as a mechanism to fill gaps in proficiency and encourage continued growth following formal surgical training. Coaching benefits have been demonstrated in other surgical fields; however, have not been evaluated within pediatric urology. The aims of this study were to survey members of The Societies for Pediatric Urology (SPU) to assess the current understanding and utilization of surgical coaching while gauging interest, potential barriers and personal goals for participation in a coaching program. METHODS Following IRB approval, members of the SPU were invited to electronically complete an anonymous survey which assessed 4 domains: 1) understanding of surgical coaching principles, 2) current utilization, 3) interest and potential barriers to participation, and 4) personal surgical goals. To evaluate understanding, questions with predefined correct answers on the key principles of coaching were posed either in multiple choice or True/False format to the SPU membership. RESULTS Of the 674 pediatric urologists invited, 146 completed the survey (22%). Of those, 46% correctly responded the definition of surgical coaching. Coaching utilization was reported in 27% of respondents currently or having previously participated in a surgical coaching program. Despite current participation rates, only 6 surgeons (4%) have completed training in surgical coaching, despite 79% expressing interest to participate in a surgical coaching program. The most influential barrier to participating in a coaching program was time commitment. Respondents largely prioritized technical and cognitive skill improvement as their primary goals for coaching (see figure below). CONCLUSIONS While interest in surgical coaching is high among pediatric urologists, the principles of surgical coaching were not universally understood. Furthermore, formal coach training is markedly deficient, representing a gap in our profession and an opportunity for significant avenues for improvement, especially for technical and cognitive skills. Development of a coaching model based on these results would best suit the needs of pediatric urologists providing that the time commitment barrier for these endeavors can be mitigated and/or reconciled.
Collapse
Affiliation(s)
- Hailey Silverii
- Seattle Children's Hospital, United States; University of Washington, United States.
| | - Mark P Cain
- Seattle Children's Hospital, United States; University of Washington, United States
| | - Jennifer Ahn
- Seattle Children's Hospital, United States; University of Washington, United States
| | - Nicolas Fernandez
- Seattle Children's Hospital, United States; University of Washington, United States
| | - Thomas Lendvay
- Seattle Children's Hospital, United States; University of Washington, United States
| | | | - Byron Joyner
- Seattle Children's Hospital, United States; University of Washington, United States
| | - Kathleen Kieran
- Seattle Children's Hospital, United States; University of Washington, United States
| | | | - Paul Merguerian
- Seattle Children's Hospital, United States; University of Washington, United States
| |
Collapse
|
4
|
Reitz ACW, Massarweh NN. Video-Based Assessment of Surgical Quality-Will Video Kill the Radio Star? JAMA Netw Open 2024; 7:e246477. [PMID: 38639941 DOI: 10.1001/jamanetworkopen.2024.6477] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 04/20/2024] Open
Affiliation(s)
- Alexandra C W Reitz
- Department of Surgery, Emory University School of Medicine, Atlanta, Georgia
| | - Nader N Massarweh
- Department of Surgery, Emory University School of Medicine, Atlanta, Georgia
- Surgery and Perioperative Care, Atlanta VA Health Care System, Decatur, Georgia
- Department of Surgery, Morehouse School of Medicine, Atlanta, Georgia
| |
Collapse
|
5
|
Balvardi S, Kaneva P, Semsar-Kazerooni K, Vassiliou M, Al Mahroos M, Mueller C, Fiore JF, Schwartzman K, Feldman LS. Effect of video-based self-reflection on intraoperative skills: A pilot randomized controlled trial. Surgery 2024; 175:1021-1028. [PMID: 38154996 DOI: 10.1016/j.surg.2023.11.028] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2023] [Revised: 10/13/2023] [Accepted: 11/26/2023] [Indexed: 12/30/2023]
Abstract
BACKGROUND The value of video-based self-assessment in enhancing surgical skills is uncertain. This study investigates the feasibility and estimates sample size for a full-scale randomized controlled trial to evaluate the effectiveness of video-based self-assessment to improve surgical performance of laparoscopic cholecystectomy in trainees. METHODS This parallel pilot randomized controlled trial included general surgery trainees performing supervised laparoscopic cholecystectomy randomized 1:1 to control (traditional intraoperative teaching) or intervention group (traditional teaching plus video-based self-assessment). Operative performance was measured by the attending surgeon blinded to group assignment at the time of surgery using standardized assessment tools (Global Operative Assessment of Laparoscopic Skills and Operative Performance Rating System). The intervention group had access to their video recordings on a web-based platform for review and self-assessment using the same instruments. The primary outcome for the estimation of sample size was the difference in faculty-assessed final operative performance (third submitted case). Feasibility criteria included >85% participation, >85% adherence to case submission and >85% completion of self-assessment. RESULTS Of 37 eligible trainees approached, 32 consented and were randomized (86%). There were 16 in the intervention group, 15 in the control group (55% male, 55% junior trainees), and 1 was excluded for protocol violation. Twenty-four (75%) of participants submitted 3 cases. Thirteen trainees (81%) accessed the platform and completed 26 (63.2%) case self-assessments. Fifty-five trainees per arm will be needed to power a full-scale laparoscopic cholecystectomy with Global Operative Assessment of Laparoscopic Skills and 130 trainees per arm with Operative Performance Rating System as the assessment tool. CONCLUSION This pilot study contributes important data to inform the design of an adequately powered randomized controlled trial of video-based self-assessment to improve trainee performance of laparoscopic cholecystectomy. Although a priori trial feasibility criteria were not achieved, automated video capture and storage could significantly improve adherence in future trials.
Collapse
Affiliation(s)
- Saba Balvardi
- Department of Surgery, McGill University, Montreal, Quebec, Canada; Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, Quebec, Canada
| | - Pepa Kaneva
- Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, Quebec, Canada
| | - Koorosh Semsar-Kazerooni
- Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, Quebec, Canada
| | - Melina Vassiliou
- Department of Surgery, McGill University, Montreal, Quebec, Canada; Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, Quebec, Canada
| | | | - Carmen Mueller
- Department of Surgery, McGill University, Montreal, Quebec, Canada; Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, Quebec, Canada
| | - Julio F Fiore
- Department of Surgery, McGill University, Montreal, Quebec, Canada; Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, Quebec, Canada
| | - Kevin Schwartzman
- Respiratory Division, Department of Medicine, McGill University and McGill International Tuberculosis Centre, Research Institute of the McGill University Health Centre, Montreal, Quebec, Canada
| | - Liane S Feldman
- Department of Surgery, McGill University, Montreal, Quebec, Canada; Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, Quebec, Canada.
| |
Collapse
|
6
|
Wu J, Hines OJ. Using Artificial Intelligence to Assess Surgeon Skill. JAMA Surg 2023; 158:e231140. [PMID: 37285127 DOI: 10.1001/jamasurg.2023.1140] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Affiliation(s)
- James Wu
- Department of Surgery, UCLA David Geffen School of Medicine, Los Angeles, California
| | - O Joe Hines
- Department of Surgery, UCLA David Geffen School of Medicine, Los Angeles, California
| |
Collapse
|
7
|
Larkins K, Khan M, Mohan H, Warrier S, Heriot A. A systematic review of video-based educational interventions in robotic surgical training. J Robot Surg 2023:10.1007/s11701-023-01605-y. [PMID: 37097494 DOI: 10.1007/s11701-023-01605-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/20/2023] [Accepted: 04/18/2023] [Indexed: 04/26/2023]
Abstract
As robotic surgical procedures become more prevalent in practice, there is a demand for effective and efficient educational strategies in robotic surgery. Video has been used in open and laparoscopic surgery to instruct trainees in the acquisition of operative knowledge and surgical skill. Robotic surgery is an ideal application of video-based technology given the access of video recording directly from the console. This review will present the evidence base for video-based educational tools in robotic surgery to guide the development of future educational interventions using this technology. A systematic review of the literature was performed using the key words "video" "robotic surgery" and "education". From a total of 538 results, 15 full text articles were screened. Inclusion criteria were the presentation of an educational intervention using video and the application of this intervention to robotic surgery. The results of 10 publications are presented in this review. Analysis of the key concepts presented in these publications revealed three themes: video as technology, video as instruction, video as feedback. All studies showed a video-based learning had a positive effect on educational outcomes. There are limited published studies looking specifically at the use of video as an educational intervention in robotic surgical training. Existing studies primarily focus on the use of video as a review tool for skill development. There is scope to expand the use of robotic video as a teaching tool through adaptation of novel technology such as 3D headsets and concepts of cognitive simulation including guided mental imagery and verbalisation.
Collapse
Affiliation(s)
- Kirsten Larkins
- Peter MacCallum Cancer Centre, 300 Grattan Street, Melbourne, VIC, 3000, Australia.
| | | | - Helen Mohan
- Department of Colorectal Surgery, Austin Health, Heidelberg, VIC, Australia
| | - Satish Warrier
- Peter MacCallum Cancer Centre, 300 Grattan Street, Melbourne, VIC, 3000, Australia
- International Medical Robotics Academy, North Melbourne, VIC, Australia
| | - Alexander Heriot
- Peter MacCallum Cancer Centre, 300 Grattan Street, Melbourne, VIC, 3000, Australia
- International Medical Robotics Academy, North Melbourne, VIC, Australia
| |
Collapse
|