Enhanced Training Benefits of Video Recording Surgery With Automated Hand Motion Analysis

Nonphysician Evaluators and Recording-Based Tools in Surgical Skill Assessment: A Feasibility Study

OBJECTIVES

To investigate the feasibility of nonmedically trained evaluators and image- and video-based tools in the assessment of surgical skills in a key orthopedic procedure.

DESIGN

Orthopedic surgeons at varying skill levels were evaluated by their ability to repair a cadaveric bi-malleolar ankle fracture. Nonphysician viewers and expert orthopedic surgeons independently scored video recordings and fluoroscopy images of the procedure through Global Rating Scales (GRS) and procedure-specific checklist tools. Statistical analysis was used to determine if the evaluators and assessment tools were able to differentiate skill level.

SETTING

An academic tertiary care hospital.

PARTICIPANTS

The surgical procedure was completed by 3 orthopedic residents, 3 orthopedic trauma fellows, and 4 orthopedic trauma attending surgeons. The procedure was independently evaluated by 2 orthopedic surgeons and 2 nonphysicians.

RESULTS

Operating participants were stratified by ≤ or >10 bimalleolar ankle fracture cases performed alone (inexperienced, n = 5 vs experienced, n = 5). Expert surgeon viewers could effectively stratify skill group through the GRS for video and fluoroscopy analysis (p < 0.05), and the video procedure-specific checklist (p < 0.05), but not the fluoroscopy procedure-specific checklist. Nonphysician viewers generally recognized skill groupings, although with less separation than surgeon viewers. These evaluators performed the best when aided by video and fluoroscopy procedure-specific checklists. Meanwhile, breakdowns of each tool into critical zones for improvement and evaluator-independent metrics such as case experience, self-reported confidence, and surgical time also indicated some skill differentiation.

CONCLUSIONS

The feasibility of using video recordings and fluoroscopic imaging based surgical skills assessment tools in orthopedic trauma was demonstrated. The tools highlighted in this study are applicable to both cadaver laboratory settings and live surgeries. The degree of training that is required by the evaluators and the utility of measuring surgical times of specific tasks should be the subject of future studies.

Characterization of Surgical Movements As a Training Tool for Improving Efficiency

INTRODUCTION

Surgical experience is often reflected by efficient, fluid, and well-calculated movements. For a new trainee, learning these characteristics is possible only by observation as there is no quantification system to define these factors. We analyzed surgeons' hand movements with different experience levels to characterize their movements according to experience.

METHODS

Hand motions were recorded by an inertial measurement unit (IMU) mounted on the hands of the surgeons during a simulated surgical procedure. IMU data provided acceleration and Eulerian angles: yaw, roll, and pitch corresponding to hand motions as radial/ulnar deviation, pronation/supination, and extension/flexion, respectively. These variables were graphically depicted and compared between three surgeons.

RESULTS

Participants were assigned to three groups based on years of surgical experience: group 1: >15 y; group 2: 3-10 y; and group 3: 0-1 y. Visualization of the roll motion, being the main motion during suturing, showed the clear difference in fluidity and regularity of the movements between the groups, showing minimal wasted movements for group 1. The angle of the roll motion, measured at the minimum, midpoint, and maximum points was significantly different between the groups. As expected, the experienced group completed the procedure first; however, the acceleration was not different between the groups.

CONCLUSIONS

Surgeons' hand movements can be easily characterized and quantified by an IMU device for automatic assessment of surgical skills. These characteristics graphically visualize a surgeon's regularity, fluidity, economy, and efficiency. The characteristics of an experienced surgeon can serve as a training model and as a reference tool for trainees.

Pilot Analysis of Surgeon Instrument Utilization Signatures Based on Shannon Entropy and Deep Learning for Surgeon Performance Assessment in a Cadaveric Carotid Artery Injury Control Simulation

BACKGROUND AND OBJECTIVES

Assessment and feedback are critical to surgical education, but direct observational feedback by experts is rarely provided because of time constraints and is typically only qualitative. Automated, video-based, quantitative feedback on surgical performance could address this gap, improving surgical training. The authors aim to demonstrate the ability of Shannon entropy (ShEn), an information theory metric that quantifies series diversity, to predict surgical performance using instrument detections generated through deep learning.

METHODS

Annotated images from a publicly available video data set of surgeons managing endoscopic endonasal carotid artery lacerations in a perfused cadaveric simulator were collected. A deep learning model was implemented to detect surgical instruments across video frames. ShEn score for the instrument sequence was calculated from each surgical trial. Logistic regression using ShEn was used to predict hemorrhage control success.

RESULTS

ShEn scores and instrument usage patterns differed between successful and unsuccessful trials (ShEn: 0.452 vs 0.370, P < .001). Unsuccessful hemorrhage control trials displayed lower entropy and less varied instrument use patterns. By contrast, successful trials demonstrated higher entropy with more diverse instrument usage and consistent progression in instrument utilization. A logistic regression model using ShEn scores (78% accuracy and 97% average precision) was at least as accurate as surgeons' attending/resident status and years of experience for predicting trial success and had similar accuracy as expert human observers.

CONCLUSION

ShEn score offers a summative signal about surgeon performance and predicted success at controlling carotid hemorrhage in a simulated cadaveric setting. Future efforts to generalize ShEn to additional surgical scenarios can further validate this metric.