Subjective vs Computerized Assessment of Surgeon Skill Level During Mastoidectomy

Convolutional Neural Network-Based Deep Learning Engine for Mastoidectomy Instrument Recognition and Movement Tracking

OBJECTIVE

To develop a convolutional neural network-based computer vision model to recognize and track 2 mastoidectomy surgical instruments-the drill and the suction-irrigator-from intraoperative video recordings of mastoidectomies.

STUDY DESIGN

Technological development and model validation.

SETTING

Academic center.

METHODS

Ten 1-minute videos of mastoidectomies done for cochlear implantation by varying levels of resident surgeons were collected. For each video, containing 900 frames, an open-access computer vision annotation tool was used to annotate the drill and suction-irrigator class images with bounding boxes. A mastoidectomy instrument tracking module, which extracts the center coordinates of bounding boxes, was developed using a feature pyramid network and layered with DETECTRON, an open-access faster-region-based convolutional neural network. Eight videos were used to train the model, and 2 videos were used for testing. Outcome measures included Intersection over Union (IoU) ratio, accuracy, and average precision.

RESULTS

For an IoU of 0.5, the mean average precision for the drill was 99% and 86% for the suction-irrigator. The model proved capable of generating maps of drill and suction-irrigator stroke direction and distance for the entirety of each video.

CONCLUSIONS

This computer vision model can identify and track the drill and suction-irrigator from videos of intraoperative mastoidectomies performed by residents with excellent precision. It can now be employed to retrospectively study objective mastoidectomy measures of expert and resident surgeons, such as drill and suction-irrigator stroke concentration, economy of motion, speed, and coordination, setting the stage for characterization of objective expectations for safe and efficient mastoidectomies.

A Data-Driven Methodology to Comprehensively Assess Bone Drilling Using Radar Plots

Background

The study aims to develop a data-driven methodology to assess bone drilling in preparation for future clinical trials in residency training. The existing assessment methods are either subjective or do not consider the interdependence among individual skill factors, such as time and accuracy. This study uses quantitative data and radar plots to visualize the balance of the selected skill factors.

Methods

In the experiment, straight vertical drilling was assessed across 3 skill levels: expert surgeons (N = 10), intermediate residents (postgraduate year-2-5, N = 5), and novice residents (postgraduate year-1, N = 10). Motion and force were measured for each drilling trial, and data from multiple trials were then converted into 5 performance indicators, including overshoot, drilling time, overshoot consistency, time consistency, and force fluctuation. Each indicator was then scored between 0 and 10, with 10 being the best, and plotted into a radar plot.

Results

Statistical difference (p < 0.05) was confirmed among 3 skill levels in force, time, and overshoot data. The radar plots revealed that the novice group exhibited the most distorted pentagons compared with the well-formed pentagons observed in the case of expert participants. The intermediate group showed slight distortion that was between the expert and novice groups.

Conclusion/Clinical Relevance

This research shows the utility of radar plots in drilling assessment in a comprehensive manner and lays the groundwork for a data-driven training scheme to prepare novice residents for clinical practice.

Learning Outcomes and Educational Effectiveness of E-Learning as a Continuing Professional Development Intervention for Practicing Surgeons and Proceduralists: A Systematic Review

BACKGROUND

Electronic learning (e-Learning) has become a prevalent mode of delivering medical education. We aimed to determine the learning outcomes and educational effectiveness of e-Learning as a continuing professional development (CPD) intervention for practicing surgeons and proceduralists.

METHODS

We searched MEDLINE databases and included studies reporting learning outcomes of e-learning CPD interventions for practicing surgeons and physicians performing technical procedures. We excluded articles only studying surgical trainees and those not reporting learning outcomes. Two reviewers independently screened, extracted data, and assessed study quality using the Critical Appraisal Skills Programme (CASP) tools. Learning outcomes and educational effectiveness were categorized using Moore's Outcomes Framework (PROSPERO: CRD42022333523).

RESULTS

Of 1307 identified articles, 12 were included- 9 cohort studies, one randomized controlled trial and 2 qualitative studies, with a total of 2158 participants. Eight studies were rated as moderate, five as strong, and 2 as weak in study quality. E-Learning CPD interventions included web-based modules, image recognition, videos, a repository of videos and schematics, and an online journal club. Seven studies reported participants' satisfaction with the e-Learning interventions (Moore's Level 2), 4 reported improvements in participants' declarative knowledge (Level 3a), 1 reported improvements in procedural knowledge (Level 3b) and five reported improvements in participants' procedural competence in an educational setting (Level 4). No studies demonstrated improvements in participants' workplace-based performance, the health of patients, or community health (Levels 5-7).

CONCLUSIONS

E-Learning as a CPD educational intervention is associated with high satisfaction and improvements in knowledge and procedural competencies of practicing surgeons and proceduralists in an educational setting. Future research is required to investigate whether e-Learning is associated with higher-level learning outcomes.

Intraoperative video recording in otolaryngology for surgical education: evolution and considerations

BACKGROUND

Otolaryngology is a surgical speciality well suited for the application of intraoperative video recording as an educational tool considering the number procedures within the speciality that utilize digital technology. Intraoperative recording has been utilized in endoscopic surgeries and in evaluating technique in mastoidectomy, myringotomy and grommet insertion. The impact of intra-operative video recording in otolaryngology education is vast in creating access to surgical videos for preparation outside the operating room to individualized coaching and assessment. The purpose of this project is to highlight the role of intraoperative video recording in otolaryngology training and elucidate the challenges and considerations associated with implementation.

METHODS

Related publications between 1999 to 2022 were reviewed from PubMed and Embase databases utilizing search terms "intraoperative videography," "video recording surgery," "otolaryngology," and "surgical education." 109 articles were screened independently by HB and SK, by title and abstract then full text review. 28 articles from the original search and 6 from the secondary reference review were included.

RESULTS

The application of intraoperative video recording is evident in otolaryngology surgeries including endoscopic sinus surgery, laryngeal surgery, and other endoscopic procedures. There have been significant advancements in recording tools, including devices that can capture the surgeon's perspective. The considerations and challenges identified with utilizing this educational tool were categorized into different themes including ethics/consent, regulation, liability, data, technology, and human resources.

CONCLUSION

Intra-operative video recording has been demonstrated to have significant impact within otolaryngology education. It is critical to elucidate the challenges and considerations involved to utilize this educational tool effectively. Future directives will see video-based performance analytics providing comparative metrics to encourage precise coaching of surgical residents.

Video Analysis of Otologic Instrument Movement During Resident Mastoidectomies

OBJECTIVE

To measure surgical instrument movement during resident mastoidectomies and identify metrics that correlate with experience.

STUDY DESIGN

Retrospective case series.

SETTING

Tertiary care center.

SUBJECTS

Ten postgraduate year (PGY) 2, 6 PGY3, 7 PGY4, and 19 PGY5 recordings of mastoidectomy performed by otolaryngology residents.

INTERVENTIONS

One-minute intraoperative recordings of mastoidectomies performed during cochlear implantation were collected. Drill and suction-irrigator motion were analyzed with sports motion tracking software.

MAIN OUTCOME MEASURES

Mean instrument speed, angle, and angular velocity were calculated. Mann-Whitney U tests compared mean instrument metrics between PGY levels. Change in drill speed for seven residents between their PGY2 to PGY5 years was individually analyzed.

RESULTS

Mean drill speed was significantly greater for PGY5 residents compared with PGY2s (2.9 versus 1.8 cm/s, p = 0.001). Compared with PGY2 residents, suction speed was greater as a PGY5 (1.2 versus 0.9 cm/s; p = 0.201) and significantly greater as a PGY4 (1.5 versus 0.9 cm/s, p = 0.039). Of the seven residents individually analyzed, group mean drill speed increased by 0.4 cm/s, yearly.

CONCLUSIONS

Drill and suction-irrigator movement during the second minute of drilling of a cortical mastoidectomy seems to increase with resident level. Objective video analysis is a potential adjunct for differentiating novices from more experienced surgeons and monitoring surgical skills progress.

Using Intraoperative Recordings to Evaluate Surgical Technique and Performance in Mastoidectomy

Importance

Otolaryngology residency programs currently lack rigorous methods for assessing surgical skill and often rely on biased tools of evaluation.

Objectives

To evaluate which techniques used in mastoidectomy can serve as indicators of surgeon level (defined as the level of training) and whether these determinations of technique can be made based solely on the movement of the drill head or suction.

Design, Setting, and Participants

In this prospective, observational study conducted from January 1, 2015, to December 31, 2019, at a single tertiary care institution, 3 independent observers made blinded evaluations on 24 intraoperative recordings of surgeons (6 junior residents, 4 senior residents, and 2 attending surgeons) performing mastoidectomies.

Main Outcomes and Measures

Observers assessed drill stroke count, drilling efficiency, stroke pattern, use of suction and irrigation, and estimated surgeon level. Assessments were made on both original videos and animated videos that show only the path of the burr head or suction as dots against a white background.

Results

Among the 24 recorded mastoidectomies performed by the 12 study surgeons, intraclass correlation was excellent for original video assessment of drill stroke count (0.98 [95% CI, 0.97-1.00]), use of suction (0.75 [95% CI, 0.52-0.89]), use of irrigation (0.83 [95% CI, 0.66-0.92]), and estimated surgeon level (0.82 [95% CI, 0.64-0.92]) and fair for drilling efficiency (0.54 [95% CI, 0.09-0.79]) and stroke pattern (0.49 [95% CI, -0.02 to 0.76]). Intraclass correlation was excellent for animated video assessment of drill stroke count per unit time (0.98 [95% CI, 0.96-0.99]) and drilling efficiency (0.80 [95% CI, 0.60-0.91]), good for stroke pattern (0.68 [95% CI, 0.38-0.85]) and estimated surgeon level (based on path of drill) (0.69 [95% CI, 0.38-0.85]), and fair for use of suction (0.58 [95% CI, 0.16-0.80]) and estimated surgeon level (based on path of suction) (0.58 [95% CI, 0.17-0.80]). On evaluation of original videos, junior residents had lower drill stroke count compared with senior residents and attending surgeons (6.0 [interquartile range (IQR), 3.0-8.0] vs 9.5 [IQR, 5.0-13.0] vs 10.5 [IQR, 5.0-17.8]; η2 = 0.14 [95% CI, 0.01-0.28]). On evaluation of animated videos, junior residents also had lower drill stroke count compared with senior residents and attending surgeons (6.0 [IQR, 4.0-9.0] vs 10.5 [IQR, 10.0-13.8] vs 10.5 [IQR, 4.3-21.0]; η2 = 0.19 [95% CI, 0.04-0.33]). Compared with junior and senior residents, attending surgeons had higher median ratings of drilling efficiency (original videos: junior residents, 4.0 [IQR, 3.0-4.0]; senior residents, 4.0 [IQR, 3.0-4.8]; attending surgeons, 5.0 [IQR, 4.3-5.0]; η2 = 0.23 [95% CI, 0.06-0.37]; animated videos: junior residents, 4.0 [IQR, 3.0-4.0]; senior residents, 3.0 [IQR, 2.0-4.0]; attending surgeons, 5.0 [IQR, 4.0-5.0]; η2 = 0.25 [95% CI, 0.08-0.39]) and stroke pattern (original videos: junior residents, 4.0 [IQR, 3.0-4.0]; senior residents, 4.0 [IQR, 3.0-4.8]; attending surgeons, 5.0 [IQR, 5.0-5.0]; η2 = 0.17 [95% CI, 0.03-0.31]; animated videos: junior residents, 4.0 [IQR, 3.0-4.0]; senior residents, 4.0 [IQR, 2.0-4.0]; attending surgeons, 5.0 [IQR, 5.0-5.0]; η2 = 0.15 [95% CI, 0.02-0.29]).

Conclusions and Relevance

This study suggests that observation of intraoperative mastoidectomy recordings is a feasible method of evaluating surgeon level. Reasonable indicators of surgeon level include the drill stroke count, drilling efficiency, stroke pattern, and use of the suction irrigator. Observing the path of the drill alone is sufficient to appreciate differences in drilling technique but not sufficient to accurately determine surgeon level. Intraoperative recordings can serve as a useful addition to resident education and evaluation.