Assessing Wire Navigation Performance in the Operating Room

Fluoroscopic image-based behavior analysis can objectively explain subjective expert assessment of wire navigation skill

Psychomotor skill and decision-making efficiency in surgical wire navigation can be objectively evaluated by analysis of intraoperative fluoroscopic image sequences. Prior work suggests that such image-based behavior analysis of operating room (OR) performance can predict performer experience level (R2  = 0.62) and agree with expert opinion (the current standard) on the quality of a final implant construct (R2  = 0.59). However, it is unclear how objective image-based evaluation compares with expert assessments for entire technical OR performances. This study examines the relationships between three key variables: (1) objective image-based criteria, (2) expert opinions, and (3) performing surgeon experience level. A paired-comparison survey of seven experts, based upon eight OR fluoroscopic wire navigation image sequences, shows that the experts' preferences are best explained by objective metrics that reflect psychomotor and decision-making behaviors which are counter-productive to successful implant placement, like image count (R2  = 0.83) and behavior tally (R2  = 0.74). One such behavior, adjustments away from goal, uniquely correlated well with all three key variables: a fluoroscopic image-based analysis composite score (R2  = 0.40), expert consensus (R2  = 0.76), and performer experience (R2  = 0.41). These results confirm that experts view less efficient technical behavior as indicative of lesser technical proficiency. While expert assessments of technical skill were reliable and consistent, neither individual nor consensus expert opinion appears to correlate with performer experience (R2  = 0.11).

The Role of Online Videos in Teaching Procedural Skills in Postgraduate Medical Education: A Scoping Review

OBJECTIVE

The purpose of this scoping review was to outline the extent of available literature including the prevalence of video quality appraisal tools, characterize how online videos were used, and identify the gaps in the literature with implications for future research.

DESIGN

The review was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses for scoping reviews (PRISMA-ScR). Six databases were searched: (1) MEDLINE (Ovid), (2) EMBASE, (3) Cochrane, (4) ERIC, (5) CINAHL PLUS, and (6) Google Scholar for Medical Subject Headings terms online videos, postgraduate health education, and health professional/s.

RESULTS

A total of 6948 articles were identified, of which 78 were included in the review. The articles included were primarily either experimental or observational studies, with most being from North America. Twelve concepts were identified: (1) procedural skills teaching; (2) video assessment; (3) validation of an assessment tool; (4) video feedback; (5) coaching; (6) broadcasting; (7) learner characteristics; (8) video characteristics; (9) video quality; (10) a quality assessment tool; (11) platforms and video library; and (12) health information governance.

CONCLUSIONS

While there is a diverse and growing body of literature on this area, information is lacking about the quality appraisal of online videos.

Minimally Trained Analysts Can Perform Fast, Objective Assessment of Orthopedic Technical Skill from Fluoroscopic Images

Skill assessment in orthopedics has traditionally relied on subjective impressions from a supervising surgeon. The feedback derived from these tools may be limited by bias and other practical issues. Objective analysis of intraoperative fluoroscopic images offers an inexpensive, repeatable, and precise assessment strategy without bias. Assessors generally refrain from using the scores of images obtained throughout the operation to evaluate skill for practical reasons. A new system was designed to facilitate rapid analysis of this fluoroscopy via minimally trained analysts. Four expert and four novice analysts independently measured one objective metric for skill using both a custom analysis software and a commercial alternative. Analysts were able to measure the objective metric three times faster when using the custom software, and without a practical difference in accuracy in comparison to the expert analysts using the commercial software. These results suggest that a well-designed fluoroscopy analysis system can facilitate inexpensive, reliable, and objective assessment of surgical skills.

Surgical Skill Can be Objectively Measured From Fluoroscopic Images Using a Novel Image-based Decision Error Analysis (IDEA) Score

BACKGROUND

To advance orthopaedic surgical skills training and assessment, more rigorous and objective performance measures are needed. In hip fracture repair, the tip-apex distance is a commonly used summative performance metric with clear clinical relevance, but it does not capture the skill exercised during the process of achieving the final implant position. This study introduces and evaluates a novel Image-based Decision Error Analysis (IDEA) score that better captures performance during fluoroscopically-assisted wire navigation.

QUESTIONS/PURPOSES

(1) Can wire navigation skill be objectively measured from a sequence of fluoroscopic images? (2) Are skill behaviors observed in a simulated environment also exhibited in the operating room? Additionally, we sought to define an objective skill metric that demonstrates improvement associated with accumulated surgical experience.

METHODS

Performance was evaluated both on a hip fracture wire navigation simulator and in the operating room during actual fracture surgery. After examining fluoroscopic image sequences from 176 consecutive simulator trials (performed by 58 first-year orthopaedic residents) and 21 consecutive surgical procedures (performed by 19 different orthopaedic residents and one attending orthopaedic surgeon), three main categories of erroneous skill behavior were identified: off-target wire adjustments, out-of-plane wire adjustments, and off-target drilling. Skill behaviors were measured by comparing wire adjustments made between consecutive images against the goal of targeting the apex of the femoral head as part of our new IDEA scoring methodology. Decision error metrics (frequency, magnitude) were correlated with other measures (image count and tip-apex distance) to characterize factors related to surgical performance on both the simulator and in the operating room. An IDEA composite score integrating decision errors (off-target wire adjustments, out-of-plane wire adjustments, and off-target drilling) and the final tip-apex distance to produce a single metric of overall performance was created and compared with the number of hip wire navigation cases previously completed (such as surgeon experience levels).

RESULTS

The IDEA methodology objectively analyzed 37,000 images from the simulator and 688 images from the operating room. The number of decision errors (7 ± 5 in the operating room and 4 ± 3 on the simulator) correlated with fluoroscopic image count (33 ± 14 in the operating room and 20 ± 11 on the simulator) in both the simulator and operating room environments (R2 = 0.76; p < 0.001 and R2 = 0.71; p < 0.001, respectively). Decision error counts did not correlate with the tip-apex distance (16 ± 4 mm in the operating room and 12 ± 5 mm on the simulator) for either the simulator or the operating room (R2 = 0.08; p = 0.15 and R2 = 0.03; p = 0.47, respectively), indicating that the tip-apex distance is independent of decision errors. The IDEA composite score correlated with surgical experience (R2 = 0.66; p < 0.001).

CONCLUSION

The fluoroscopic images obtained in the course of placing a guide wire contain a rich amount of information related to surgical skill. This points the way to an objective measure of skill that also has potential as an educational tool for residents. Future studies should expand this analysis to the wide variety of procedures that rely on fluoroscopic images.

CLINICAL RELEVANCE

This study has shown how resident skill development can be objectively assessed from fluoroscopic image sequences. The IDEA scoring provides a basis for evaluating the competence of a resident. The score can be used to assess skill at key timepoints throughout residency, such as when rotating onto/off of a new surgical service and before performing certain procedures in the operating room, or as a tool for debriefing/providing feedback after a procedure is completed.

Surgical data recording in the operating room: a systematic review of modalities and metrics

INTRODUCTION

Operating room recording, via video, audio and sensor-based recordings, is increasingly common. Yet, surgical data science is a new field without clear guidelines. The purpose of this study is to examine existing published studies of surgical recording modalities to determine which are available for use in the operating room, as a first step towards developing unified standards for this field.

METHODS

Medline, EMBASE, CENTRAL and PubMed databases were systematically searched for articles describing modalities of data collection in the operating room. Search terms included 'video-audio media', 'bio-sensing techniques', 'sound', 'movement', 'operating rooms' and others. Title, abstract and full-text screening were completed to identify relevant articles. Descriptive statistical analysis was performed for included studies.

RESULTS

From 3756 citations, 91 studies met inclusion criteria. These studies described 10 unique data-collection modalities for 17 different purposes in the operating room. Data modalities included video, audio, kinematic and eye-tracking among others. Data-collection purposes described included surgical trainee assessment, surgical error, surgical team communication and operating room efficiency.

CONCLUSION

Effective data collection and utilization in the operating room are imperative for the provision of superior surgical care. The future operating room landscape undoubtedly includes multiple modalities of data collection for a plethora of purposes. This review acts as a foundation for employing operating room data in a way that leads to meaningful benefit for patient care.

A Vision for Using Simulation & Virtual Coaching to Improve the Community Practice of Orthopedic Trauma Surgery

BACKGROUND

Many orthopedic surgeries involve the challenging integration of fluoroscopic image interpretation with skillful tool manipulation to enable procedures to be performed through less invasive approaches. Simulation has proved beneficial for teaching and improving these skills for residents, but similar benefits have not yet been realized for practicing orthopedic surgeons. A vision is presented to elevate community orthopedic practice and improve patient safety by advancing the use of simulators for training and assessing surgical skills.

METHODS

Key elements of this vision that are established include 1) methods for the objective and rigorous assessment of the performance of practicing surgeons now exist, 2) simulators are sufficiently mature and sophisticated that practicing surgeons will use them, and 3) practicing surgeons can improve their performance with appropriate feedback and coaching.

RESULTS

Data presented indicate that surgical performance can be adequately and comparably measured using structured observations made by experts or non-expert crowds, with the crowdsourcing approach being more expedient and less expensive. Rigorous measures of the surgical result and intermediate objectives obtained semi-automatically from intra-operative fluoroscopic image sequences can distinguish performances of experts from novices. Experience suggests that practicing orthopedic surgeons are open to and can be constructively engaged by a family of mature simulators as a means to evaluate and improve their surgical skills.

CONCLUSIONS

The results presented support our contention that new objective assessment measures are sufficient for evaluating the performance of working surgeons. The novel class of orthopedic surgical simulators available were tested and approved by practicing physicians. There exists a clear opportunity to combine purpose-designed simulator exercises with virtual coaching to help practicing physicians retain, retrain, and improve their technical skills. This will ultimately reduce cost, increase the quality of care, and decrease complication rates.

CLINICAL RELEVANCE

This vision articulates a means to boost the confidence of practitioners and ease their anxiety so that they perform impactful procedures more often in community hospitals, which promises to improve treatment and reduce the cost of care while keeping patients closer to their homes and families.

Do Skills Acquired from Training with a Wire Navigation Simulator Transfer to a Mock Operating Room Environment?

BACKGROUND

Skills training and simulation play an increasingly important role in orthopaedic surgical education. The intent of simulation is to improve performance in the operating room (OR), a trait known as transfer validity. No prior studies have explored how simulator-based wire navigation training can transfer to higher-level tasks. Additionally, there is a lack of knowledge on the format in which wire navigation training should be deployed.

QUESTIONS/PURPOSES

(1) Which training methods (didactic content, deliberate practice, or proficiency-based practice) lead to the greatest improvement in performing a wire navigation task? (2) Does a resident's performance using a wire navigation simulator correlate with his or her performance on a higher-level simulation task in a mock OR involving a C-arm, a radiopaque femur model, and a large soft tissue surrogate surrounding the femur?

METHODS

Fifty-five residents from four different medical centers participated in this study over the course of 2 years. The residents were divided into three groups: traditional training (included first-year residents from the University of Iowa, University of Minnesota, and the Mayo Clinic), deliberate practice (included first-year residents from the University of Nebraska and the University of Minnesota), and proficiency training (included first-year residents from the University of Minnesota and the Mayo Clinic). Residents in each group received a didactic introduction covering the task of placing a wire to treat an intertrochanteric fracture, and this was considered traditional training. Deliberate practice involved training on a radiation-free simulator that provided specific feedback throughout the practice sessions. Proficiency training used the same simulator to train on specific components of wire navigation, like finding the correct starting point, to proficiency before moving to assessment. The wire navigation simulator uses a camera system to track the wire and provide computer-generated fluoroscopy. After training, task performance was assessed in a mock OR. Residents from each group were assessed in the mock OR based on their use of fluoroscopy, total time, and tip-apex distance. Correlation analysis was performed to examine the relationship between resident performance on the simulator and in the mock OR.

RESULTS

Residents in the two simulation-based training groups had a lower tip-apex distance than those in the traditional training group (didactic training tip-apex distance: 24 ± 7 mm, 95% CI, 20-27; deliberate practice tip-apex distance: 16 ± 5 mm, 95% CI, 13-19, p = 0.001; proficiency training tip-apex distance: 15 ± 4 mm, 95% CI, 13-18, p < 0.001). Residents in the proficiency training group used more images than those in the other groups (didactic training: 22 ± 12 images, p = 0.041; deliberate practice: 19 ± 8 images; p = 0.012, proficiency training: 31 ± 14 images). In the two simulation-based training groups, resident performance on the simulator, that is, tip-apex distance, image use, and overall time, was correlated with performance in the mock OR (r-square = 0.15 [p = 0.030], 0.61 [p < 0.001], and 0.43 [p < 0.001], respectively).

CONCLUSIONS

As residency programs are designing their curriculum to train wire navigation skills, emphasis should be placed on providing an environment that allows for deliberate practice with immediate feedback about their performance. Simulators such as the one presented in this study offer a safe environment for residents to learn this key skill.

LEVEL OF EVIDENCE

Level II, therapeutic study.

Developing an objective assessment of surgical performance from operating room video and surgical imagery

An unbiased, repeatable process for assessing operating room performance is an important step toward quantifying the relationship between surgical training and performance. Hip fracture surgeries offer a promising first target in orthopedic trauma because they are common and they offer quantitative performance metrics that can be assessed from video recordings and intraoperative fluoroscopic images. Hip fracture repair surgeries were recorded using a head-mounted point-of-view camera. Intraoperative fluoroscopic images were also saved. The following performance metrics were analyzed: duration of wire navigation, number of fluoroscopic images collected, degree of intervention by the surgeon's supervisor, and the tip-apex distance (TAD). Two orthopedic traumatologists graded surgical performance in each video independently using an Objective Structured Assessment of Technical Skill (OSATS). Wire navigation duration correlated with weeks into residency and prior cases logged. TAD correlated with cases logged. There was no significant correlation between the OSATS total score and experience metrics. Total OSATS score correlated with duration and number of fluoroscopic images. Our results indicate that two metrics of hip fracture wire navigation performance, duration and TAD, significantly differentiate surgical experience. The methods presented have the potential to provide truly objective assessment of resident technical performance in the OR.