Objective Structured Assessments of Technical Skills (OSATS) Does Not Assess the Quality of the Surgical Result Effectively

Nonphysician Evaluators and Recording-Based Tools in Surgical Skill Assessment: A Feasibility Study

OBJECTIVES

To investigate the feasibility of nonmedically trained evaluators and image- and video-based tools in the assessment of surgical skills in a key orthopedic procedure.

DESIGN

Orthopedic surgeons at varying skill levels were evaluated by their ability to repair a cadaveric bi-malleolar ankle fracture. Nonphysician viewers and expert orthopedic surgeons independently scored video recordings and fluoroscopy images of the procedure through Global Rating Scales (GRS) and procedure-specific checklist tools. Statistical analysis was used to determine if the evaluators and assessment tools were able to differentiate skill level.

SETTING

An academic tertiary care hospital.

PARTICIPANTS

The surgical procedure was completed by 3 orthopedic residents, 3 orthopedic trauma fellows, and 4 orthopedic trauma attending surgeons. The procedure was independently evaluated by 2 orthopedic surgeons and 2 nonphysicians.

RESULTS

Operating participants were stratified by ≤ or >10 bimalleolar ankle fracture cases performed alone (inexperienced, n = 5 vs experienced, n = 5). Expert surgeon viewers could effectively stratify skill group through the GRS for video and fluoroscopy analysis (p < 0.05), and the video procedure-specific checklist (p < 0.05), but not the fluoroscopy procedure-specific checklist. Nonphysician viewers generally recognized skill groupings, although with less separation than surgeon viewers. These evaluators performed the best when aided by video and fluoroscopy procedure-specific checklists. Meanwhile, breakdowns of each tool into critical zones for improvement and evaluator-independent metrics such as case experience, self-reported confidence, and surgical time also indicated some skill differentiation.

CONCLUSIONS

The feasibility of using video recordings and fluoroscopic imaging based surgical skills assessment tools in orthopedic trauma was demonstrated. The tools highlighted in this study are applicable to both cadaver laboratory settings and live surgeries. The degree of training that is required by the evaluators and the utility of measuring surgical times of specific tasks should be the subject of future studies.

A Surgical Skills Rotation for Mid-Level Residents

Introduction

The University of Iowa orthopaedic residency previously designed a month-long surgical skill rotation for postgraduate year (PGY)-1 residents. This successful initiative has become a model of interest for other teaching institutions. In addition to the intern year, an important phase in residency occurs during the transition from PGY2 to PGY3, when residents assume greater responsibility and autonomy in leading surgical procedures.

Methods

To directly address this transition and assess residents' readiness, our residency program established a week-long PGY2 surgical skills rotation. This rotation serves as a training checkpoint and focuses on both training and evaluation of level-appropriate skills in joint arthroplasty, trauma, arthroscopy, and wire navigation. The primary objective of the PGY2 surgical skills rotation is to enhance orthopaedic residents' operative skills and experience by providing increased exposure and practice of requisite technical skills. Similar to the Orthopedic In-Training Examinations that assess residents' knowledge, this week-long program, aimed at assessing residents' proficiency in fundamental orthopaedic technical skills, occurs before their PGY3 year.

Results

Faculty-led training and assessment sessions in each area offer residents many opportunities for dedicated practice and improvement. Transferring these acquired skills from the laboratory to the operating room is essential for a training program. To confirm improvement, the final day of the surgical skills rotation was exclusively dedicated to structured performance evaluations, with a specific emphasis on establishing proficiency benchmarks.

Conclusion

We herein describe the University of Iowa's PGY2 surgical skills rotation, providing insights into its development, implementation, and outcomes. By sharing our experience, we offer a framework for other academic departments seeking to optimize surgical skills education and ensure the successful transition of mid-level residents.

Measuring Competency: Improving the validity of your procedural performance assessments

Objective

The objective of the study was to compare the use of ordinal scales and interval scales for capturing surgical competency information for general surgeons performing 3 complex trauma procedures.

Background

Surgical performance assessment is typically captured using nonparametric data (eg, checklists) that do not support inferential analyses. Interval scales support parametric analyses that are essential for determining competency. We compared assessment outcomes for surgeons performing 3 complex trauma procedures using ordinal and interval scales.

Methods

All participants were board-certified or eligible general surgeons. Each participant was assessed by an experienced trauma surgeon while performing 3 trauma procedures on cadavers. All assessors completed a rigorous assessment certification process. We calculated descriptive statistics to examine the differences between interval (parametric) and ordinal (nonparametric) outcomes.

Results

Ordinal scales overestimated competence in up to 100% of the participants and did not identify specific performance gaps. Interval scales provided more granularity and identified specific capability gaps.

Conclusions

Imprecise instrumentation conveys a false sense of competence and deprives surgeons of opportunities to close capability gaps. Measuring discrete procedural components with interval scales provides a more precise measurement of surgical competency.

Predicting Resident Competence for Otolaryngology Key Indicator Procedures

OBJECTIVE

Competency-based surgical education requires practical assessments and meaningful benchmarks. In otolaryngology, key indicator procedure (KIP) minima are indicators of surgical exposure during training, yet it remains unknown how many times trainees must be evaluated on KIPs to ensure operative competence. Herein, we used Bayesian mixed effects models to compute predicted performance expectations for KIPs.

METHODS

From November 2017 to September 2021, a smartphone application (SIMPL OR) was used by attendings at five otolaryngology training programs to rate resident operative performance after each case on a five-level scale. Bayesian mixed effects models were used to estimate the probability that postgraduate year (PGY) 3, 4, or 5 trainees would earn a "practice-ready" (PR) rating on a subsequent evaluation based on their previously earned PR ratings for each KIP. Probabilities of earning a subsequent PR rating were examined for interpretability, and cross-validation was used to assess predictive validity.

RESULTS

A total of 842 assessments of KIPs were submitted by 72 attendings for 92 residents PGY 2-5. The predictive model had an average Area Under the Receiver Operating Curve of 0.77. The number of prior PR ratings that senior residents needed to attain a 95% probability of earning a PR rating on a subsequent evaluation was estimated for each KIP. For example, for mastoidectomies, PGY4 residents needed to earn 10 PR ratings whereas PGY5 residents needed 4 PR ratings on average to have a 95% probability of attaining a PR rating on a subsequent evaluation.

CONCLUSION

Predictive modeling can inform assessment benchmarks for competency-based surgical education.

LEVEL OF EVIDENCE

NA Laryngoscope, 133:3341-3345, 2023.

Using video-based assessment (VBA) to document fellow improvement in safely completing the jejunojejunostomy portion of laparoscopic Roux-en-Y gastric bypass (RYGB) surgery

BACKGROUND

Surgical assessment instruments are used for formative and summative trainee evaluations. To characterize the features of existing instruments and a novel 12-item objective, procedure-specific assessment tool for Roux-en-Y Gastric Bypass (RYGB-OPSA), we evaluated the progress of a single surgical fellow over 17 consecutive surgeries.

METHODS

Seventeen consecutive RYGB videos completed between 8/2021 and 1/2022 by an academic hospital surgical fellow were de-identified and assessed by four board-certified bariatric surgeons using Global Operative Assessment of Laparoscopic Skills (GOALS), General Assessment of Surgical Skill (GASS), and RYGB-OPSA which includes the reflection of transverse colon, identification of ligament of Treitz, biliopancreatic and Roux limbs orientation, jejunal division point selection, stapler use, mesentery division, bleeding control, jejunojejunostomy (JJ) anastomotic site selection, apposition of JJ anastomotic site, JJ creation, common enterotomy closure of JJ, and integrity of anastomosis. The GASS measured economy of motion, tissue handling, appreciating operative anatomy, bimanual dexterity, and achievement of hemostasis. RYGB-OPSA and GASS items were scored "poor-unsafe," "acceptable-safe," or "good-safe." Change in performance was measured by linear trendline slope.

RESULTS

Over the course of 17 procedures, significant improvement was demonstrated by three GOALS items, GOALS overall score, GASS bimanual dexterity, and three RYGB-OPSA tasks: JJ creation, jejunal division point selection, and stapler use. Achievement of hemostasis declined but never rated "poor-unsafe." Overall RYGB-OPSA and GOALS trendlines documented significant increase across the 17 procedures.

CONCLUSION

This examination of a bariatric surgery fellow's operative training experience as measured by three surgical assessment instruments demonstrated anticipated improvements in general skills and safe completion of procedure-specific tasks. Effective surgical assessment instruments have enough sensitivity to show improvement to enable meaningful trainee feedback (low-stakes assessments) as well as the ability to determine safe surgical practice to enable promotion to greater autonomous practice.

Assessment of Technical Competence in Distal Radius Fracture Fixation by a Volar Locking Plate: A Global Delphi Consensus Study

PURPOSE

Volar locking plate fixation of distal radius fractures is a common orthopedic procedure and should be mastered by graduating orthopedic residents. Surgical education is transitioning from a traditional time-based approach to competency-based medical education. Valid and objective assessment is essential for successful transition. The purpose of this study was to develop a comprehensive, procedure-specific assessment tool to evaluate technical competence in volar locking plate osteosynthesis of a distal radius fracture.

METHODS

International orthopedic/trauma experts involved in resident education participated as panelists in a four-round online Delphi process to reach consensus on the content of the assessment tool. Round 1 was an item-generating round, in which the panelists identified potential assessment parameters. In round 2, the panelists rated the importance of each suggested assessment parameter and reached consensus on which to include in the assessment tool. Round 3 yielded specific assessment score intervals for specific bone and fracture models and is not reported in this study. In round 4, the panelists assigned weights to the assessment parameters on a 1-10 scale to determine how each parameter should have an impact on the overall results.

RESULTS

Eighty-seven surgeons, representing 42 countries, participated in the study. Round 1 resulted in 45 assessment parameters, grouped into five procedural steps. After round 2, the number of parameters was reduced to 39. After the final round, an additional parameter was removed and weights were assigned to the remaining parameters.

CONCLUSIONS

Using a systematic methodology, a preliminary assessment tool to evaluate technical competence in distal radius fracture fixation was developed. A consensus of international experts supports the content validity of the assessment tool.

CLINICAL RELEVANCE

This assessment tool represents the first step in the evidence-based assessment essential for competency-based medical education. Before implementation, further studies exploring validity of variations of the assessment tool in different educational contexts are required.

A Dedicated Simulator Training Curriculum Improves Resident Performance in Surgical Management of Pediatric Supracondylar Humerus Fractures

Background

The primary goal of including simulation in residency training is to improve technical skills while working outside of the operating room. Such simulation-related skill improvements have seldom been measured in the operating room. This is largely because uncontrolled variables, such as injury severity, patient comorbidity, and anatomical variation, can bias evaluation of an operating surgeon's skill. In this study, performance during the wire navigation phase of pediatric supracondylar humerus fracture fixation was quantitatively compared between 2 groups of orthopaedic residents: a standard training group consisting of residents who participated in a single simulator session of wire navigation training and an expanded training group consisting of residents who participated in a dedicated multifaceted wire navigation simulation training curriculum.

Methods

To evaluate performance in the operating room, the full sequence of fluoroscopic images collected during wire navigation was quantitatively analyzed. Objective performance metrics included number of fluoroscopic images acquired, duration from placement of the first wire to that of the final wire, and wire spread at the level of the fracture. These metrics were measured from 97 pediatric supracondylar humerus fracture pinning surgeries performed by 28 different orthopaedic residents.

Results

No differences were observed between the groups for wire spread in the final fluoroscopic images (t(94) = 0.75, p = 0.45), an important clinical objective of the surgery. Residents who received the expanded simulator training used significantly fewer fluoroscopic images (mean of 46 vs. 61 images, t(85) = 2.25, p < 0.03) and required less time from first to final wire placement (mean of 11.2 vs. 14.9 minutes, t(83) = 2.53, p = 0.013) than the standard training group. A post hoc review of Accreditation Council for Graduate Medical Education case logs for 24 cases from the standard training group and for 21 cases from the expanded training group indicated that, at the time of surgeries, residents who received expanded training had completed fewer comparable cases than residents in the standard training group (mean of 13 vs. 21, t(42) = 2.40 p = 0.02). Further regression analysis indicated that the expanded simulator training produced an effect comparable with that associated with completing 10.5 similar surgical case experiences.

Conclusions

This study demonstrates that training on a wire navigation simulator can lead to improved performance in the operating room on a critical skill for all orthopaedic residents. By taking fewer images and less time while maintaining sufficient pin spread, simulator-trained residents were objectively measured to have improved performance in comparison with residents who had not participated in the pediatric elbow simulator curriculum.

Clinical Relevance

As programs aim to provide safe and effective training for critical orthopaedic skills such as pinning a pediatric elbow, this study demonstrates a simulator curriculum that has demonstrated the transfer of skill from a learning environment to the operating room.

Computer-assisted simulated workplace-based assessment in surgery: application of the universal framework of intraoperative performance within a mixed-reality simulation

Objectives

Workplace-based assessment (WBA) is a key requirement of competency-based medical education in postgraduate surgical education. Although simulated workplace-based assessment (SWBA) has been proposed to complement WBA, it is insufficiently adopted in surgical education. In particular, approaches to criterion-referenced and automated assessment of intraoperative surgical competency in contextualized SWBA settings are missing.Main objectives were (1) application of the universal framework of intraoperative performance and exemplary adaptation to spine surgery (vertebroplasty); (2) development of computer-assisted assessment based on criterion-referenced metrics; and (3) implementation in contextualized, team-based operating room (OR) simulation, and evaluation of validity.

Design

Multistage development and assessment study: (1) expert-based definition of performance indicators based on framework's performance domains; (2) development of respective assessment metrics based on preoperative planning and intraoperative performance data; (3) implementation in mixed-reality OR simulation and assessment of surgeons operating in a confederate team. Statistical analyses included internal consistency and interdomain associations, correlations with experience, and technical and non-technical performances.

Setting

Surgical simulation center. Full surgical team set-up within mixed-reality OR simulation.

Participants

Eleven surgeons were recruited from two teaching hospitals. Eligibility criteria included surgical specialists in orthopedic, trauma, or neurosurgery with prior VP or kyphoplasty experience.

Main outcome measures

Computer-assisted assessment of surgeons' intraoperative performance.

Results

Performance scores were associated with surgeons' experience, observational assessment (Objective Structured Assessment of Technical Skill) scores and overall pass/fail ratings. Results provide strong evidence for validity of our computer-assisted SWBA approach. Diverse indicators of surgeons' technical and non-technical performances could be quantified and captured.

Conclusions

This study is the first to investigate computer-assisted assessment based on a competency framework in authentic, contextualized team-based OR simulation. Our approach discriminates surgical competency across the domains of intraoperative performance. It advances previous automated assessment based on the use of current surgical simulators in decontextualized settings. Our findings inform future use of computer-assisted multidomain competency assessments of surgeons using SWBA approaches.

Analyses of Orthopaedic Surgery Residency Interviews

Interviews are a critical component of orthopaedic surgery residency selection for both the applicant and the program. Some institutions no longer report Alpha Omega Alpha (AOA) designation or class rank, and US Medical Licensing Examination (USMLE) Step 1 recently switched to pass/fail scoring. During the coronavirus disease 2019 (COVID-19) pandemic, all Accreditation Council for Graduate Medical Education programs conducted virtual interviews and subinternship rotations were restricted. These changes offer significant challenges to the residency match process. The purpose of this study was to examine the residency applicant interview and ranking process at a large urban academic university setting. We hypothesized that large variability exists among evaluations submitted by faculty interviewers and also that applicant academic factors (i.e., USMLE Step 1 score) would show association with final ranking.

Methods

We retrospectively reviewed the 2020-2021 and 2021-2022 residency interview cycles, both conducted virtually due to the COVID-19 pandemic. Residency application (i.e., applicant demographic and academic backgrounds) and interview data (i.e., faculty interviewer scores) were recorded. Interobserver reliability among faculty interviewers was calculated. Statistical analysis was performed to determine factors associated with ranking of applicants.

Results

There were 195 included applicants from the 2020 and 2021 interview cycles. There was no true agreement of interviewers' scoring of shared applicants (kappa intraclass coefficient range 0-0.2). Applicant factors associated with being ranked include applying to the match for the first time, USMLE Step 1 and 2 scores, educational break (vs. consecutive completion of college and medical school in 4 years each), higher class rank, and greater interviewer scores. Factors associated with better rank included additional degrees (i.e., PhD or MBA), couples match, AOA designation, educational break, underrepresented minority status, and notable attributes (i.e., collegiate athletics or Eagle Scout participation). Factors associated with worse rank included male sex, international medical graduate, prior match history, science major, extended research (i.e., >1 year spent in a research role), and home medical school students.

Conclusions

There was significant variability and no reliability at our institution among faculty interviewers' applicant ratings. Being ranked was based more on academic record and interview performance while final rank number seemed based on applicant qualities. The removal of merit-based objective applicant measurements offers challenges to optimal residency applicant and program match.

Level of Evidence

III (retrospective cohort study).

Process-oriented metrics to provide feedback and assess the performance of students who are learning surgical procedures: The percutaneous dilatational tracheostomy case

PURPOSE

Assessing competency in surgical procedures is key for instructors to distinguish whether a resident is qualified to perform them on patients. Currently, assessment techniques do not always focus on providing feedback about the order in which the activities need to be performed. In this research, using a Process Mining approach, process-oriented metrics are proposed to assess the training of residents in a Percutaneous Dilatational Tracheostomy (PDT) simulator, identifying the critical points in the execution of the surgical process.

MATERIALS AND METHODS

A reference process model of the procedure was defined, and video recordings of student training sessions in the PDT simulator were collected and tagged to generate event logs. Three process-oriented metrics were proposed to assess the performance of the residents in training.

RESULTS

Although the students were proficient in classic metrics, they did not reach the optimum in process-oriented metrics. Only in 25% of the stages the optimum was achieved in the last session. In these stages, the four more challenging activities were also identified, which account for 32% of the process-oriented metrics errors.

CONCLUSIONS

Process-oriented metrics offer a new perspective on surgical procedures performance, providing a more granular perspective, which enables a more specific and actionable feedback for both students and instructors.

Validation of a Newly Developed Competency Assessment Tool for the Posterior Sagittal Anorectoplasty

INTRODUCTION

 The correction of an anorectal malformation (ARM) is complex and relatively infrequent. Simulation training and subsequent assessment may result in better clinical outcomes. Assessment can be done using a competency assessment tool (CAT). This study aims to develop and validate a CAT for the posterior sagittal anorectoplasty (PSARP) on a simulation model.

MATERIALS AND METHODS

 The CAT-PSARP was developed after consultation with experts in the field. The PSARP was divided into five steps, while tissue and instrument handling were scored separately. Participants of pediatric colorectal hands-on courses in 2019 and 2020 were asked to participate. They performed one PSARP procedure on an ARM simulation model, while being assessed by two objective observers using the CAT-PSARP.

RESULTS

 A total of 82 participants were enrolled. A fair interobserver agreement was found for general skills (intraclass correlation coefficient [ICC] = 0.524, p < 0.001), a good agreement for specific skills (ICC = 0.646, p < 0.001), and overall performance (ICC = 0.669, p < 0.001). The experienced group scored higher on all steps (p < 0.001), except for "anoplasty (p = 0.540)," compared with an inexperienced group.

CONCLUSION

 The CAT-PSARP is a suitable objective assessment tool for the overall performance of the included steps of the PSARP for repair of an ARM on a simulation model.

What's the evidence? A review of current instruction and assessment in veterinary surgical education

Veterinary surgical education is improved by studying the relevant literature. The aim of this literature review was to report the theory and methods used to provide surgical education to veterinary students; to discuss the training most likely to create a competent general practitioner; and to review assessment methods for simulated and live surgeries. The literature reviewed demonstrates that new graduates are expected to perform small animal sterilization, dental extraction, onychectomy, abscess treatment, aural hematoma repair, and mass removal with little or no supervision. Students require repetitive practice to reach competence; some practice can take place in a clinical skills laboratory on models or cadavers. When training novices, distributing practice over a longer time improves retention, but months without practice causes skills to decay. Suturing skills may be taught on models at a ratio of 1 instructor per 10 students. Veterinary students require 6-10 repetitions of each small animal sterilization surgery to reach competence; however, learning curves for other surgeries have not been established. Assessment of surgical skills has been undertaken using objective structured clinical examinations (OSCEs) and observed model and live surgeries. Additional validated rubrics are needed in the field. Surgical educators are urged to continue to search for innovative ways to foster deliberate practice, assess skill, and provide feedback for students.

Effectivity of Distance Learning in the Training of Basic Surgical Skills—A Randomized Controlled Trial

Background: Distance learning is an interactive way of education when teachers and students are physically separated. Our purpose was to examine its effectivity in training of basic surgical techniques and to provide an alternative sustainable methodology for the training of medical professionals. Methods: Sixty students were involved in our single blinded randomized controlled study. Six homogenized groups were created then randomized into three groups of distance learning and three groups of in-person teaching. The groups completed the same curriculum using our own “SkillBox”. All students took the same pre- and post-course test evaluated blindly. The students filled out an online feedback form after the course. A financial analysis was also made. Results: There was no significant difference in the post-course exam results (distance 28.200 vs. in-person 25.200). We managed to achieve significantly better improvements in the distance learning of suturing (distance 19.967 vs. in-person 15.900, p = 0.043). According to 93% of the study group students, the quality of teaching did not decrease compared to the traditional classes. Conclusion: The results of the students improved similarly in distance learning and in-person education. The online form of teaching was received positively among the students; they found it an effective and good alternative.

Implementación de las escalas de evaluación formativa OSATS en habilidades técnicas aplicadas en el laboratorio de microcirugía

Introducción: la simulación como estrategia de aprendizaje activo se ha constituido en una herramienta centrada en el estudiante, en la construcción de aprendizajes en entornos seguros, la creación y perfección de sus habilidades técnicas. Es allí donde la evaluación forma parte vital del proceso que busca identificar aprendizajes reales. Los OSATS son herramienta evaluativa en el área quirúrgica, implementable a los programas en microcirugía y escalas de desempeño en los modulos básicos. Metodología: estudio descriptivo, corte transversal y doble ciego; incluye 34 estudiantes de especilidades quirúrgicas quienes fueron evaluados mediante los OSATS para los módulos finales; modulo 1: disección vascular, M2: ejecución de sutura de puntos independientes, M3: ejercicios revascularización, al finalizar la materia los estudiantes evaluaron la implementación de esta metodología en evaluación por medio de una encuesta de satisfacción. Resultados: la evaluación del diseño de la guía práctica tipo OSATS por los estudiantes fue satisfactoria con 88,2 %,  al evaluar la actividad de disección y sutura en microcirugía la consideran de gran utilidad con 88,2 %, para los ejercicios de anastomosis y revascularización con una aprobación del 79.5 %, aplicabilidad práctica y calidad de los ejercicios mediante guías de evaluación OSATS 94.2 % de aceptación. Discusión: las escalas OSATS son un intrumento útil, valido y objetivo para el modelo de evaluación por competencias en habilidades técnicas, implementable en los modelos de enseñanza en simulación, contando con una confiabilidad, validez de contenido y de constructo, aplicable en los programas de microcirugía con una percepción positiva por los estudiantes a lo largo de su implementación.

Contemporary Issues in the Acquisition of Orthopaedic Surgical Skills During Residency: Competency-Based Medical Education and Simulation

➤

Orthopaedic education should produce surgeons who are competent to function independently and can obtain and maintain board certification.

➤

Contemporary orthopaedic training programs exist within a fixed 5-year time frame, which may not be a perfect match for each trainee.

➤

Most modern orthopaedic residencies have not yet fully adopted objective, proficiency-based, surgical skill training methods despite nearly 2 decades of evidence supporting the use of this methodology.

➤

Competency-based medical education backed by surgical simulation rooted in proficiency-based progression has the potential to address surgical skill acquisition challenges in orthopaedic surgery.

Resident Training and the Assessment of Orthopaedic Surgical Skills

Medical knowledge and technical skills are foundations of surgical competency. The American Board of Orthopaedic Surgery (ABOS) and the Resident Review Committee for Orthopaedic Surgery recently mandated simulation training to improve surgical skills, listing 17 surgical skills modules to improve residents' technical skills. However, there is no established tool to measure the effectiveness of these modules. The Global Index for Technical Skills (GRITS) tool has been previously validated for evaluating general surgery residents. The aim of this study was to determine whether the GRITS tool is valid, practical, and reliable in evaluating the skills of orthopaedic residents in a simulation setting, whether the outcomes correlate to performance in the operating room, and to what extent these simulation modules are valued by residents.

Learning a MIPO technique for distal radius fractures: Mentoring versus simple experience versus deliberate practice

BACKGROUND

Performance skills of a surgeon for a specific surgical technique range from 1 (novice) to 5 (expert). Mentoring can bring the surgeon to level 3. Simple experience rarely allows achievement of level 5, in contrast to deliberate practice, in which performance skills are improved by setting learning goals based on feedback about the trainee's previous performance.

HYPOTHESIS

When learning a technique for the internal fixation of distal radius fractures, the level of performance skills achieved is higher with deliberate practice than with mentoring or simple experience.

MATERIAL AND METHODS

Four surgeons each performed minimally invasive plate osteosynthesis (MIPO) of 15 distal radius fractures. The procedures were recorded by an HD camera. The first five fractures (step 1) were treated after mentoring, the next 5 fractures after reading an article and viewing a reference video (step 2), and the last 5 fractures after viewing and commenting 20 short videos highlighting possible errors (step 3). Each recording of the procedures performed by the surgeons was scored using the Objective Structured Assessment of Technical Skills (OSATS, 10 items on basic skills [B] and 10 on MIPO-specific skills [S]).

RESULTS

For the basic skills, the mean OSATS scores (on 50) were 31 for step 1, 31 for step 2, and 43 for step 3. For the specific skills, the mean OSATS scores (on 50) were 25 for step 1, 29 for step 2, and 48 for step 3. For overall skills (B+S), the mean OSATS score (on 100) was 56 for step 1, 60 for step 2, and 91 for step 3.

DISCUSSION

Our study confirms that the deliberate practice teaching method is more efficient in improving surgeon skills than simple experience. Deliberate practice consists of four essential steps: (i) setting a well-defined goal; (ii) being motivated to improve one's performance; (iii) receiving immediate feedback; and (iv) having multiple opportunities to repeat and gradually perfect one's performance. Our main hypothesis was verified, since the results of learning a MIPO technique, as assessed using the OSATS scale, were significantly better with a video-assisted deliberate practice technique than with mentoring or simple experience.

CONCLUSION

Video-assisted deliberate practice deserves to be widely used in order to optimise learning curves and to improve risk management in surgery.

LEVEL OF EVIDENCE

III.

Virtual and augmented reality for surgical training and simulation in knee arthroplasty

BACKGROUND

Immersive virtual reality (IVR), augmented reality and mixed reality form a spectrum of extended reality technology integration that has gained popularity in orthopaedics recently. This review article examines the role of extended reality technologies in knee arthroplasty.

METHODS

Existing literature on the applications of extended reality technologies in preoperative planning and intraoperative navigation were reviewed. A sample workflow of a novel IVR simulator for improving surgical training was also provided to demonstrate its utility in educating trainees on knee arthroplasty techniques.

RESULTS

Extended reality technologies enable the surgeon to visualise patient-specific anatomy in real-time, enhancing preoperative planning and providing intraoperative guidance. IVR technology has the potential to revolutionise modern surgical training and optimise surgical performance in a cost-efficient manner, with current evidence demonstrating favourable immediate skill acquisition and transfer.

CONCLUSIONS

Extended reality technologies have a myriad of potential applications in orthopaedic surgery. Further research is needed to evaluate the cost-effectiveness of its incorporation into training programmes.

Years of experience is more effective in defining experts in the gaze analysis of laparoscopic suturing task than task duration

In this study, the relationship between gaze patterns, task duration, and years of experience, which are commonly used to define and evaluate expert surgeons in laparoscopic surgery, was identified. Ten surgeons with 1-28 years of experience and six inexperienced students were included. Subjects used forceps to repeat the task of suturing a suture pad. Each subject wore an eye-marking recorder, and gaze points were recorded and analyzed. No significant relationship between task duration and gaze pattern was observed. However, there was a significant relationship between a surgeon's years of experience and the percentage of time spent gazing at the forceps. Subjects with more years of experience operated without looking at the forceps and fixed their gaze on the operational target. Therefore, when analyzing laparoscopic gazing patterns, it may be more appropriate to define an "expert" based on the years of experience rather than task duration.

Communication in Health Care: Impact of Language and Accent on Health Care Safety, Quality, and Patient Experience

Permanent or temporary migration results in communication issues related to language barriers. The migrant's mother tongue is often different from that of the host country. Even when the same language is spoken, communication barriers arise because of differences in accent. These communication barriers have a significant negative impact on migrants accessing health care and their ability to understand instructions and seek follow-up care. A multidisciplinary team often has professionals from various countries. These migrant health care professionals find it difficult to communicate with patients of the host country and with their colleagues. Communication barriers, therefore, result in miscommunication or no communication between health care professionals and between health care professionals and patients. This increases the risk of medical errors and impacts quality of care and patient safety. This review looks at the impact of communication barriers in health care and endeavors to find effective solutions.

Analysis of Tools Used in Assessing Technical Skills and Operative Competence in Trauma and Orthopaedic Surgical Training: A Systematic Review

BACKGROUND

Robust assessment of skills acquisition and surgical performance during training is vital to ensuring operative competence among orthopaedic surgeons. A move to competency-based surgical training requires the use of tools that can assess surgical skills objectively and systematically. The aim of this systematic review was to describe the evidence for the utility of assessment tools used in evaluating operative performance in trauma and orthopaedic surgical training.

METHODS

We performed a comprehensive literature search of MEDLINE, Embase, and Google Scholar databases to June 2019. From eligible studies we abstracted data on study aim, assessment format (live theater or simulated setting), skills assessed, and tools or metrics used to assess surgical performance. The strengths, limitations, and psychometric properties of the assessment tools are reported on the basis of previously defined utility criteria.

RESULTS

One hundred and five studies published between 1990 and 2019 were included. Forty-two studies involved open orthopaedic surgical procedures, and 63 involved arthroscopy. The majority (85%) were used in the simulated environment. There was wide variation in the type of assessment tools in used, the strengths and weaknesses of which are assessor and setting-dependent.

CONCLUSIONS

Current technical skills-assessment tools in trauma and orthopaedic surgery are largely procedure-specific and limited to research use in the simulated environment. An objective technical skills-assessment tool that is suitable for use in the live operative theater requires development and validation, to ensure proper competency-based assessment of surgical performance and readiness for unsupervised clinical practice.

CLINICAL RELEVANCE

Trainers and trainees can gain further insight into the technical skills assessment tools that they use in practice through the utility evidence provided.

Surgical Skill Can be Objectively Measured From Fluoroscopic Images Using a Novel Image-based Decision Error Analysis (IDEA) Score

BACKGROUND

To advance orthopaedic surgical skills training and assessment, more rigorous and objective performance measures are needed. In hip fracture repair, the tip-apex distance is a commonly used summative performance metric with clear clinical relevance, but it does not capture the skill exercised during the process of achieving the final implant position. This study introduces and evaluates a novel Image-based Decision Error Analysis (IDEA) score that better captures performance during fluoroscopically-assisted wire navigation.

QUESTIONS/PURPOSES

(1) Can wire navigation skill be objectively measured from a sequence of fluoroscopic images? (2) Are skill behaviors observed in a simulated environment also exhibited in the operating room? Additionally, we sought to define an objective skill metric that demonstrates improvement associated with accumulated surgical experience.

METHODS

Performance was evaluated both on a hip fracture wire navigation simulator and in the operating room during actual fracture surgery. After examining fluoroscopic image sequences from 176 consecutive simulator trials (performed by 58 first-year orthopaedic residents) and 21 consecutive surgical procedures (performed by 19 different orthopaedic residents and one attending orthopaedic surgeon), three main categories of erroneous skill behavior were identified: off-target wire adjustments, out-of-plane wire adjustments, and off-target drilling. Skill behaviors were measured by comparing wire adjustments made between consecutive images against the goal of targeting the apex of the femoral head as part of our new IDEA scoring methodology. Decision error metrics (frequency, magnitude) were correlated with other measures (image count and tip-apex distance) to characterize factors related to surgical performance on both the simulator and in the operating room. An IDEA composite score integrating decision errors (off-target wire adjustments, out-of-plane wire adjustments, and off-target drilling) and the final tip-apex distance to produce a single metric of overall performance was created and compared with the number of hip wire navigation cases previously completed (such as surgeon experience levels).

RESULTS

The IDEA methodology objectively analyzed 37,000 images from the simulator and 688 images from the operating room. The number of decision errors (7 ± 5 in the operating room and 4 ± 3 on the simulator) correlated with fluoroscopic image count (33 ± 14 in the operating room and 20 ± 11 on the simulator) in both the simulator and operating room environments (R2 = 0.76; p < 0.001 and R2 = 0.71; p < 0.001, respectively). Decision error counts did not correlate with the tip-apex distance (16 ± 4 mm in the operating room and 12 ± 5 mm on the simulator) for either the simulator or the operating room (R2 = 0.08; p = 0.15 and R2 = 0.03; p = 0.47, respectively), indicating that the tip-apex distance is independent of decision errors. The IDEA composite score correlated with surgical experience (R2 = 0.66; p < 0.001).

CONCLUSION

The fluoroscopic images obtained in the course of placing a guide wire contain a rich amount of information related to surgical skill. This points the way to an objective measure of skill that also has potential as an educational tool for residents. Future studies should expand this analysis to the wide variety of procedures that rely on fluoroscopic images.

CLINICAL RELEVANCE

This study has shown how resident skill development can be objectively assessed from fluoroscopic image sequences. The IDEA scoring provides a basis for evaluating the competence of a resident. The score can be used to assess skill at key timepoints throughout residency, such as when rotating onto/off of a new surgical service and before performing certain procedures in the operating room, or as a tool for debriefing/providing feedback after a procedure is completed.

Hanging up the surgical cap: Assessing the competence of aging surgeons

BACKGROUND

As the average age of surgeons continues to rise, determining when a surgeon should retire is an important public safety concern.

AIM

To investigate strategies used to determine competency in the industrial workplace that could be transferrable in the assessment of aging surgeons and to identify existing competency assessments of practicing surgeons.

METHODS

We searched websites describing non-medical professions within the United States where cognitive and physical competency are necessary for public safety. The mandatory age and certification process, including cognitive and physical requirements, were reported for each profession. Methods for determining surgical competency currently in use, and those existing in the literature, were also identified.

RESULTS

Four non-medical professions requiring mental and physical aptitude that involve public safety and have mandatory testing and/or retirement were identified: Airline pilots, air traffic controllers, firefighters, and United States State Judges. Nine late career practitioner policies designed to evaluate the ageing physician, including surgeons, were described. Six of these policies included subjective performance testing, 4 using peer assessment and 2 using dexterity testing. Six objective testing methods for evaluation of surgeon technical skill were identified in the literature. All were validated for surgical trainees. Only Objective Structured Assessment of Technical Skills (OSATS) was capable of distinguishing between surgeons of different skill level and showing a relationship between skill level and post-operative outcomes.

CONCLUSION

A surgeon should not be forced to hang up his/her surgical cap at a predetermined age, but should be able to practice for as long as his/her surgical skills are objectively maintained at the appropriate level of competency. The strategy of using skill-based simulations in evaluating non-medical professionals can be similarly used as part of the assessment of the ageing surgeons' surgical competency, showing who may require remediation or retirement.

Do Resident Surgical Volumes and Level of Training Correlate with Improved Performance on Psychomotor Skills Tasks: Construct Validity Testing of an ASSH Training Platform (STEP)?

The Surgical Training and Educational Platform (STEP) was developed by the American Society for Surgery of the Hand (ASSH) as a cost-effective set of surgical simulation modules designed to assess critical skills in hand surgery. Previous study demonstrated that STEP can differentiate between novice trainees and board-certified, certificate of added qualification hand surgeons. The purpose of this study was to assess construct validity of STEP by testing its ability to differentiate psychomotor skill level among intermediate trainees.

A live porcine model for surgical training in tracheostomy, neck dissection, and total laryngectomy

OBJECTIVES

Head and neck surgery is associated with a steep learning curve, creating a need for effective training methods. Live animal models can be useful in acquiring the required skills and techniques to perform tracheostomy, neck dissection, and laryngectomy. The aim of this study is to assess the feasibility and usefulness of a porcine model to teach tracheostomy, neck dissection, and laryngectomy, describing the similarities between a swine head and neck surgical model and the human cervical area.

METHODS

Twelve surgical training sessions were held with the use of a live porcine model maintained under general anaesthesia; a total of 39 procedures were performed. In each session, one otolaryngology resident per year performed a procedure to acquire the targeted competencies for their respective year of training. Each procedure was time recorded and evaluated based on a modified Objective Structured Assessment of Technical Skills. Linear regression was used to analyse the data and express the results as percentages.

RESULTS

Training exercises were performed in shorter times over the study period. A 9% reduction in the time spent was observed. There was a global improvement in the evaluation made by OSATS score of 6% in the first period (2018-2019) and 58% in the second period (2019-2020).

CONCLUSION

The porcine model was both manageable and helpful in providing training despite slight anatomical differences between human beings and pigs. However, more studies are necessary to guarantee that the competencies acquired with this type of training are transferable to real-life interventions.

LEVEL OF EVIDENCE

Level 5.

Establishing Validity of a Comprehensive Hand Surgical Training and Educational Platform (STEP)

PURPOSE

The purpose of this study was to develop and perform the initial validation for the Surgical Training and Educational Platform (STEP), a cost-effective psychomotor training and assessment instrument designed to teach and evaluate fundamental skills considered critical to competency in hand surgery.

METHODS

An American Society for Surgery of the Hand (ASSH) taskforce of 13 board-certified hand surgeons developed 8 skills considered fundamental to competency in hand surgery including: (1) lag screw fixation of an oblique fracture, (2) depth of plunge during bicortical drilling, (3) central axis scaphoid fixation, (4) phalangeal fracture pinning, (5) flexor tendon repair, (6) microsurgical suturing, (7) full-thickness skin graft harvest, and (8) wrist arthroscopy. The tasks were developed from commercially available, nonclinical supplies at low cost. The startup cost for the entire system was less than $600 USD, with a cost-per-trainee-assessment of approximately $25 USD. After the tasks were finalized, 2 examiners traveled to 8 sites around the country to evaluate 93 surgeons, including 57 board-certified hand surgeons and 36 first-year orthopedic and plastic surgical residents. Scoring criteria for each task were based on a combination of time and clinically relevant quality metrics.

RESULTS

Board-certified hand surgeons significantly outperformed surgical trainees on all 8 tasks, demonstrating that the STEP appropriately differentiates the skill level of expert and novice hand surgeons.

CONCLUSIONS

The STEP is an effective simulator that encompasses a wide range of fundamental psychomotor skills considered critical to competency in hand surgery. Although the STEP tasks are a valuable surgical training tool, further work is required to evaluate and refine the scoring system prior to using it as a critical evaluation of performance.

CLINICAL RELEVANCE

The STEP simulator is a valuable, validated instrument for resident and fellow education and evaluation in hand surgery outside of the operating room.

Innovation in Orthopaedic Surgery Education: Novel Tools for Modern Times

Orthopaedic residency training has and will continue to evolve with a wide variety of changes. Hands-on surgical simulation "boot camps," computerized simulation of surgical process, and even virtual reality simulators, all can help trainees acquire surgical experience without compromising patient care. Low-cost training modules help remedy the rising costs associated with teaching complex orthopaedic surgery skills. Motion tracking and checklists help refine standardization of assessment. As technology and healthcare systems continue to grow, we encourage training programs to keep pace by considering engagement of these tools.

Scoping Evidence Review on Training and Skills Assessment for Open Emergency Surgery

OBJECTIVE

Scope evidence on technical performance metrics for open emergency surgery. Identify surgical performance metrics and procedures used in trauma training courses.

DESIGN

Structured literature searches of electronic databases were conducted from January 2010 to December 2019 to identify systematic reviews of tools to measure surgical skills employed in vascular or trauma surgery evaluation and training.

SETTING AND PARTICIPANTS

Faculty of Shock Trauma Anesthesiology Research Center, University of Maryland School of Medicine, Uniformed Services University of Health Sciences, Bethesda, Maryland and Implementation Science, King's College, London.

RESULTS

The evidence from 21 systematic reviews including over 54,000 subjects enrolled into over 840 eligible studies, identified that the Objective Structured Assessment of Technical Skill was used for elective surgery not for emergency trauma and vascular control surgery procedures. The Individual Procedure Score (IPS), used to evaluate emergency trauma procedures performed before and after training, distinguished performance of residents from experts and practicing surgeons. IPS predicted surgeons who make critical errors and need remediation interventions. No metrics showed Kirkpatrick's Level 4 evidence of technical skills training benefit to emergency surgery outcomes.

CONCLUSIONS

Expert benchmarks, errors, complication rates, task completion time, task-specific checklists, global rating scales, Objective Structured Assessment of Technical Skills, and IPS were found to identify surgeons, at all levels of seniority, who are in need of remediation of technical skills for open surgical hemorrhage control. Large-scale, multicenter studies are needed to evaluate any benefit of trauma technical skills training on patient outcomes.

Workplace based assessment: A review of available tools and their relevance

Workplace-based assessment (WPBA) appears to be a promising tool for more comprehensive assessment of the learners. Relevant literature in this direction was collated and analyzed for its relevance, salience, and merit. Many WPBA tools are being used by various educational institutions which spans across multiple domains and over the entire duration of the workplace learning. It caters to holistic assessment with structured measures, real-time feedback, and continuous professional development. It is being used more for formative assessment and has limited utility in summative assessment as on date. WPBA tools have promising prospect in bringing novelty, objectivity, and holistic approach in assessment.

Improved Complex Skill Acquisition by Immersive Virtual Reality Training: A Randomized Controlled Trial

BACKGROUND

There has been limited literature on immersive virtual reality (VR) simulation in orthopaedic education. The purpose of this multicenter, blinded, randomized controlled trial was to determine the validity and efficacy of immersive VR training in orthopaedic resident education.

METHODS

Nineteen senior orthopaedic residents (resident group) and 7 consultant shoulder arthroplasty surgeons (expert group) participated in the trial comparing immersive VR with traditional learning using a technical journal article as a control. The examined task focused on achieving optimal glenoid exposure. Participants completed demographic questionnaires, knowledge tests, and a glenoid exposure on fresh-frozen cadavers while being examined by blinded shoulder arthroplasty surgeons. Training superiority was determined by the outcome measures of the Objective Structured Assessment of Technical Skills (OSATS) score, a developed laboratory metric, verbal answers, and time to task completion.

RESULTS

Immersive VR had greater realism and was superior in teaching glenoid exposure than the control (p = 0.01). The expert group outperformed the resident group on knowledge testing (p = 0.04). The immersive VR group completed the learning activity and knowledge tests significantly faster (p < 0.001) at a mean time (and standard deviation) of 11 ± 3 minutes than the control group at 20 ± 4 minutes, performing 3 to 5 VR repeats for a reduction in learning time of 570%. The immersive VR group completed the glenoid exposure significantly faster (p = 0.04) at a mean time of 14 ± 7 minutes than the control group at 21 ± 6 minutes, with superior OSATS instrument handling scores (p = 0.03). The immersive VR group scored equivalently in surprise verbal scores (p = 0.85) and written knowledge scores (p = 1.0).

CONCLUSIONS

Immersive VR demonstrated substantially improved translational technical and nontechnical skills acquisition over traditional learning in senior orthopaedic residents. Additionally, the results demonstrate the face, content, construct, and transfer validity for immersive VR.

CLINICAL RELEVANCE

This adequately powered, randomized controlled trial demonstrated how an immersive VR system can efficiently (570%) teach a complex surgical procedure and also demonstrate improved translational skill and knowledge acquisition when compared with a traditional learning method.

Development of a certification examination for orthopedic sports medicine fellows

Background

The purpose of this study was to develop a multifaceted examination to assess the competence of fellows following completion of a sports medicine fellowship.

Methods

Orthopedic sports medicine fellows over 2 academic years were invited to participate in the study. Clinical skills were evaluated with objective structured clinical examinations, multiple-choice question examinations, an in-training evaluation report and a surgical logbook. Fellows’ performance of 3 technical procedures was assessed both intraoperatively and on cadavers: anterior cruciate ligament reconstruction (ACLR), arthroscopic rotator cuff repair (RCR) and arthroscopic shoulder Bankart repair. Technical procedural skills were assessed using previously validated task-specific checklists and the Arthroscopic Surgical Skill Evaluation Tool (ASSET) global rating scale.

Results

Over 2 years, 12 fellows were assessed. The Cronbach α for the technical assessments was greater than 0.8, and the interrater reliability for the cadaveric assessments was greater than 0.78, indicating satisfactory reliability. When assessed in the operating room, all fellows were determined to have achieved a minimal level of competence in the 3 surgical procedures, with the exception of 1 fellow who was not able achieve competence in ACLR. When their performance on cadaveric specimens was assessed, 2 of 12 (17%) fellows were not able to demonstrate a minimal level of competence in ACLR, 2 of 10 (20%) were not able to demonstrate a minimal level of competence for RCR and 3 of 10 (30%) were not able to demonstrate a minimal level of competence for Bankart repair.

Conclusion

There was a disparity between fellows’ performance in the operating room and their performance in the high-fidelity cadaveric setting, suggesting that technical performance in the operating room may not be the most appropriate measure for assessment of fellows’ competence.

A Vision for Using Simulation & Virtual Coaching to Improve the Community Practice of Orthopedic Trauma Surgery

BACKGROUND

Many orthopedic surgeries involve the challenging integration of fluoroscopic image interpretation with skillful tool manipulation to enable procedures to be performed through less invasive approaches. Simulation has proved beneficial for teaching and improving these skills for residents, but similar benefits have not yet been realized for practicing orthopedic surgeons. A vision is presented to elevate community orthopedic practice and improve patient safety by advancing the use of simulators for training and assessing surgical skills.

METHODS

Key elements of this vision that are established include 1) methods for the objective and rigorous assessment of the performance of practicing surgeons now exist, 2) simulators are sufficiently mature and sophisticated that practicing surgeons will use them, and 3) practicing surgeons can improve their performance with appropriate feedback and coaching.

RESULTS

Data presented indicate that surgical performance can be adequately and comparably measured using structured observations made by experts or non-expert crowds, with the crowdsourcing approach being more expedient and less expensive. Rigorous measures of the surgical result and intermediate objectives obtained semi-automatically from intra-operative fluoroscopic image sequences can distinguish performances of experts from novices. Experience suggests that practicing orthopedic surgeons are open to and can be constructively engaged by a family of mature simulators as a means to evaluate and improve their surgical skills.

CONCLUSIONS

The results presented support our contention that new objective assessment measures are sufficient for evaluating the performance of working surgeons. The novel class of orthopedic surgical simulators available were tested and approved by practicing physicians. There exists a clear opportunity to combine purpose-designed simulator exercises with virtual coaching to help practicing physicians retain, retrain, and improve their technical skills. This will ultimately reduce cost, increase the quality of care, and decrease complication rates.

CLINICAL RELEVANCE

This vision articulates a means to boost the confidence of practitioners and ease their anxiety so that they perform impactful procedures more often in community hospitals, which promises to improve treatment and reduce the cost of care while keeping patients closer to their homes and families.

Technical and Nontechnical Skills in Surgery: A Simulated Operating Room Environment Study

STUDY DESIGN

Observational simulation study.

OBJECTIVE

The goal of this study was to investigate the relationship between technical and nontechnical skills (NTS) in a simulated surgical procedure.

SUMMARY OF BACKGROUND DATA

Although surgeons' technical and NTS during surgery are crucial determinants for clinical outcomes, little literature is available in spine surgery. Moreover, evidence regarding how surgeons' technical and NTS are related is limited.

METHODS

A mixed-reality and full-scale simulated operating room environment was employed for the surgical team. Eleven surgeons performed the vertebroplasty procedure (VP). Technical skills (TS) were assessed using Objective Structured Assessment of Technical Skill scores and senior expert-evaluated VP outcome assessment. NTS were assessed with the Observational Teamwork Assessment for Surgery. Kendall-Tau-b tests were performed for correlations. We further controlled the influence of surgeons' experience (based on professional tenure and number of previous VPs performed).

RESULT

Surgeons' NTS correlated significantly with their technical performance (τ = 0.63; P = 0.006) and surgical outcome scores (τ = 0.60; P = 0.007). This association was attenuated when controlling for surgeons' experience.

CONCLUSION

Our results suggest that spine surgeons with higher levels of TS also apply better communication, leadership, and coordination behaviors during the procedure. Yet, the role of surgeons' experience needs further investigation for improving surgeons' intraoperative performance during spine surgery.

LEVEL OF EVIDENCE

3.

Intraoperative Point of View Video Capture and Surgical Segmentation in Carpal Tunnel Release: A Feasibility Analysis

OBJECTIVE

The purpose of this study was to (1) examine the feasibility of intraoperative point of view video while performing open and endoscopic carpal tunnel release (CTR), (2) define surgical segments of CTR, and (3) describe the duration of various surgical steps of open versus endoscopic CTR in a teaching setting.

DESIGN

Fellowship trained hand surgeons reached consensus on surgical segments for CTR. Adult patients 18 and older previously indicated for CTR in clinic were eligible. Head-mounted point-of-view cameras were worn during endoscopic and open CTR by resident surgeons. Video was reviewed to determine segment duration. Independent sample t tests were used for comparison of duration by technique with statistical significance set as p < 0.05.

SETTING

University of Iowa Hospitals and Clinics; 200 Hawkins Dr, Iowa City, IA 52242; Tertiary Academic Medical Center.

PARTICIPANTS

Orthopedic Surgery Residents and Orthopedic Surgery Faculty.

RESULTS

Surgical segments were defined as incision, dissection of superficial soft tissue structures, transection of the carpal ligament, and surgical incision closure. Twelve of 14 video capture events yielded data. In the teaching setting, the average duration of endoscopic CTR was 609.5 seconds (±111.07) versus 547.75 seconds (±82.06) for open with p value = 0.406. No surgical segments were significantly different. Transition time from dissection to ligament transection differed significantly (p = 0.004) between endoscopic (46.88 seconds ± 19.19) and open (9.0 seconds ± 7.90) CTR. Transition time between ligament transection and closure was significantly different (p = 0.029) among endoscopic (50.5 seconds ± 15.0) and open (26.25 seconds ± 2.99) CTR.

CONCLUSIONS

Point-of-view video capture is feasible for the capture of video during a common hand surgery procedure. A method for managing device battery power is necessary for future applications. CTR can be defined as, and described in, individual procedure segments potentially useful for surgical education as well as efficiency improvements. Identification of surgical segments may aid the development of better objective tools for the assessment of surgeon skill and competency for common orthopedic procedures.

Advancing Simulation-Based Orthopaedic Surgical Skills Training: An Analysis of the Challenges to Implementation

Simulation-based surgical skills training is recognized as a valuable method to improve trainees' performance and broadly perceived as essential for the establishment of a comprehensive curriculum in surgical education. However, there needs to be improvement in several areas for meaningful integration of simulation into surgical education. The purpose of this focused review is to summarize the obstacles to a comprehensive integration of simulation-based surgical skills training into surgical education and board certification and suggest potential solutions for those obstacles. First and foremost, validated simulators need to be rigorously assessed to ensure their feasibility and cost-effectiveness. All simulation-based courses should include clear objectives and outcome measures (with metrics) for the skills to be practiced by trainees. Furthermore, these courses should address a wide range of issues, including assessment of trainees' problem-solving and decision-making abilities and remediation of poor performance. Finally, which simulation-based surgical skills courses will become a standard part of the curriculum across training programs and which will be of value in board certification should be precisely defined. Sufficient progress in these areas will prevent excessive development of training and assessment tools with duplicative effort and large variability in quality.

Competency assessment tool for laparoscopic suturing: development and reliability evaluation

BACKGROUND

Laparoscopic suturing can be technically challenging and requires extensive training to achieve competency. To date no specific and objective assessment method for laparoscopic suturing and knot tying is available that can guide training and monitor performance in these complex surgical skills. In this study we aimed to develop a laparoscopic suturing competency assessment tool (LS-CAT) and assess its inter-observer reliability.

METHODS

We developed a bespoke CAT tool for laparoscopic suturing through a structured, mixed methodology approach, overseen by a steering committee with experience in developing surgical assessment tools. A wide Delphi consultation with over twelve experts in laparoscopic surgery guided the development stages of the tool. Following, subjects with different levels of laparoscopic expertise were included to evaluate this tool, using a simulated laparoscopic suturing task which involved placing of two surgical knots. A research assistant video recorded and anonymised each performance. Two blinded expert surgeons assessed the anonymised videos using the developed LS-CAT. The LS-CAT scores of the two experts were compared to assess the inter-observer reliability. Lastly, we compared the subjects' LS-CAT performance scores at the beginning and end of their learning curve.

RESULTS

This study evaluated a novel LS-CAT performance tool, comprising of four tasks. Thirty-six complete videos were analysed and evaluated with the LS-CAT, of which the scores demonstrated excellent inter-observer reliability. Cohen's Kappa analysis revealed good to excellent levels of agreement for almost all tasks of both instrument handling and tissue handling (0.87; 0.77; 0.75; 0.86; 0.85, all with p < 0.001). Subjects performed significantly better at the end of their learning curve compared to their first attempt for all LS-CAT items (all with p < 0.001).

CONCLUSIONS

We developed the LS-CAT, which is a laparoscopic suturing grading matrix, with excellent inter-rater reliability and to discriminate between experience levels. This LS-CAT has a potential for wider use to objectively assess laparoscopic suturing skills.

Original Homebuilt Off-the-Job Training System for Vascular Surgeons: System Analysis and Assessment

Objective: We assessed the effectiveness and appropriateness of our original off-the-job training (Off JT) system using data acquired from recruited medical students and doctors.

Materials and Methods: We presented our original homebuilt Off JT system, which is simple and inexpensive. In our unique system, we performed anastomosis at the bottom of a plastic pot, which mimics the actual open surgical procedure at a deep site. There were four evaluation points: (A) operating time, (B) performance of anastomosis by semi-automatically analyzing the image with the coefficient of variation (standard deviation/length) of the “bite” and the “pitch,” (C) scoring of the total surgical skill evaluated by the trainers according to the Operative Performance Rating System (OPRS), and (D) the relationship of these three factors (A, B, and C).

Results: The procedural time and coefficient of variation of the bite and pitch decreased and the OPRS score increased after training. There was a strong correlation between procedural time, anastomotic performance, and OPRS score.

Conclusion: The effectiveness of our original homebuilt system was shown by reduced procedural time, improved anastomotic quality, and increased OPRS score.

Validation of the Omni: A Novel, Multimodality, and Longitudinal Surgical Skills Assessment

OBJECTIVE

The breadth of technical skills included in general surgery training continues to expand. The current competency-based training model requires assessment tools to measure acquisition, learning, and mastery of technical skill longitudinally in a reliable and valid manner. This study describes a novel skills assessment tool, the Omni, which evaluates performance in a broad range of skills over time.

DESIGN

The 5 Omni tasks, consisting of open bowel anastomosis, knot tying, laparoscopic clover pattern cut, robotic needle drive, and endoscopic bubble pop, were developed by general surgery faculty. Component performance metrics assessed speed, accuracy, and quality, which were scaled into an overall score ranging from 0 to 10 for each task. For each task, ANOVAs with Scheffé's post hoc comparisons and Pearson's chi-squared tests compared performance between 6 resident cohorts (clinical years (CY1-5) and research fellows (RF)). Paired samples t-tests evaluated changes in performance across academic years. Cronbach's alpha coefficient determined the internal consistency of the Omni as an overall assessment.

SETTING

The Omni was developed by the Department of Surgery at Duke University. Annual assessment and this research study took place in the Surgical Education and Activities Lab.

PARTICIPANTS

All active general surgery residents in 2 consecutive academic years spanning 2015 to 2017.

RESULTS

A total of 62 general surgery residents completed the Omni and 39 (67.2%) of those residents completed the assessment in 2 consecutive years. Based on data from all residents' first assessment, statistically significant differences (p < 0.05) were observed among CY cohorts for bowel anastomosis, robotic, and laparoscopic task metrics. By pair-wise comparisons, mean bowel anastomosis scores distinguished CY1 from CY3-5 and CY2 from CY5. Mean robotic scores distinguished CY1 from RF, and mean laparoscopic scores distinguished CY1 from RF, CY3, and CY5 in addition to CY2 from CY3. Mean scores in performance on the knot tying and endoscopic tasks were not significantly different. Statistically significant improvement in mean scores was observed for all tasks from year 1 to year 2 (all p < 0.02). The internal consistency analysis revealed an alpha coefficient of 0.656.

CONCLUSIONS

The Omni is a novel composite assessment tool for surgical technical skill that utilizes objective measures and scoring algorithms to evaluate performance. In this pilot study, 3 tasks demonstrated discriminative ability of performance by CY, and all 5 tasks demonstrated construct validity by showing longitudinal improvement in performance. Additionally, the Omni has adequate internal consistency for a formative assessment. These results suggest the Omni holds promise for the evaluation of resident technical skill and early identification of outliers requiring intervention.

The Effectiveness of Bioskills Training for Simulated Lumbar Pedicle Screw Placement

STUDY DESIGN

Prospective randomized study.

OBJECTIVES

To define the impact of an inexpensive, user-friendly, and reproducible lumbar pedicle screw instrumentation bioskills training module and evaluation protocol.

METHODS

Participants were randomized to control (n = 9) or intervention (n = 10) groups controlling for level of experience (medical students, junior resident, or senior resident). The intervention group underwent a 20-minute bioskills training module while the control group spent the same time with self-directed study. Pre- and posttest performance was self-reported (Physician Performance Diagnostic Inventory Scale [PPDIS]). Objective outcome scores were obtained from a blinded fellowship-trained attending orthopedic spine surgeon using Objective Structured Assessment of Technical Skills (OSATS) and Objective Pedicle Instrumentation Score metrics. In addition, identification of pedicle breach and breach anatomic location was measured pre- and posttest in lumbar spine models.

RESULTS

The intervention group showed a 30.8% improvement in PPDIS scores, compared with 13.4% for the control group (P = .01). The intervention group demonstrated statistically significant 66% decrease in breaches (P = .001) compared with 28% decrease in the control group (P = .06). Breach identification demonstrated no change in accuracy of the control group (incorrect identification from 32.2% pre- to posttest 35%; P = .71), whereas the intervention group's improvement was statistically significant (42% pre- to posttest 36.5%; P = .0047).

CONCLUSIONS

We conclude that a concise lumbar pedicle screw instrumentation bioskills training session can be a useful educational tool to augment clinical education.

Examination of Skill Acquisition and Grader Bias in a Distal Radius Fracture Fixation Model

OBJECTIVES

Primary: Assess the ability of faculty graders to predict the objectively measured strength of distal radius fracture fixation. Secondary: Compare resident skill variation and retention related to other knowable training data.

DESIGN

Residents were allowed 60 minutes to stabilize a standardized distal radius fracture using an assigned fixed-angle volar plate. Faculty observed and subjectively graded the residents without providing real-time feedback. Objective biomechanical evaluation (construct strength and stiffness) was compared to subjective grades. Resident-specific characteristics (sex, PGY, and ACGME case log) were also used to compare the objective data.

SETTING

A simulated operating room in our laboratory.

PARTICIPANTS

Post-graduate year 2, 3, 4, and 5 orthopedic residents.

RESULTS

Primary: Faculty were not successful at predicting objectively measured fixation, and their subjective scoring suggests confirmation bias as PGY increased. Secondary: Resident year-in-training alone did not predict objective measures (p = 0.53), but was predictive of subjective scores (p < 0.001). Skills learned were not always retained, as 29% of residents objectively failed subsequent to passing. Notably, resident-reported case-specific experience alone was inversely correlated with objective fixation strength.

CONCLUSIONS

This testing model enabled the collection of objective and subjective resident skill scores. Faculty graders did not routinely predict objective measures, and their subjective assessment appears biased related to PGY. Also, in vivo case volume alone does not predict objective results. Familiar faculty teaching consistency, and resident grading by external faculty unfamiliar with tested residents, might alter these results.

Developing an objective assessment of surgical performance from operating room video and surgical imagery

An unbiased, repeatable process for assessing operating room performance is an important step toward quantifying the relationship between surgical training and performance. Hip fracture surgeries offer a promising first target in orthopedic trauma because they are common and they offer quantitative performance metrics that can be assessed from video recordings and intraoperative fluoroscopic images. Hip fracture repair surgeries were recorded using a head-mounted point-of-view camera. Intraoperative fluoroscopic images were also saved. The following performance metrics were analyzed: duration of wire navigation, number of fluoroscopic images collected, degree of intervention by the surgeon's supervisor, and the tip-apex distance (TAD). Two orthopedic traumatologists graded surgical performance in each video independently using an Objective Structured Assessment of Technical Skill (OSATS). Wire navigation duration correlated with weeks into residency and prior cases logged. TAD correlated with cases logged. There was no significant correlation between the OSATS total score and experience metrics. Total OSATS score correlated with duration and number of fluoroscopic images. Our results indicate that two metrics of hip fracture wire navigation performance, duration and TAD, significantly differentiate surgical experience. The methods presented have the potential to provide truly objective assessment of resident technical performance in the OR.

The Effectiveness of Bioskills Training for Simulated Open Lumbar Laminectomy

STUDY DESIGN

Randomized, prospective study within an orthopedic surgery resident program at a large urban academic medical center.

OBJECTIVES

To develop an inexpensive, user-friendly, and reproducible lumbar laminectomy bioskills training module and evaluation protocol that can be readily implemented into residency training programs to augment the clinical education of orthopedic and neurosurgical physicians-in-training.

METHODS

Twenty participants comprising senior medical students and orthopedic surgical residents. Participants were randomized to control (n = 9) or intervention (n = 11) groups controlling for level of experience (medical students, junior resident, or senior resident). The intervention group underwent a 40-minute bioskills training module, while the control group spent the same time with self-directed study. Pre- and posttest performance was self-reported by each participant (Physician Performance Diagnostic Inventory Scale [PPDIS]). Objective outcome scores were obtained from a blinded fellowship-trained attending orthopedic spine surgeon using Objective Structured Assessment of Technical Skills (OSATS) and Objective Decompression Score metrics.

RESULTS

When compared with the control group, the intervention group yielded a significant mean improvement in OSATS (P = .022) and PPDIS (P = .0001) scores. The Objective Decompression Scores improved in the intervention group with a trend toward significance (P = .058).

CONCLUSIONS

We conclude that a concise lumbar laminectomy bioskills training session can be a useful educational tool for to augment clinical education. Although no direct clinical correlation can be concluded from this study, the improvement in trainee's technical and procedural skills suggests that Sawbones training modules can be an efficient and effective tool for teaching fundamental spine surgical skills outside of the operating room.

Measuring Surgical Skills in Simulation-based Training

Simulation-based surgical skills training addresses several concerns associated with the traditional apprenticeship model, including patient safety, efficient acquisition of complex skills, and cost. The surgical specialties already recognize the advantages of surgical training using simulation, and simulation-based methods are appearing in surgical education and assessment for board certification. The necessity of simulation-based methods in surgical education along with valid, objective, standardized techniques for measuring learned skills using simulators has become apparent. The most commonly used surgical skill measurement techniques in simulation-based training include questionnaires and post-training surveys, objective structured assessment of technical skills and global rating scale of performance scoring systems, structured assessments using video recording, and motion tracking software. The literature shows that the application of many of these techniques varies based on investigator preference and the convenience of the technique. As simulators become more accepted as a teaching tool, techniques to measure skill proficiencies will need to be standardized nationally and internationally.

Objective Assessment of Surgical Technical Skill and Competency in the Operating Room

Training skillful and competent surgeons is critical to ensure high quality of care and to minimize disparities in access to effective care. Traditional models to train surgeons are being challenged by rapid advances in technology, an intensified patient-safety culture, and a need for value-driven health systems. Simultaneously, technological developments are enabling capture and analysis of large amounts of complex surgical data. These developments are motivating a "surgical data science" approach to objective computer-aided technical skill evaluation (OCASE-T) for scalable, accurate assessment; individualized feedback; and automated coaching. We define the problem space for OCASE-T and summarize 45 publications representing recent research in this domain. We find that most studies on OCASE-T are simulation based; very few are in the operating room. The algorithms and validation methodologies used for OCASE-T are highly varied; there is no uniform consensus. Future research should emphasize competency assessment in the operating room, validation against patient outcomes, and effectiveness for surgical training.