Validation of orthopaedic bench models for trauma surgery

Mechanical Properties of Synthetic Bones Made by Synbone: A Review

Biomechanical engineers and physicists commonly employ biological bone for biomechanics studies, since they are good representations of living bone. Yet, there are challenges to using biological bone, such as cost, degradation, disease, ethics, shipping, sourcing, storage, variability, etc. Therefore, the Synbone® company has developed a series of synthetic bones that have been used by biomechanical investigators to offset some drawbacks of biological bone. There have been a number of published biomechanical reports using these bone surrogates for dental, injury, orthopedic, and other applications. But, there is no prior review paper that has summarized the mechanical properties of these synthetic bones in order to understand their general performance or how well they represent biological bone. Thus, the goal of this article was to survey the English-language literature on the mechanical properties of these synthetic bones. Studies were included if they quantitatively (a) characterized previously unknown values for synthetic bone, (b) validated synthetic versus biological bone, and/or (c) optimized synthetic bone performance by varying geometric or material parameters. This review of data, pros, cons, and future work will hopefully assist biomechanical engineers and physicists that use these synthetic bones as they develop experimental testing regimes and computational models.

Simulation Training in Spine Surgery

Simulated surgery is part of a growing paradigm shift in surgical education as a whole. Various modalities from cadaver models to virtual reality have been developed and studied within the context of surgical education. Simulation training in spine surgery has an immense potential to improve education and ultimately improve patient safety. This is due to the inherent risk of operating the spine and the technical difficulty of modern techniques. Common procedures in the modern orthopaedic armamentarium, such as pedicle screw placement, can be simulated, and proficiency is rapidly achieved before application in patients. Furthermore, complications such as dural tears can be simulated and effectively managed in a safe environment with simulation. New techniques with steeper learning curves, such as minimally invasive techniques, can now be safely simulated. Hence, augmenting surgical education through simulation has great potential to benefit trainees and practicing orthopaedic surgeons in modern spine surgery techniques. Additional work will aim to improve access to such technologies and integrate them into the current orthopaedic training curriculum.

Evaluating the effectiveness of the Imperial Femoral Intramedullary Nailing Cognitive Task Analysis (IFINCTA) tool in a real-time simulation setting (Distributed Interactive Simulation): a randomized controlled trial

INTRODUCTION

Working time regulations, senior led service delivery and increasing complexity of surgical technology has led to significant strains in surgical training. Additionally, the current COVID-19 pandemic has placed substantial limitations on surgical training worldwide. Contact free, remote, web-based, validated learning tools which are easily accessible and allows repeated, sustained practice are the need of the hour. Cognitive Task Analysis (CTA) have been used extensively to train pilots and military personnel and has shown excellent early results within orthopaedic training. We designed a femoral nailing CTA tool which showed objective benefits in the enhancement of cognitive knowledge in medical students. The aim of this study was to evaluate the effectiveness of this CTA tool to enhance practical skills in orthopaedic trainees in a real time interactive simulation setting (Distributed Interactive Simulation (DIS)).

METHODS

This was a double blinded, randomized controlled trial. 14 junior orthopaedic residents who met the inclusion criteria were recruited in the study. They were randomized into two equal groups. The intervention group were given the CTA learning tool, the control group were given a standard operative technique manual used for antegrade femoral intramedullary nailing. The participants were assessed on a high-fidelity phantom femur model with actual femoral nailing instruments in a simulation mobile operating theatre where the candidate had a simulation patient, an acting anesthetist and a scrub nurse (DIS). They were assessed using the modified Objective Structured Assessment of Technical Skills (OSATS) rating scale which has been validated for orthopaedic trauma.

RESULTS

The median OSATS score in the intervention group was 49 (±4.93, range 39-55) compared to 17 in the control group (±14.98, range 12-51). The median improvement was by 32 points (p = 0.02). The ICC between the two raters was 0.977.

CONCLUSIONS

This study has demonstrated objective benefits of a novel femoral nailing CTA tool in the enhancement of practical skills for junior trainees in the DIS setting. This adds to the growing evidence supporting the use of CTA in orthopaedic training. This tool can be accessed remotely, is contact free and allows repeated sustained practice which is key in simulation training.

Analysis of Tools Used in Assessing Technical Skills and Operative Competence in Trauma and Orthopaedic Surgical Training: A Systematic Review

BACKGROUND

Robust assessment of skills acquisition and surgical performance during training is vital to ensuring operative competence among orthopaedic surgeons. A move to competency-based surgical training requires the use of tools that can assess surgical skills objectively and systematically. The aim of this systematic review was to describe the evidence for the utility of assessment tools used in evaluating operative performance in trauma and orthopaedic surgical training.

METHODS

We performed a comprehensive literature search of MEDLINE, Embase, and Google Scholar databases to June 2019. From eligible studies we abstracted data on study aim, assessment format (live theater or simulated setting), skills assessed, and tools or metrics used to assess surgical performance. The strengths, limitations, and psychometric properties of the assessment tools are reported on the basis of previously defined utility criteria.

RESULTS

One hundred and five studies published between 1990 and 2019 were included. Forty-two studies involved open orthopaedic surgical procedures, and 63 involved arthroscopy. The majority (85%) were used in the simulated environment. There was wide variation in the type of assessment tools in used, the strengths and weaknesses of which are assessor and setting-dependent.

CONCLUSIONS

Current technical skills-assessment tools in trauma and orthopaedic surgery are largely procedure-specific and limited to research use in the simulated environment. An objective technical skills-assessment tool that is suitable for use in the live operative theater requires development and validation, to ensure proper competency-based assessment of surgical performance and readiness for unsupervised clinical practice.

CLINICAL RELEVANCE

Trainers and trainees can gain further insight into the technical skills assessment tools that they use in practice through the utility evidence provided.

Success of Surgical Simulation in Orthopedic Training and Applications in Spine Surgery

STUDY DESIGN

This was a narrative review.

OBJECTIVE

This study aimed to review the current literature on surgical simulation in orthopedics and its application to spine surgery.

SUMMARY OF BACKGROUND DATA

As orthopedic surgery increases in complexity, training becomes more relevant. There have been mandates in the United States for training orthopedic residents the fundamentals of surgical skills; however, few studies have examined the various training options available. Lack of funding, availability, and time are major constraints to surgical simulation options.

METHODS

A PubMed review of the current literature was performed on all relevant articles that examined orthopedic trainees using surgical simulation options. Studies were examined for their thoroughness and application of simulation options to orthopedic surgery.

RESULTS

Twenty-three studies have explored orthopedic surgical simulation in a setting that objectively assessed trainee performance, most in the field of trauma and arthroscopy. However, there was a lack of consistency in measurements made and skills tested by these simulators. There has only been one study exploring surgical simulation in spine surgery.

CONCLUSIONS

While there has been a growing number of surgical simulators to train orthopedic residents the fundamentals of surgical skills, most of these simulators are not feasible, reproducible, or available to the majority of training centers. Furthermore, the lack of consistency in the objective measurements of these studies makes interpretation of their results difficult. There is a need for more simulation in spine surgery, and future simulators and their respective studies should be reproducible, affordable, applicable to the surgical setting, and easily assembled by various programs across the world.

Virtual Reality and Augmented Reality-Translating Surgical Training into Surgical Technique

PURPOSE OF REVIEW

As immersive learning outside of the operating room is increasingly recognized as a valuable method of surgical training, virtual reality (VR) and augmented reality (AR) are increasingly utilized in orthopedic surgical training. This article reviews the evolving nature of these training tools and provides examples of their use and efficacy. The practical and ethical implications of incorporating this technology and its impact on both orthopedic surgeons and their patients are also discussed.

RECENT FINDINGS

Head-mounted displays (HMDs) represent a possible adjunct to surgical accuracy and education. While the hardware is advanced, there is still much work to be done in developing software that allows for seamless, reliable, useful integration into clinical practice and training. Surgical training is changing: AR and VR will become mainstays of future training efforts. More evidence is needed to determine which training technology translates to improved clinical performance. Volatility within the HMD industry will likely delay advances in surgical training.

"Virtual" Boot Camp: Orthopaedic Intern Education in the Time of COVID-19 and Beyond

The emergence of COVID-19 as a viral pandemic in early 2020 resulted in notable changes to the daily practice, workflow, and education of orthopaedic residencies internationally. In particular, social distancing, residency restructuring, and redeployment to other services has increased heterogeneity in schedules and made the in-person gathering of trainees for education increasingly challenging. These changes may last until 2024 based on some mathematical models, resulting in notable disruptions to orthopaedic education, especially for junior residents. Therefore, in this study, we describe how we converted our in-person PGY-1 skills course into a "virtual" boot camp based on validated training modules and existing American Board of Orthopaedic Surgeons guidelines. Lessons learned from the experience and potential areas for improvement in the use of newer technology to teach cognitive knowledge and skills modules are highlighted with the hope that this can be useful to other orthopaedic residency programs, during the pandemic and also beyond.

Current provision of simulation in the UK and Republic of Ireland trauma and orthopaedic specialist training: a national survey

Aims

The primary aim of the survey was to map the current provision of simulation training within UK and Republic of Ireland (RoI) trauma and orthopaedic (T&O) specialist training programmes to inform future design of a simulation based-curriculum. The secondary aims were to characterize; the types of simulation offered to trainees by stage of training, the sources of funding for simulation, the barriers to providing simulation in training, and to measure current research activity assessing the educational impact of simulation.

Methods

The development of the survey was a collaborative effort between the authors and the British Orthopaedic Association Simulation Group. The survey items were embedded in the Performance and Opportunity Dashboard, which annually audits quality in training across several domains on behalf of the Speciality Advisory Committee (SAC). The survey was sent via email to the 30 training programme directors in March 2019. Data were retrieved and analyzed at the Warwick Clinical Trials Unit, UK.

Results

Overall, 28 of 30 programme directors completed the survey (93%). 82% of programmes had access to high-fidelity simulation facilities such as cadaveric laboratories. More than half (54%) had access to a non-technical skills simulation training. Less than half (43%) received centralized funding for simulation, a third relied on local funding such as the departmental budget, and there was a heavy reliance on industry sponsorship to partly or wholly fund simulation training (64%). Provision was higher in the mid-stages (ST3-5) compared to late-stages (ST6-8) of training, and was formally timetabled in 68% of prostgrammes. There was no assessment of the impact of simulation training using objective behavioural measures or real-world clinical outcomes.

Conclusion

There is currently widespread, but variable, provision of simulation in T&O training in the UK and RoI, which is likely to expand further with the new curriculum. It is important that research activity into the impact of simulation training continues, to develop an evidence base to support investment in facilities and provision.

Nonbiological Microsurgery Simulators in Plastic Surgery Training: A Systematic Review

BACKGROUND

Simulation has gained notable recognition for its role as an effective training and assessment modality in the present era of competency-based medical education. Despite the well-documented efficacy of both live and cadaveric animal models, several ethical, financial, and accessibility issues persist with their use. Lower fidelity nonbiological simulators have gained recognition for their ability to circumvent these challenges. This systematic review reports on all prosthetic and virtual reality simulators in use for microsurgery training, with an emphasis on each model's complexity, characteristics, advantages, disadvantages, and validation measures taken.

METHODS

A systematic search was performed using the National Library of Medicine (PubMed), MEDLINE, and Embase databases. Search terms were those pertaining to prosthetic and virtual reality models with relevance to microsurgical training in plastic surgery. Three independent reviewers evaluated all articles retrieved based on strict inclusion and exclusion criteria.

RESULTS

Fifty-seven articles met the inclusion criteria for review, reporting on 20 basic prosthetic models, 20 intermediate models, 13 advanced models, and six virtual reality simulators.

CONCLUSIONS

A comprehensive summary has been compiled of all nonbiological simulators in use for microsurgery training in plastic surgery, demonstrating efficacy for the acquisition and retention of microsurgical skills. Metrics-based validation efforts, however, were often lacking in the literature. As plastic surgery programs continue to innovate, ensure accountability, and safely meet today's training standards, prosthetic simulators are set to play a larger role in the development of a standardized, ethical, accessible, and objectively measurable microsurgery training curriculum for the modern-day plastic and reconstructive surgery resident.

Artificial Intelligence Distinguishes Surgical Training Levels in a Virtual Reality Spinal Task

BACKGROUND

With the emergence of competency-based training, the current evaluation scheme of surgical skills is evolving to include newer methods of assessment and training. Artificial intelligence through machine learning algorithms can utilize extensive data sets to analyze operator performance. This study aimed to address 3 questions: (1) Can artificial intelligence uncover novel metrics of surgical performance? (2) Can support vector machine algorithms be trained to differentiate "senior" and "junior" participants who are executing a virtual reality hemilaminectomy? (3) Can other algorithms achieve a good classification performance?

METHODS

Participants from 4 Canadian universities were divided into 2 groups according to their training level (senior and junior) and were asked to perform a virtual reality hemilaminectomy. The position, angle, and force application of the simulated burr and suction instruments, along with tissue volumes that were removed, were recorded at 20-ms intervals. Raw data were manipulated to create metrics to train machine learning algorithms. Five algorithms, including a support vector machine, were trained to predict whether the task was performed by a senior or junior participant. The accuracy of each algorithm was assessed through leave-one-out cross-validation.

RESULTS

Forty-one individuals were enrolled (22 senior and 19 junior participants). Twelve metrics related to safety of the procedure, efficiency, motion of the tools, and coordination were selected. Following cross-validation, the support vector machine achieved a 97.6% accuracy. The other algorithms achieved accuracy of 92.7%, 87.8%, 70.7%, and 65.9%, respectively.

CONCLUSIONS

Artificial intelligence defined novel metrics of surgical performance and outlined training levels in a virtual reality spinal simulation procedure.

CLINICAL RELEVANCE

The significance of these results lies in the potential of artificial intelligence to complement current educational paradigms and better prepare residents for surgical procedures.

Video-Based Assessment in Surgical Education: A Scoping Review

BACKGROUND AND OBJECTIVE

Video-based assessment of residents' surgical skills may offer several advantages over direct observations of clinical performance in terms of objectivity, time-efficiency, and feasibility. Although video-based assessment is becoming more common in surgical training, a broad understanding of its utility is lacking. This scoping review explores video-based assessment in surgical training and presents the evidence supporting its use.

DESIGN

A literature search was conducted using the Web of Science database with key words related to video-based assessment and surgical training. Exclusion criteria included articles not published in English and articles on undergraduate medical education, continuing professional development, or non-surgical disciplines. Initially, 702 articles were identified; after title, abstract, and full-text screening by two independent reviewers (SM and VM), 199 articles remained.

RESULTS

We present the benefits of video-based assessment, including the ability to capture clinical ability in the operating room without decreasing intraoperative efficiency, as well as the potential to improve formative assessment and feedback practices. We describe the validity, reliability, and challenges of video-based assessment, as well as the use of video-based methods in clinical and simulated settings. We conclude by discussing questions that remain to be addressed.

CONCLUSIONS

Although further research and cost-benefit analyses are required, greater adoption of video-based assessment into surgical training may help meet increased assessment demands in an era of competency-based medical education.

Using Laser Range-finding to Measure Bore Depth in Surgical Drilling of Bone

BACKGROUND

Measuring drilled bore depth in bone is an important part of osteosynthesis surgery. Current methods have substantial limitations in terms of reliability, leading to placement of incorrectly sized screws and unsatisfactory user experience.

QUESTIONS/PURPOSES

(1) Can a prototype laser range-finder measure bore depth in bone as well as or better than a conventional depth gauge in terms of accuracy and precision (that is, variability)?

METHODS

A conventional analog orthopaedic surgical depth gauge was compared with a laser range-finder-based prototype. Experiments were conducted on four pig hind limbs, with bicortical holes drilled in the femur and the tibia. Two surgeons alternated drilling bores in three different clinically relevant conditions: straight drilling through the diaphysis, angled drilling through the diaphysis, and straight drilling through the metaphysis. Depth measurements were taken with the laser range-finder-based prototype, and the conventional depth gauge and compared against depth measurements obtained from a CT image that served as the reference measurement.

RESULTS

In straight diaphyseal drilling the laser range-finder-based prototype had a larger mean error of 1.34 mm (± 0.7 mm) compared with a mean error of -0.06 mm (± 1.38 mm) using the conventional gauge (95% CI 0.824 to 1.976; p < 0.001). In angled diaphyseal drilling, there was no difference in mean error between the laser range-finder-based prototype (1.66 ± 0.86 mm) and the conventional gauge (2.36 ± 3.79 mm [95% CI -2.338 to 0.938]; p = 0.393). In straight metaphyseal drilling, there was no difference in mean error between the laser range-finder-based prototype (2.11 ± 0.8 mm) and the conventional gauge (1.51 ± 3.19 mm [95% CI -0.500 to 1.700]; p = 0.280). The laser range-finder-based prototype had greater precision (smaller variance) than the conventional depth gauge in straight diaphyseal drilling (p < 0.001), angled diaphyseal drilling (p < 0.001), and straight metaphyseal drilling (p < 0.001).

CONCLUSIONS

A laser range-finder-based prototype mounted on a conventional surgical drill demonstrated overall similar accuracy and better precision in measuring drilled bore depth in bone compared with the conventional depth gauge.

CLINICAL RELEVANCE

A device based on this concept could improve the reliability of bore depth measurement in surgical practice and could therefore reduce the frequency of screw replacement and associated complications.

Simulation Training Methods in Neurological Surgery

Simulation training plays a paramount role in medicine, especially when it comes to mastering surgical skills. By simulating, students gain not only confidence, but expertise, learning to apply theory in a safe environment. As the technological arsenal improved, virtual reality and physical simulators have developed and are now an important part of the Neurosurgery training curriculum. Based on deliberate practice in a controlled space, simulation allows psychomotor skills augment without putting neither patients nor students at risk. When compared to the master-apprentice ongoing model of teaching, simutation becomes even more appealing as it is time-efficient, shortening the learning curve and ultimately leading to error reduction, which is reflected by diminished health care costs in the long run. In this chapter we will discuss the current state of neurosurgery simulation, highlight the potential benefits of this approach, assessing specific training methods and making considerations towards the future of neurosurgical simulation.

Trauma simulation training: a randomized controlled trial -evaluating the effectiveness of the Imperial Femoral Intramedullary Nailing Cognitive Task Analysis (IFINCTA) tool

Background and purpose - Cognitive task analysis (CTA) has been used extensively to train pilots and in other surgical specialties. However, the use of CTA within orthopedics is in its infancy. We evaluated the effectiveness of a novel CTA tool to improve understanding of the procedural steps in antegrade femoral intramedullary nailing. Material and methods - Design: A modified Delphi technique was used to generate a CTA from 3 expert orthopedic trauma surgeons for antegrade femoral intramedullary nailing. The written and audiovisual information was combined to describe the technical steps, decision points, and errors for each phase of this procedure Validation: A randomized double-blind controlled trial was undertaken with 22 medical students (novices) randomized into 2 equal groups. The intervention group were given the CTA tool and the control group were given a standard operative technique manual. They were assessed using the validated "Touch Surgery™" application assessment tool on femoral intramedullary nailing. Results - The pre-test scores between the two groups were similar. However, the post-test scores were statistically significantly better in the intervention group compared with the control group. The improvement (post-test median scores) in the intervention group compared with the control group was 20% for patient positioning and preparation, 21% for femoral preparation, 10% for proximal locking, and 19% for distal locking respectively (p < 0.001 for all comparisons). Interpretation - This is the first multimedia CTA tool in femoral intramedullary nailing that is easily accessible, user-friendly, and has demonstrated significant benefits in training novices over the traditional use of operative technique manuals.

Developing an objective assessment of surgical performance from operating room video and surgical imagery

An unbiased, repeatable process for assessing operating room performance is an important step toward quantifying the relationship between surgical training and performance. Hip fracture surgeries offer a promising first target in orthopedic trauma because they are common and they offer quantitative performance metrics that can be assessed from video recordings and intraoperative fluoroscopic images. Hip fracture repair surgeries were recorded using a head-mounted point-of-view camera. Intraoperative fluoroscopic images were also saved. The following performance metrics were analyzed: duration of wire navigation, number of fluoroscopic images collected, degree of intervention by the surgeon's supervisor, and the tip-apex distance (TAD). Two orthopedic traumatologists graded surgical performance in each video independently using an Objective Structured Assessment of Technical Skill (OSATS). Wire navigation duration correlated with weeks into residency and prior cases logged. TAD correlated with cases logged. There was no significant correlation between the OSATS total score and experience metrics. Total OSATS score correlated with duration and number of fluoroscopic images. Our results indicate that two metrics of hip fracture wire navigation performance, duration and TAD, significantly differentiate surgical experience. The methods presented have the potential to provide truly objective assessment of resident technical performance in the OR.

Use and Effectiveness of the Cadaver-Lab in Orthopaedic and Traumatology Education: An Italian Survey

Purpose  The purpose of this study is to provide basic information on the availability and current use of cadaver laboratories in the education of orthopaedic residents and trainees and to determine the interest for the implementation of this type of training.

Methods  All Orthopaedic residents and trainees who attended a cadaver laboratory organized by SIGASCOT (Italian Society of the Knee, Arthroscopy, Sports Traumatology, Cartilage and Orthopaedic Technology) between 2013 and 2016 were asked to complete a survey on the availability and current use of cadaver laboratories in the education of Orthopaedic residents and trainees. The survey was sent via e-mail to 102 Orthopaedic residents and trainees. All data were analyzed and all responses are presented as counts, percentages, or means.

Results  Thirty-eight (37.2%) Orthopaedics and traumatology residents and trainees completed the survey and were included in this analysis. Eighteen trainees (18/38; 44.3%) attended a cadaver laboratory focused on lower limb surgery, whereas 20 (20/38; 52.7%) on upper limb surgery. Twenty participants (55.7%) perceived skills laboratory sessions as extremely beneficial to the understanding and becoming familiar with the normal surgical anatomy; moreover, 16 (45.7%) participants considered the cadaver laboratory extremely beneficial to the understanding of a specific surgical technique and very beneficial (44.4%) to become confident with arthroscopic or other specific surgical instruments. Over 60% of participants perceived cadaver laboratory to be very to extremely beneficial to increase confidence and speed in the operating room (OR), and more than a half of them considered skills laboratory sessions to be extremely beneficial to increase participation and decrease the occurrence of damages in the real surgical activity.

Conclusion  Orthopaedic residents and trainees found the addition of a cadaver laboratory for teaching surgical skills a significant benefit to both their overall education and surgical skills training.

Level of Evidence  Level IV, survey study.

The timing and importance of motor skills course in knee arthroscopy training

Objectives

The aim of this prospective study is to evaluate the impact of the simulation training program in learning duration of arthroscopic motor skills. Furthermore, we investigated the difference between junior and experienced residents in the improvement of arthroscopic motor skills duration.

Methods

We established 2 study groups according to participants' year of experience in orthopedic residency with junior group residents with three years or less than three years experience as group 1 and experienced group with over 3 years of experience as group 2. We calculated duration change of motor skill test results for each participant before and after the course. The tools used were; auto scoring mirror tracer(ASMT), 0'Conner the tweezer dexterity test(OCTDT), etch-a-sketch with overlay(ESOT), purdue the pegboard test(PPT), two-arm coordination test(TACT) and grooved pegboard test(GPT) which were all produced by Lafayette firm. These instruments were used to practice and measure the basic motor skills.

Results

All post–course test durations for participants decreased significantly when compared to pre-course. We calculated percentage change of motor skill test results for each participant before and after the course. All motor skill test percentage changes were similar between two groups. In comparison of participants according to their experiences, results revealed that there was no difference in test results of experienced and junior surgeons. Both groups had provided equal improvement in terms of motor skills.

Conlusion

As our results revealed, residents will be able to act with a strong motivation to learn applications through basic arthroscopic information gained in early period of orthopedic training and will make more successful applications of real patients.

Variability in Trauma Case Volume in Orthopedic Surgery Residents

Orthopedic trauma surgery is a critical component of resident education. Surgical case logs obtained from the Accreditation Council of Graduate Medical Students from 2009 to 2013 for orthopedic surgery residents were examined for variability between the 90^th and 10^th percentiles in regards to the volume of cases performed. There was an upward trend in the mean number of cases performed by senior residents from 484.4 in 2009 to 534.5 in 2013, representing a 10.3% increase. There was a statistically significant increase in the number of cases performed for humerus/elbow, forearm/wrist, and pelvis/hip during this period (P<0.05). Although the difference between the 10^th and 90^th percentile case volumes narrowed over the study period, the difference between these groups remained significant in 2013 (P=0.02). In 2013, all categories of trauma cases had a greater than 2.2-fold difference between the 10^th and 90^th percentile of residents for numbers of trauma cases performed. Although case volume is not the sole determinant of residency education and competency, evidence suggests that case volume plays a crucial role in surgeon confidence and efficiency in performing surgery. Further studies are needed to better understand the effect of this variability seen among residents performing orthopedic trauma surgery.

Resident education in orthopaedic trauma

As residency training programmes around the globe move towards competency-based medical education (CBME), there is a need to review current teaching and assessment practices as they relate to education in orthopaedic trauma. Assessment is the cornerstone of CBME, as it not only helps to determine when a trainee is fit to practice independently, but it also provides feedback on performance and guides the development of competence. Although a standardised core knowledge base for trauma care has been developed by the leading national accreditation bodies and international agencies that teach and perform research in orthopaedic trauma, educators have not yet established optimal methods for assessing trainees’ performance in managing orthopaedic trauma patients.

This review describes the existing knowledge from the literature on assessment in orthopaedic trauma and highlights initiatives that have recently been undertaken towards CBME in the United Kingdom, Canada and the United States.

In order to support a CBME approach, programmes need to improve the frequency and quality of assessments and improve on current formative and summative feedback techniques in order to enhance resident education in orthopaedic trauma.

Cite this article: Bone Joint J 2016;98-B:1320–5.

Validation of the PASSPORT V2 training environment for arthroscopic skills

PURPOSE

Virtual reality simulators used in the education of orthopaedic residents often lack realistic haptic feedback. To solve this, the (Practice Arthroscopic Surgical Skills for Perfect Operative Real-life Treatment) PASSPORT simulator was developed, which was subjected to fundamental changes: improved realism and user interface. The purpose was to demonstrate its face and construct validity.

METHODS

Thirty-one participants were divided into three groups having different levels of arthroscopic experience. Participants answered questions regarding general information and the outer appearance of the simulator for face validity. Construct validity was assessed with one standardized navigation task, which was timed. Face validity, educational value and user-friendliness were determined with two representative exercises and by asking participants to fill out the questionnaire. A value of 7 or greater was considered sufficient.

RESULTS

Construct validity was demonstrated between experts and novices. Median task time for the fifth trial was 55 s (range 17-139 s) for the novices, 33 s (range 17-59 s) for the intermediates, and 26 s (range 14-52 s) for the experts. Median task times of three trials were not significantly different between the novices and intermediates, and none of the trials between intermediates and experts. Face validity, educational value and user-friendliness were perceived as sufficient (median >7). The presence of realistic tactile feedback was considered the biggest asset of the simulator.

CONCLUSION

Proper preparation for arthroscopic operations will increase the quality of real-life surgery and patients' safety. The PASSPORT simulator can assist in achieving this, as it showed construct and face validity, and its physical nature offered adequate haptic feedback during training. This indicates that PASSPORT has potential to evolve as a valuable training modality.

Sawbones laboratory in orthopedic surgical training

Sawbones are artificial bones designed to simulate the bone architecture, as well as the bone’s physical properties. The incorporation of sawbones simulation laboratories in many orthopedic training programs has provided the residents with flexibility in learning and scheduling that align with their working hour limitations. This review paper deliberates the organization of sawbones simulation in orthopedic surgical training to enhance trainee’s future learning. In addition, it explores the implications of sawbones simulation in orthopedic surgical teaching and evaluation. It scrutinizes the suitability of practicing on sawbones at the simulation laboratory to improve orthopedic trainee’s learning. This will be followed with recommendations for future enhancement of sawbones simulation-based learning in orthopedic surgical training.

Global Rating Scales and Motion Analysis Are Valid Proficiency Metrics in Virtual and Benchtop Knee Arthroscopy Simulators

BACKGROUND

Work-hour restrictions and fatigue management strategies in surgical training programs continue to evolve in an effort to improve the learning environment and promote safer patient care. In response, training programs must reevaluate how various teaching modalities such as simulation can augment the development of surgical competence in trainees. For surgical simulators to be most useful, it is important to determine whether surgical proficiency can be reliably differentiated using them. To our knowledge, performance on both virtual and benchtop arthroscopy simulators has not been concurrently assessed in the same subjects.

QUESTIONS/PURPOSES

(1) Do global rating scales and procedure time differentiate arthroscopic expertise in virtual and benchtop knee models? (2) Can commercially available built-in motion analysis metrics differentiate arthroscopic expertise? (3) How well are performance measures on virtual and benchtop simulators correlated? (4) Are these metrics sensitive enough to differentiate by year of training?

METHODS

A cross-sectional study of 19 subjects (four medical students, 12 residents, and three staff) were recruited and divided into 11 novice arthroscopists (student to Postgraduate Year [PGY] 3) and eight proficient arthroscopists (PGY 4 to staff) who completed a diagnostic arthroscopy and loose-body retrieval in both virtual and benchtop knee models. Global rating scales (GRS), procedure times, and motion analysis metrics were used to evaluate performance.

RESULTS

The proficient group scored higher on virtual (14 ± 6 [95% confidence interval {CI}, 10-18] versus 36 ± 5 [95% CI, 32-40], p < 0.001) and benchtop (16 ± 8 [95% CI, 11-21] versus 36 ± 5 [95% CI, 31-40], p < 0.001) GRS scales. The proficient subjects completed nearly all tasks faster than novice subjects, including the virtual scope (579 ±169 [95% CI, 466-692] versus 358 ± 178 [95% CI, 210-507] seconds, p = 0.02) and benchtop knee scope + probe (480 ± 160 [95% CI, 373-588] versus 277 ± 64 [95% CI, 224-330] seconds, p = 0.002). The built-in motion analysis metrics also distinguished novices from proficient arthroscopists using the self-generated virtual loose body retrieval task scores (4 ± 1 [95% CI, 3-5] versus 6 ± 1 [95% CI, 5-7], p = 0.001). GRS scores between virtual and benchtop models were very strongly correlated (ρ = 0.93, p < 0.001). There was strong correlation between year of training and virtual GRS (ρ = 0.8, p < 0.001) and benchtop GRS (ρ = 0.87, p < 0.001) scores.

CONCLUSIONS

To our knowledge, this is the first study to evaluate performance on both virtual and benchtop knee simulators. We have shown that subjective GRS scores and objective motion analysis metrics and procedure time are valid measures to distinguish arthroscopic skill on both virtual and benchtop modalities. Performance on both modalities is well correlated. We believe that training on artificial models allows acquisition of skills in a safe environment. Future work should compare different modalities in the efficiency of skill acquisition, retention, and transferability to the operating room.

Constructing a validity argument for the Objective Structured Assessment of Technical Skills (OSATS): a systematic review of validity evidence

In order to construct and evaluate the validity argument for the Objective Structured Assessment of Technical Skills (OSATS), based on Kane's framework, we conducted a systematic review. We searched MEDLINE, EMBASE, CINAHL, PsycINFO, ERIC, Web of Science, Scopus, and selected reference lists through February 2013. Working in duplicate, we selected original research articles in any language evaluating the OSATS as an assessment tool for any health professional. We iteratively and collaboratively extracted validity evidence from included articles to construct and evaluate the validity argument for varied uses of the OSATS. Twenty-nine articles met the inclusion criteria, all focussed on surgical technical skills assessment. We identified three intended uses for the OSATS, namely formative feedback, high-stakes assessment and program evaluation. Following Kane's framework, four inferences in the validity argument were examined (scoring, generalization, extrapolation, decision). For formative feedback and high-stakes assessment, there was reasonable evidence for scoring and extrapolation. However, for high-stakes assessment there was a dearth of evidence for generalization aside from inter-rater reliability data and an absence of evidence linking multi-station OSATS scores to performance in real clinical settings. For program evaluation, the OSATS validity argument was supported by reasonable generalization and extrapolation evidence. There was a complete lack of evidence regarding implications and decisions based on OSATS scores. In general, validity evidence supported the use of the OSATS for formative feedback. Research to provide support for decisions based on OSATS scores is required if the OSATS is to be used for higher-stakes decisions and program evaluation.

Kinematic analysis of wrist motion during simulated colonoscopy in first-year gastroenterology fellows

BACKGROUND AND STUDY AIMS

Gastroenterology trainees acquire skill and proficiency in performing colonoscopies at different rates. The cause for heterogeneous competency among the trainees is unclear. Kinematic analysis of the wrist joint while performing colonoscopy can objectively assess the variation in wrist motion. Our objective was to test the hypothesis that the time spent by the trainees in extreme ranges of wrist motion will decrease as the trainees advance through the fellowship year.

SUBJECTS AND METHODS

Five first-year gastroenterology fellows were prospectively studied at four intervals while performing simulated colonoscopies. The setting was an endoscopy simulation laboratory at a tertiary care center. Kinematic assessment of wrist motion was done using a magnetic position/orientation tracker held in place by a custom-made arm sleeve and hand glove. The main outcome measure was time spent performing each of four ranges of wrist motion (mid, center, extreme, and out) for each wrist degree of freedom (pronation/supination, flexion/extension, and adduction/abduction).

RESULTS

There were no statistically significant differences in the time spent for wrist movements across the three degrees of freedom throughout the study period. However, fellows spent significantly less time in extreme range (1.47 ± 0.34 min vs. 2.44 ± 0.34 min, P = 0.004) and center range (1.02 ± 0.34 min vs 1.9 ± 0.34 min, P = 0.01) at the end of the study compared to the baseline evaluation. The study was limited by the small number of subjects and performance of colonoscopies on a simulator rather than live patients.

CONCLUSIONS

Gastroenterology trainees alter the time spent at the extreme range of wrist motion as they advance through training. Endoscopy training during the first 10 months of fellowship may have beneficial effects on learning ergonomically correct motion patterns.

A Hybrid Reality Radiation-Free Simulator for Teaching Wire Navigation Skills

OBJECTIVES

Surgical simulation is an increasingly important method to facilitate the acquiring of surgical skills. Simulation can be helpful in developing hip fracture fixation skills because it is a common procedure for which performance can be objectively assessed [ie, the tip-apex distance (TAD)]. The procedure requires fluoroscopic guidance to drill a wire along an osseous trajectory to a precise position within bone. The objective of this study was to assess the construct validity for a novel radiation-free simulator designed to teach wire navigation skills in hip fracture fixation.

METHODS

Novices (n = 30) with limited to no surgical experience in hip fracture fixation and experienced surgeons (n = 10) participated. Participants drilled a guide wire in the center-center position of a synthetic femoral head in a hip fracture simulator, using electromagnetic sensors to track the guide-wire position. Sensor data were gathered to generate fluoroscopic-like images of the hip and guide wire. Simulator performance of novice and experienced participants was compared to measure construct validity.

RESULTS

The simulator was able to discriminate the accuracy in guide-wire position between novices and experienced surgeons. Experienced surgeons achieved a more accurate TAD than novices (13 vs. 23 mm, respectively, P = 0.009). The magnitude of improvement on successive simulator attempts was dependent on the level of expertise; TAD improved significantly in the novice group, whereas it was unchanged in the experienced group.

CONCLUSIONS

This hybrid reality, radiation-free hip fracture simulator, which combines real-world objects with computer-generated imagery, demonstrates construct validity by distinguishing the performance of novices and experienced surgeons. There is a differential effect depending on the level of experience, and it could be used as an effective training tool in novice surgeons.

Do orthopaedic fracture skills courses improve resident performance?

INTRODUCTION

We hypothesized that resident participation in a hands-on fracture fixation course leads to significant improvement in their performance as assessed in a simulated fracture fixation model.

METHODS

Twenty-three junior orthopaedic surgery residents were tasked to treat radial shaft fractures with standard fixation techniques in a sawbones fracture fixation simulation twice during the year. Before the first simulation, 6 of the residents participated in a fraction fixation skills course. The simulation repeated 6 months later after all residents attended the course. Residents also completed a 15-question written examination. Assessment included evaluation of each step of the procedure, a score based on the objective structured assessment of technical skill (OSATS) system, and grade on the examination. Comparisons were made between the two cohorts and the two testing time points.

RESULTS

Significant improvements were present in the percentage of tasks completed correctly (64.1% vs 84.3%) the overall OSATS score (13.8 vs 17.1) and examination correct answers (8.6 vs 12.5) for the overall cohort between the two testing time points (p<0.001, p<0.03, p<0.04 respectively). Residents who had not participated in the surgical skills course at the time of their initial simulation demonstrated significant improvements in percentage of tasks completed correctly (61.3% vs 81.2%) and OSATS score (12.4 vs 17.0) (p<0.002, p<0.01 respectively). No significant difference was noted in performance for the cohort who had already participated in the course (p=0.87 and p=0.68). The cohort that had taken the course prior to the initial simulation showed significantly higher scores at initial evaluation (88.5% vs 58.5% percentage of tasks completed correctly, 17.3 vs 12.0 OSATS score, 12.5 vs 8.6 correct answers on the examination). At the second simulation, no significant difference was seen with task completion or examination grade, but a significant difference still existed with respect to the OSATS score (20.0 vs 17.0; p<0.03).

CONCLUSION

Participation in a formal surgical skills course significantly improved practical operative skills as assessed by the simulation. The benefits of the course were maintained to 6 months with residents who completed the training earlier continuing to demonstrate an advantage in skills. Such courses are a valuable training resource which directly impact resident performance.

Technical skills assessment toolbox: a review using the unitary framework of validity

OBJECTIVE

The purpose of this study was to create a technical skills assessment toolbox for 35 basic and advanced skills/procedures that comprise the American College of Surgeons (ACS)/Association of Program Directors in Surgery (APDS) surgical skills curriculum and to provide a critical appraisal of the included tools, using contemporary framework of validity.

BACKGROUND

Competency-based training has become the predominant model in surgical education and assessment of performance is an essential component. Assessment methods must produce valid results to accurately determine the level of competency.

METHODS

A search was performed, using PubMed and Google Scholar, to identify tools that have been developed for assessment of the targeted technical skills.

RESULTS

A total of 23 assessment tools for the 35 ACS/APDS skills modules were identified. Some tools, such as Operative Performance Rating System (OSATS) and Objective Structured Assessment of Technical Skill (OPRS), have been tested for more than 1 procedure. Therefore, 30 modules had at least 1 assessment tool, with some common surgical procedures being addressed by several tools. Five modules had none. Only 3 studies used Messick's framework to design their validity studies. The remaining studies used an outdated framework on the basis of "types of validity." When analyzed using the contemporary framework, few of these studies demonstrated validity for content, internal structure, and relationship to other variables.

CONCLUSIONS

This study provides an assessment toolbox for common surgical skills/procedures. Our review shows that few authors have used the contemporary unitary concept of validity for development of their assessment tools. As we progress toward competency-based training, future studies should provide evidence for various sources of validity using the contemporary framework.

Surgical skills simulation in trauma and orthopaedic training

Changing patterns of health care delivery and the rapid evolution of orthopaedic surgical techniques have made it increasingly difficult for trainees to develop expertise in their craft. Working hour restrictions and a drive towards senior led care demands that proficiency be gained in a shorter period of time whilst requiring a greater skill set than that in the past. The resulting conflict between service provision and training has necessitated the development of alternative methods in order to compensate for the reduction in ‘hands-on’ experience. Simulation training provides the opportunity to develop surgical skills in a controlled environment whilst minimising risks to patient safety, operating theatre usage and financial expenditure. Many options for simulation exist within orthopaedics from cadaveric or prosthetic models, to arthroscopic simulators, to advanced virtual reality and three-dimensional software tools. There are limitations to this form of training, but it has significant potential for trainees to achieve competence in procedures prior to real-life practice. The evidence for its direct transferability to operating theatre performance is limited but there are clear benefits such as increasing trainee confidence and familiarity with equipment. With progressively improving methods of simulation available, it is likely to become more important in the ongoing and future training and assessment of orthopaedic surgeons.

The role of simulation in developing surgical skills

Surgical training has followed the master-apprentice model for centuries but is currently undergoing a paradigm shift. The traditional model is inefficient with no guarantee of case mix, quality, or quantity. There is a growing focus on competency-based medical education in response to restrictions on doctors' working hours and the traditional mantra of "see one, do one, teach one" is being increasingly questioned. The medical profession is subject to more scrutiny than ever before and is facing mounting financial, clinical, and political pressures. Simulation may be a means of addressing these challenges. It provides a way for trainees to practice technical tasks in a protected environment without putting patients at risk and helps to shorten the learning curve. The evidence for simulation-based training in orthopedic surgery using synthetic models, cadavers, and virtual reality simulators is constantly developing, though further work is needed to ensure the transfer of skills to the operating theatre.

The need for specialist training exposure in residency programs

In modern orthopedic practice, some degree of subspecialist expertise is required and training programs must provide for this. Specialist training in university teaching hospitals provides opportunities beyond just exposure to the complex case mix of a tertiary center. Specialist orthopedic hospitals provide care and leadership with national and international expertise. Their role in orthopedic training is substantial. With legislative reductions in permissible working hours orthopedic training continues to adapt to provide surgeons with the opportunities to fully develop skills, with teaching hospitals well placed to facilitate this. This article discusses the role of specialist training and teaching hospitals in contemporary orthopedic surgery.

Composite bone models in orthopaedic surgery research and education

Composite bone models are increasingly used in orthopaedic biomechanics research and surgical education-applications that traditionally relied on cadavers. Cadaver bones are suboptimal for many reasons, including issues of cost, availability, preservation, and inconsistency between specimens. Further, cadaver samples disproportionately represent the elderly, whose bone quality may not be representative of the greater orthopaedic population. The current fourth-generation composite bone models provide an accurate reproduction of the biomechanical properties of human bone when placed under bending, axial, and torsional loads. The combination of glass fiber and epoxy resin components into a single phase has enabled manufacturing by injection molding. The high level of anatomic fidelity of the cadaver-based molds and negligible shrinkage properties of the epoxy resin results in a process that allows for excellent definition of anatomic detail in the cortical wall and optimized consistency of features between models. Recent biomechanical studies of composites have validated their use as a suitable substitute for cadaver specimens.

A Locking Contoured Plate for Distal Fibular Fractures: Mechanical Evaluation in an Osteoporotic Bone Model Using Screws of Different Length

Osteoporotic bone with poor mechanical capacity provides limited stability after fixation of ankle fractures. Stabilization with an implant providing increased fixation strength in osteoporotic bone could reduce failure rates of fixation and allow a more functional treatment. The purpose of this study was to evaluate a locking contoured plate for fixation of distal fibular fractures in comparison with a conventional contoured plate in an osteoporotic bone model. Eighty cylinders of osteoporotic bone surrogates were fixed with the two plates. We performed torque-to-failure and cyclic testing experiments using screws of different length with a Zwick/Roell testing machine. The locking system showed higher torque-to-failure and maximum torque levels as compared with the conventional plate in torque-to-failure experiments and torsional cyclic testing. The locking contoured plate provides improved fixation strength in the osteoporotic bone model. The locking system may be appropriate for fixation of distal fibular fractures, especially in osteoporotic bone with poor mechanical capacity.

Validating a global rating scale to monitor individual resident learning curves during arthroscopic knee meniscal repair

PURPOSE

To determine whether a global rating scale (GRS) with construct validity can also be used to assess the learning curve of individual orthopaedic trainees during simulated arthroscopic knee meniscal repair.

METHODS

An established arthroscopic GRS was used to evaluate the technical skill of 19 orthopaedic residents performing a standardized arthroscopic meniscal repair in a bioskills laboratory. The residents had diagnostic knee arthroscopy experience but no experience with arthroscopic meniscal repair. Residents were videotaped performing an arthroscopic meniscal repair on 12 separate occasions. Their performance was assessed by use of the GRS and motion analysis objectively measuring the time taken to complete tasks, path length of the subject's hands, and number of hand movements. One author assessed all 228 videos, whereas 2 other authors rated 34 randomly selected videos, testing the interobserver reliability of the GRS. The validity of the GRS was tested against the motion analysis.

RESULTS

Objective assessment with motion analysis defined the surgeon's learning curve, showing significant improvement by each subject over 12 episodes (P < .0001). The GRS also showed a similar learning curve with significant improvements in performance (P < .0001). The median GRS score improved from 15 of 34 (interquartile range, 14 to 17) at baseline to 22 of 34 (interquartile range, 19 to 23) in the final period. There was a moderate correlation (P < .0001, Spearman test) between the GRS and motion analysis parameters (r = -0.58 for time, r = -0.58 for path length, and r = -0.51 for hand movements). The inter-rater reliability among 3 trained assessors using the GRS was excellent (Cronbach α = 0.88).

CONCLUSIONS

When compared with motion analysis, an established arthroscopic GRS, with construct validity, also offers a moderately feasible method to monitor the learning curve of individual residents during simulated knee meniscal repair.

CLINICAL RELEVANCE

An arthroscopic GRS can be used for monitoring skill improvement during knee meniscal repair and has the potential for use as a training and assessment tool in the real operating room.

Arthroscopic proficiency: methods in evaluating competency

BACKGROUND

The current paradigm of arthroscopic training lacks objective evaluation of technical ability and its adequacy is concerning given the accelerating complexity of the field. To combat insufficiencies, emphasis is shifting towards skill acquisition outside the operating room and sophisticated assessment tools. We reviewed (1) the validity of cadaver and surgical simulation in arthroscopic training, (2) the role of psychomotor analysis and arthroscopic technical ability, (3) what validated assessment tools are available to evaluate technical competency, and (4) the quantification of arthroscopic proficiency.

METHODS

The Medline and Embase databases were searched for published articles in the English literature pertaining to arthroscopic competence, arthroscopic assessment and evaluation and objective measures of arthroscopic technical skill. Abstracts were independently evaluated and exclusion criteria included articles outside the scope of knee and shoulder arthroscopy as well as original articles about specific therapies, outcomes and diagnoses leaving 52 articles cited in this review.

RESULTS

Simulated arthroscopic environments exhibit high levels of internal validity and consistency for simple arthroscopic tasks, however the ability to transfer complex skills to the operating room has not yet been established. Instrument and force trajectory data can discriminate between technical ability for basic arthroscopic parameters and may serve as useful adjuncts to more comprehensive techniques. There is a need for arthroscopic assessment tools for standardized evaluation and objective feedback of technical skills, yet few comprehensive instruments exist, especially for the shoulder. Opinion on the required arthroscopic experience to obtain proficiency remains guarded and few governing bodies specify absolute quantities.

CONCLUSIONS

Further validation is required to demonstrate the transfer of complex arthroscopic skills from simulated environments to the operating room and provide objective parameters to base evaluation. There is a deficiency of validated assessment tools for technical competencies and little consensus of what constitutes a sufficient case volume within the arthroscopy community.

Preliminary study of the feasibility and accuracy of percutaneous peri-acetabular screw insertion in a porcine model

The aim of this pilot study was to assess a new method of training for peri-acetabular screw placement under indirect vision using standard C-arm fluoroscopy using a porcine model. Two novice orthopaedic residents placed 72 screws (36 each) about the acetabula of six porcine pelves under C-arm fluoroscopic guidance. Unsatisfactory screw position was noted in 22 of 72, with five instances of screw ingress into the hip joint. All of these cases occurred in the first half of each resident's series. Screw direction and final position improved over subsequent trials. This pilot study demonstrates that surgical simulation techniques are applicable in percutaneous screw fixation. Such an approach could be useful for both residents in training and more experienced surgeons who wish to perform this procedure in cases where it is appropriate.

Validation of a novel method for assessing competency in polypectomy

BACKGROUND

There is a gap in the formal assessment of technical skills in polypectomy that is now considered an integral part of colonoscopy. Polypectomy has been shown to reduce the incidence of colorectal cancer but does have associated complications. Polypectomy competency assessment should arguably be a part of the certification process for all endoscopists. A polypectomy competency assessment tool (Direct Observation of Polypectomy Skills [DOPyS]) has been developed and its reliability examined. This study examined the ability of the DOPyS to reliably distinguish between endoscopists with different levels of experience, ie, its construct validity.

OBJECTIVE

To determine the construct validity of the DOPyS.

DESIGN

Videos of 32 polypectomies (endoscopic view only) were collected from 2 expert (> 1000 colonoscopies) endoscopists (17 polyps) and 6 intermediate-level (100-500 colonoscopies) endoscopists (15 polyps). The videos were edited to include only the entire polypectomy procedure, arranged in random order, and assessed blindly by 4 experienced endoscopists, only 2 of whom were familiar with polypectomy assessment by using the DOPyS before scoring. The differences in overall competency scores (range 1-4; competency, scores ≥ 3) for the expert and intermediate groups were compared by using the Fisher exact test.

SETTING

Single center.

RESULTS

The analysis suggested that both trained assessors familiar with the DOPyS could reliably distinguish between the expert and intermediate endoscopists (P = .049 and P < .001), with the expert group scoring higher than the intermediate one. For the assessors with no previous experience of the DOPyS, no such difference could be seen (P = .71 and P = .15).

LIMITATIONS

Small sample and polyp size.

CONCLUSIONS

The results of the analysis suggested that the DOPyS could reliably differentiate between polypectomies performed by endoscopists of different levels of experience, but only if the assessors were trained in the use of the assessment tool. Training is therefore required to use this tool reliably.

Observational tools for assessment of procedural skills: a systematic review

BACKGROUND

Assessment by direct observation of procedural skills is an important source of constructive feedback. The aim of this study was to identify observational tools for technical skill assessment, to assess characteristics of these tools, and to assess their usefulness for assessment.

METHODS

Included studies reported tools for observational assessment of technical skills. A total of 106 articles were included.

RESULTS

Three main categories included global assessment scales evaluating generic skills (n = 29), task-specific methods assessing procedure-specific skills (n = 30), and combinations of tools evaluating both generic and task-specific skills (n = 47). In most studies, content validity was not evaluated using an accepted scientific method. All tools were assessed for inter-rater reliability and construct validity. Data on feasibility, acceptability, and educational impact were sparse.

CONCLUSIONS

There is evidence of validity and reliability for observational assessment tools at the trainee level. In most studies a comprehensive analysis of the tools was not achieved. Evaluation of technical skill using current observational assessment tools is not reliable and valid at the specialist level. Future research needs to focus on further systematic tool development and analysis, especially at the specialist level.

Development and validation of a novel method for assessing competency in polypectomy: direct observation of polypectomy skills

BACKGROUND

Despite its ubiquitous use over the past 4 decades, there is no structured, formal method with which to assess polypectomy.

OBJECTIVE

To develop and validate a new method with which to assess competency in polypectomy.

DESIGN

Polypectomy underwent task deconstruction, and a structured checklist and global assessment scale were developed (direct observation of polypectomy skills [DOPyS]). Sixty bowel cancer screening polypectomy videos were randomly chosen for analysis and were scored independently by 7 expert assessors by using DOPyS. Each parameter and the global rating were scored from 1 to 4 (scores ≥3 = competency). The scores were analyzed by using generalizability theory (G theory).

SETTING

Multicenter.

RESULTS

Fifty-nine of the 60 videos were assessable and scored. The majority of the assessors agreed across the pass/fail divide for the global assessment scale in 58 of 59 (98%) polyps. For G-theory analysis, 47 of the 60 videos were analyzed. G-theory analysis suggested that DOPyS is a reliable assessment tool, provided that it is used by 2 assessors to score 5 polypectomy videos all performed by 1 endoscopist. DOPyS scores obtained in this format would reflect the endoscopist's competence.

LIMITATIONS

Small sample and polyp size.

CONCLUSIONS

This study is the first attempt to develop and validate a tool designed specifically for the assessment of technical skills in performing polypectomy. G-theory analysis suggests that DOPyS could reliably reflect an endoscopist's competence in performing polypectomy provided a requisite number of assessors and cases were used.