Assessment of surgery residents' operative skills in the operating theater using a modified Objective Structured Assessment of Technical Skills (OSATS): a prospective multicenter study

A New Procedure-Based Assessment of Operative Skills in Gastric Bypass Surgery, Evaluated by Video Fragment Rating

PURPOSE

Feedback on technical and procedural skills is essential during the training of residents and fellows. The aim of this study was to assess the performance of a newly created instrument for the assessment of operative skills using laparoscopic Roux-en-Y gastric bypass (LRYGB) video fragments.

MATERIALS AND METHODS

A new procedure-based assessment (PBA) was created by combining LRYGB key steps with a 5-point independence scale. LRYGB performed by residents and surgeons with different levels of expertise were video recorded. Fragments of the pouch creation, gastro-jejunostomy and jejunojejunostomy, were review by 12 expert bariatric surgeons and the operative skills assessed with the PBA, Objective Structured Assessment of Technical Skill (OSATS), and the Bariatric OSATS (BOSATS). The PBA was compared to the OSATS and BOSATS. Mean scores for all items of the different assessments were summarized and compared using a T-test.

RESULTS

The scores of the procedural steps were combined and compared for all levels. The mean scores for beginner, intermediate, and expert level were 2.71, 3.70, and 3.90 for the PBA; for the OSATS 1.84, 2.86, and 3.44; and for the BOSATS 2.78, 3.56, and 4.19. Each of these assessments differentiated between the three skill levels (all p < 0.05).

CONCLUSION

The PBA discriminates well between different levels of operative skills. Similar patterns were found for the OSATS and BOSATS, showing that the randomly selected video fragments are representative samples for assessing skill level. Future research will demonstrate whether these results can be extrapolated to clinical training, and which scores allow for procedure certification.

Using Artificial Intelligence to Gauge Competency on a Novel Laparoscopic Training System

OBJECTIVE

Laparoscopic surgical skill assessment and machine learning are often inaccessible to low-and-middle-income countries (LMIC). Our team developed a low-cost laparoscopic training system to teach and assess psychomotor skills required in laparoscopic salpingostomy in LMICs. We performed video review using AI to assess global surgical techniques. The objective of this study was to assess the validity of artificial intelligence (AI) generated scoring measures of laparoscopic simulation videos by comparing the accuracy of AI results to human-generated scores.

DESIGN

Seventy-four surgical simulation videos were collected and graded by human participants using a modified OSATS (Objective Structured Assessment of Technical Skills). The videos were then analyzed via AI using 3 different time and distance-based calculations of the laparoscopic instruments including path length, dimensionless jerk, and standard deviation of tool position. Predicted scores were generated using 5-fold cross validation and K-Nearest-Neighbors to train classifiers.

SETTING

Surgical novices and experts from a variety of hospitals in Ethiopia, Cameroon, Kenya, and the United States contributed 74 laparoscopic salpingostomy simulation videos.

RESULTS

Complete accuracy of AI compared to human assessment ranged from 65-77%. There were no statistical differences in rank mean scores for 3 domains, Flow of Operation, Respect for Tissue, and Economy of Motion, while there were significant differences in ratings for Instrument Handling, Overall Performance, and the total summed score of all 5 domains (Summed). Estimated effect sizes were all less than 0.11, indicating very small practical effect. Estimated intraclass correlation coefficient (ICC) of Summed was 0.72 indicating moderate correlation between AI and Human scores.

CONCLUSIONS

Video review using AI technology of global characteristics was similar to that of human review in our laparoscopic training system. Machine learning may help fill an educational gap in LMICs where direct apprenticeship may not be feasible.

Assessment of Technical Competence in Distal Radius Fracture Fixation by a Volar Locking Plate: A Global Delphi Consensus Study

PURPOSE

Volar locking plate fixation of distal radius fractures is a common orthopedic procedure and should be mastered by graduating orthopedic residents. Surgical education is transitioning from a traditional time-based approach to competency-based medical education. Valid and objective assessment is essential for successful transition. The purpose of this study was to develop a comprehensive, procedure-specific assessment tool to evaluate technical competence in volar locking plate osteosynthesis of a distal radius fracture.

METHODS

International orthopedic/trauma experts involved in resident education participated as panelists in a four-round online Delphi process to reach consensus on the content of the assessment tool. Round 1 was an item-generating round, in which the panelists identified potential assessment parameters. In round 2, the panelists rated the importance of each suggested assessment parameter and reached consensus on which to include in the assessment tool. Round 3 yielded specific assessment score intervals for specific bone and fracture models and is not reported in this study. In round 4, the panelists assigned weights to the assessment parameters on a 1-10 scale to determine how each parameter should have an impact on the overall results.

RESULTS

Eighty-seven surgeons, representing 42 countries, participated in the study. Round 1 resulted in 45 assessment parameters, grouped into five procedural steps. After round 2, the number of parameters was reduced to 39. After the final round, an additional parameter was removed and weights were assigned to the remaining parameters.

CONCLUSIONS

Using a systematic methodology, a preliminary assessment tool to evaluate technical competence in distal radius fracture fixation was developed. A consensus of international experts supports the content validity of the assessment tool.

CLINICAL RELEVANCE

This assessment tool represents the first step in the evidence-based assessment essential for competency-based medical education. Before implementation, further studies exploring validity of variations of the assessment tool in different educational contexts are required.

Learning surgical knot tying and suturing technique - effects of different forms of training in a controlled randomized trial with dental students

Objective

The acquisition of surgical skills requires motor learning. A special form of this is intermanual transfer by transferring motor skills from the nondominant hand (NDH) to the dominant hand (DH). The purpose of this study was to determine the learning gains that can be achieved for the DH by training with the DH, the NDH, and by non-surgical alternative training (AT).

Methods

124 preclinical (n=62) and clinical (n=62) dental students completed surgical knot tying and suturing technique training with the DH, with the NDH, and an AT in a controlled randomized trial.

Results

A statistically significant learning gain in knot tying and suture technique with the DH was evident only after training with the DH when compared to training with the NDH (p<0.001 and p=0.004, respectively) and an AT (p=0.001 and p=0.010, respectively). Of those students who achieved a learning gain ≥4 OSATS points, 46.4% (n=32) benefited in their knot tying technique with the DH from training with the DH, 29.0% (n=20) from training with the NDH, and 24.6% (n=17) from an AT while 45.7% (n=32) benefited in their suturing technique with the DH from training with the DH, 31.4% (n=22) from training with the NDH, and 22, 9% (n=16) from an AT.

Conclusions

Training with the DH enabled significantly better learning gains in the surgical knot tying and suturing techniques with the DH.

Universal meta-competencies of operative performances: a literature review and qualitative synthesis

BACKGROUND

Prioritizing patient health is essential, and given the risk of mortality, surgical techniques should be objectively evaluated. However, there is no comprehensive cross-disciplinary system that evaluates skills across all aspects among surgeons of varying levels. Therefore, this study aimed to uncover universal surgical competencies by decomposing and reconstructing specific descriptions in operative performance assessment tools, as the basis of building automated evaluation system using computer vision and machine learning-based analysis.

METHODS

The study participants were primarily expert surgeons in the gastrointestinal surgery field and the methodology comprised data collection, thematic analysis, and validation. For the data collection, participants identified global operative performance assessment tools according to detailed inclusion and exclusion criteria. Thereafter, thematic analysis was used to conduct detailed analyses of the descriptions in the tools where specific rules were coded, integrated, and discussed to obtain high-level concepts, namely, "Skill meta-competencies." "Skill meta-competencies" was recategorized for data validation and reliability assurance. Nine assessment tools were selected based on participant criteria.

RESULTS

In total, 189 types of skill performances were extracted from the nine tool descriptions and organized into the following five competencies: (1) Tissue handling, (2) Psychomotor skill, (3) Efficiency, (4) Dissection quality, and (5) Exposure quality. The evolutionary importance of these competences' different evaluation targets and purpose over time were assessed; the results showed relatively high reliability, indicating that the categorization was reproducible. The inclusion of basic (tissue handling, psychomotor skill, and efficiency) and advanced (dissection quality and exposure quality) skills in these competencies enhanced the tools' comprehensiveness.

CONCLUSIONS

The competencies identified to help surgeons formalize and implement tacit knowledge of operative performance are highly reproducible. These results can be used to form the basis of an automated skill evaluation system and help surgeons improve the provision of care and training, consequently, improving patient prognosis.

Multi-Modal Deep Learning for Assessing Surgeon Technical Skill

This paper introduces a new dataset of a surgical knot-tying task, and a multi-modal deep learning model that achieves comparable performance to expert human raters on this skill assessment task. Seventy-two surgical trainees and faculty were recruited for the knot-tying task, and were recorded using video, kinematic, and image data. Three expert human raters conducted the skills assessment using the Objective Structured Assessment of Technical Skill (OSATS) Global Rating Scale (GRS). We also designed and developed three deep learning models: a ResNet-based image model, a ResNet-LSTM kinematic model, and a multi-modal model leveraging the image and time-series kinematic data. All three models demonstrate performance comparable to the expert human raters on most GRS domains. The multi-modal model demonstrates the best overall performance, as measured using the mean squared error (MSE) and intraclass correlation coefficient (ICC). This work is significant since it demonstrates that multi-modal deep learning has the potential to replicate human raters on a challenging human-performed knot-tying task. The study demonstrates an algorithm with state-of-the-art performance in surgical skill assessment. As objective assessment of technical skill continues to be a growing, but resource-heavy, element of surgical education, this study is an important step towards automated surgical skill assessment, ultimately leading to reduced burden on training faculty and institutes.

Automated Microsurgical Tool Segmentation and Characterization in Intra-Operative Neurosurgical Videos

Checklist based routine evaluation of surgical skills in any medical school demands quality time and effort from the supervising expert and is highly influenced by assessor bias. Alternatively, automated video based surgical skill assessment is a simple and viable method to analyse surgical dexterity offline without the need for acute presence of an expert surgeon throughout the surgery. In this paper, a novel approach and results for the automated segmentation of microsurgical instruments from the real-world neurosurgical video dataset was presented. The proposed tool segmentation model showcased mean average precision of 96.7% in detecting, and localizing five surgical instruments from the real-world neurosurgical videos. Accurate detection and characterization of motion features of the microsurgical tool from the novel annotated neurosurgical video dataset forms the key step towards automated surgical skill evaluation. Clinical Relevance- Tool segmentation, localization, and characterization in neurosurgical video, has several applications including assessing surgeons skills, training novice surgeons, understanding critical operating procedures post surgery, characterizing any critical anatomical response to the tool that leads to the success or failure of the surgery, and building models for conducting autonomous robotic surgery. Semantic segmentation, and characterization of the microsurgical tools forms the basis of the modern neurosurgery.

Feasibility of Smartphone Application in Plastic Surgery Operative Assessments

Background:

The evaluation model of operative competence is based on aggregate tabulations of procedures and end-of-rotation feedback from faculty members. Procedural tabulations do not detail the level of resident involvement in the case, and end of rotation feedback is infrequent and inaccurate due to the necessity of long-term recall. Smart phone-based evaluation systems provide residents with immediate and permanent feedback for surgical encounters. In this study, we examine the feasibility of smart phone-based evaluations in plastic surgery residency.

Methods:

This was a 6-month prospective, single institution pilot study at three teaching hospitals, assessing all PGY levels. We utilized our department mobile application (Wayne State University Surgery Department application), which includes intraoperative evaluations based on the Zwisch scale. Prestudy and poststudy surveys were conducted. An unstructured interview of the Clinical Competency Committee provided feedback for the new evaluation tool against the previous evaluation forms.

Results:

Eleven physicians participated in the study, resulting in 126 encounters and 184 procedures. A 10-question prestudy survey was given with answers ranging from 1 (strongly disagree) to 5 (strongly agree). The Clinical Competency Committee faculty ranked the prestudy resident assessment tool 2.82 of 5, whereas the poststudy survey scored 4.64 of 5.

Conclusions:

Residents and faculty both rated the smartphone application as a useful tool for evaluating residents. The success of the application proves its feasibility within plastic surgery residency and may play an important role in rating resident operative competency in the future.

The "chicken-leg anastomosis": Low-cost tissue-realistic simulation model for esophageal atresia training in pediatric surgery

INTRODUCTION

Shifting the training from the operating room (OR) to simulation models has been proven effective in enhancing patient safety and reducing the learning time to achieve competency and increase the operative efficiency. Currently the field of pediatric surgery only offers few low-cost trainers for specialized training and these feature predominantly artificial and often unrealistic tissue. The aim of this study was to develop an easy access low-cost tissue-realistic simulation model for open training of esophageal atresia and to evaluate the acceptance in trainees and junior pediatric surgeons.

MATERIALS AND METHODS

The model is fashioned using reconfigured chicken skin from a chicken leg. To create a model of esophageal atresia, the chicken skin is dissected off the muscle and reconfigured around a foley catheter balloon to recreate the proximal pouch and a feeding tube to recreate the distal pouch. Surrounding structures such as the tracheo-esophageal fistula and the azygos vein can be easily added, obtaining a realistic esophageal atresia (Type C) prototype. Evaluation of model construction, usage and impact on user were performed by both a self-assessment questionnaire with pre- and post-training questions as well as observer-based variables and a revised Objective Structured Assessment of Technical Skills (OSATS) score.

RESULTS

A total of 10 participants were constructing and using the model at two different timepoints. OSATS score for overall performance was significantly higher (p = 0.005, z = -2.78) during the second observational period [median (MD): 4,95% confidence interval CI: 3.4, 5.1] compared to the first (MD: 3, 95% CI 2.4, 4.1). Self-reported boost in confidence after model usage for performing future esophageal atresia (EA) repair and bowel anastomosis (BA) in general was significantly higher (EA: U = 1, z = -2.3, p = 0.021, BA: U = 1, z = -2.41, p = 0.016) in participants with more years in training/attending status (EA MD:5, BA MD: 5.5) compared to less experienced participants (EA MD: 1.5, BA: 1).

CONCLUSION

Our easy access low-cost simulation model represents a feasible and tissue realistic training option to increase surgical performance of pediatric surgical trainees outside the OR.

Procedure-based assessment for laparoscopic cholecystectomy can replace global rating scales

INTRODUCTION

Global rating scales (GRSs) such as the Objective Structured Assessment of Technical Skills (OSATS) and Global Operative Assessment of Laparoscopic Surgery (GOALS) are assessment methods for surgical procedures. The aim of this study was to establish construct validity of Procedure-Based Assessment (PBA) and to compare PBA with GRSs for laparoscopic cholecystectomy.

MATERIAL AND METHODS

OSATS and GOALS GRSs were compared with PBA in their ability to discriminate between levels of performance between trainees who can perform the procedure independently and those who cannot. Three groups were formed based on the number of procedures performed by the trainee: novice (1-10), intermediate (11-20) and experienced (>20). Differences between groups were assessed using the Kruskal-Wallis and Mann-Whitney U tests.

RESULTS

Increasing experience correlated significantly with higher GRSs and PBA scores (all p < .001). Scores of novice and intermediate groups overlapped substantially on the OSATS (p = .1) and GOALS (p = .1), while the PBA discriminated between these groups (p = .03). The median score in the experienced group was higher with less dispersion for PBA (97.2[85.3-100]) compared to OSATS (82.1[60.7-100]) and GOALS (80[60-100]).

CONCLUSION

For assessing skill level or the capability of performing a laparoscopic cholecystectomy independently, PBA has a higher discriminative ability compared to the GRSs.

Development of a laparoscopic surgical skills simulation curriculum: Enhancing resident training through directed coaching and closed-loop feedback

BACKGROUND

Performance feedback through peer coaching and rigorous self-assessment is a critical part of technical skills improvement. However, formal collaborative programs using operative video-based skills assessments to generate peer coaching feedback have only been validated among attending surgeons. In this study, we developed a unique longitudinal, simulation video-based laparoscopic skills resident curriculum using video-based peer coaching and evaluated its association with skills acquisition among surgical trainees.

METHODS

The laparoscopic simulation curriculum consists of a pre-practice laparoscopic skill video recording, followed by receipt of directed coaching and feedback on performance from a faculty coach, a peer coach, and self-coaching. Residents then completed 6 weeks of feedback-directed practice and submitted a second post-practice laparoscopic skill video recording of the same skill, which was evaluated by a minimally invasive surgery expert grader. All general surgery residents in a single institution were enrolled, with 107 residents completing the curriculum in its initial 2 years.

RESULTS

Overall, more than two-thirds of residents achieved skills proficiency on their expert assessments, with similar rates of residents achieving skills proficiency at all postgraduate year levels. Significant improvements between the pre-practice assessments and post-practice assessments were most frequently seen in the instrument handling, precision, and motion & flow categories (P < .05 each). Faculty provided the highest number and proportion of closed-loop comments; residents' self-coaching feedback had the lowest number of closed-loop comments, with 83% of self-assessments containing none.

CONCLUSION

In this study, we describe the successful implementation of a longitudinal laparoscopic skills video-based coaching curriculum designed to improve residents' laparoscopic technical abilities through iterative directed practice supplemented by formative closed-loop feedback. This feasible, reproducible, and low-cost simulation curriculum can be adapted to other training programs and skills acquisition endeavors. This program also prepares trainees for ongoing performance feedback after completion of residency through rigorous self-assessment and peer-to-peer coaching.

Objective structured assessment of technical skill in temporal bone dissection: validation of a novel tool

OBJECTIVE

This study developed an assessment tool that was based on the objective structured assessment for technical skills principles, to be used for evaluation of surgical skills in cortical mastoidectomy. The objective structured assessment of technical skill is a well-established tool for evaluation of surgical ability. This study also aimed to identify the best material and printing method to make a three-dimensional printed temporal bone model.

METHODS

Twenty-four otolaryngologists in training were asked to perform a cortical mastoidectomy on a three-dimensional printed temporal bone (selective laser sintering resin). They were scored according to the objective structured assessment of technical skill in temporal bone dissection tool developed in this study and an already validated global rating scale.

RESULTS

Two external assessors scored the candidates, and it was concluded that the objective structured assessment of technical skill in temporal bone dissection tool demonstrated some main aspects of validity and reliability that can be used in training and performance evaluation of technical skills in mastoid surgery.

CONCLUSION

Apart from validating the new tool for temporal bone dissection training, the study showed that evolving three-dimensional printing technologies is of high value in simulation training with several advantages over traditional teaching methods.

Assessing Operative Competency in Cochlear Implantation Across the Residency Training Continuum

OBJECTIVE

To derive normative data based on postgraduate year (PGY) and number of completed otology rotations using a validated tool for cochlear implant surgical competency assessment.

STUDY DESIGN

Prospective, blinded validation study.

SETTING

Otolaryngology-Head and Neck Surgery residency training program at a tertiary academic medical center.

PATIENTS

Fourteen unique otolaryngology trainees, ranging from PGY-1 to PGY-6, were assessed a total of 26 times in a temporal bone laboratory setting using a validated assessment tool for cochlear implantation.

MAIN OUTCOME MEASURES

Construct validity, internal consistency, and inter-rater reliability.

RESULTS

Inter-rater reliability was calculated using the Cohen's kappa. Overall agreement between raters was excellent (κ = 0.82). Construct validity was supported by a positive association between higher task-based checklist and global rating scale scores with both trainee PGY level and number of otology rotations completed. Time to complete the exercise was inversely associated with PGY level and number of otology rotations completed.

CONCLUSION

Herein, we build upon a previously validated instrument to provide a method to reliably assess surgical competency in a core otologic procedure among residents across the training continuum. This data can be used to provide objective feedback on overall and task-specific competency in cochlear implantation to identify surgical deficiencies early on and to guide supportive course correction.

Design and evaluation of a simulated wound management course for postgraduate year one surgery residents

Background

It is vital to cover wound management knowledge and operations in the early stages of resident training. With this in mind, a simulated wound management course for postgraduate year one surgery residents (PGY1s) was designed and its effectiveness was evaluated.

Methods

A retrospective quasi-experimental method was used. PGY1s in 2014 constituted the control group, and PGY1s in 2015 and 2016 constituted the intervention group. The course given to the control group comprised didactic teaching followed by deliberate practice plus immediate personalized feedback. The newly designed course given to the intervention group was reconstructed and disassembled into four components according to the simulation-based mastery learning model, which were baseline test, interactive learning, basic skills practice, and reflective learning. The same performance assessments were used in the control and intervention group, including process measurement and outcome measurement.

Results

The process measurement showed that the intervention group’s scores were significantly higher in the “dissociation of subcutaneous tissue” and “quality of suturing and knots”. The outcome measurement showed that the accuracy of debridement was greatly improved and both key and total suture numbers were significantly higher in the intervention group.

Conclusions

Simulation-based mastery learning was incorporated into our proposed course framework, promoting the learning outcome of PGY1s. It has the potential to be adapted for other surgical training sites for residents in China.

Development, translation, and preliminary validation of the neck dissection assessment tool

BACKGROUND

The objective was to develop an assessment tool to evaluate residents' competency for neck dissection and provide preliminary evidence of feasibility, reliability, and validity.

METHODS

Six surgeons developed a neck dissection assessment tool using a modified Delphi method and evaluated 58 neck dissections from six junior and six senior otolaryngology residents.

RESULTS

The assessment tool uses a double checklist: a previously validated global rating scale (GRS) and a task-specific checklist (TSC). Use of the instrument appeared feasible and the average scores on the GRS and TSC differed significantly between junior and senior residents. The Pearson correlation coefficient between both checklists was 0.87. Intraclass correlation (ICC) for inter-rater reliability was 0.69 for the GRS, and 0.80 for the TSC.

CONCLUSION

This study provides preliminary evidence of feasibility, reliability, and validity for the first neck dissection assessment tool and provides a foundation for further psychometric analysis and research.

Automation of surgical skill assessment using a three-stage machine learning algorithm

Surgical skills are associated with clinical outcomes. To improve surgical skills and thereby reduce adverse outcomes, continuous surgical training and feedback is required. Currently, assessment of surgical skills is a manual and time-consuming process which is prone to subjective interpretation. This study aims to automate surgical skill assessment in laparoscopic cholecystectomy videos using machine learning algorithms. To address this, a three-stage machine learning method is proposed: first, a Convolutional Neural Network was trained to identify and localize surgical instruments. Second, motion features were extracted from the detected instrument localizations throughout time. Third, a linear regression model was trained based on the extracted motion features to predict surgical skills. This three-stage modeling approach achieved an accuracy of 87 ± 0.2% in distinguishing good versus poor surgical skill. While the technique cannot reliably quantify the degree of surgical skill yet it represents an important advance towards automation of surgical skill assessment.

Global versus task-specific postoperative feedback in surgical procedure learning

BACKGROUND

Task-specific checklists and global rating scales are both recommended assessment tools to provide constructive feedback on surgical performance. This study evaluated the most effective feedback tool by comparing the effects of the Observational Clinical Human Reliability Analysis (OCHRA) and the Objective Structured Assessment of Technical Skills (OSATS) on surgical performance in relation to the visual-spatial ability of the learners.

METHODS

In a randomized controlled trial, medical students were allocated to either the OCHRA (n = 25) or OSATS (n = 25) feedback group. Visual-spatial ability was measured by a Mental Rotation Test. Participants performed an open inguinal hernia repair procedure on a simulation model twice. Feedback was provided after the first procedure. Improvement in performance was evaluated blindly using a global rating scale (performance score) and hand-motion analysis (time and path length).

RESULTS

Mean improvement in performance score was not significantly different between the OCHRA and OSATS feedback groups (P = .100). However, mean improvement in time (371.0 ± 223.4 vs 274.6 ± 341.6; P = .027) and path length (53.5 ± 42.4 vs 34.7 ± 39.0; P = .046) was significantly greater in the OCHRA feedback group. When stratified by mental rotation test scores, the greater improvement in time (P = .032) and path length (P = .053) was observed only among individuals with low visual-spatial abilities.

CONCLUSION

A task-specific (OCHRA) feedback is more effective in improving surgical skills in terms of time and path length in novices compared to a global rating scale (OSATS). The effects of a task-specific feedback are present mostly in individuals with lower visual-spatial abilities.

Effect of a Spay Simulator on Student Competence and Anxiety

Spay simulation has gained attention at colleges of veterinary medicine that seek to utilize low-cost models in lieu of more cost-prohibitive high-fidelity devices or cadaveric specimens. A spay simulator was developed to provide veterinary students at the University of Florida College of Veterinary Medicine a reusable, inexpensive, and error-enabled device for self-practice in anticipation of a live canine ovariohysterectomy. Seventy-four students were recruited, half of whom participated in spay simulation training. A survey was designed to capture students' state and trait anxiety, as well as their self-assessed perceived levels of competence, confidence, and knowledge of anatomy, before and after their live animal surgery. During the live surgical laboratories, surgical competencies were assessed using the Objective Structured Assessment of Technical Skills (OSATS) for operative performance. We hypothesized that the spay simulation training group would have higher reported levels of competence, confidence, and knowledge of anatomy. Additionally, students enrolled in spay simulation training were expected to exhibit a lower level of post-operative anxiety and higher OSATS scores compared with the control group. Results demonstrated a significant increase in perceived anatomical knowledge and improvement in perceived competence level following spay simulation training as compared with the control group. Areas of no difference included perceived confidence, OSATS scores, and overall level of anxiety. The results of this study demonstrate that this low-fidelity spay simulator has a unique place in student surgical training, producing novice surgeons with increased perceived competence and knowledge of anatomy following spay simulation training and live animal surgery.

Postgraduate Assessment Approaches Across Surgical Specialties: A Systematic Review of the Published Evidence

PURPOSE

Ensuring competence for surgical trainees requires holistic assessment of the qualities and competencies necessary to practice safely and effectively. To determine the next steps toward achieving this aim, the authors conducted a systematic review to summarize and appraise the available evidence related to any assessment approach to postgraduate surgical training and to identify the dominant themes for assessment approaches across different specialties or countries.

METHOD

Medline and Embase were searched on January 10, 2019, without language or time restrictions. Any peer-reviewed study that described an assessment framework (in practice or novel) throughout postgraduate surgical training globally was included. An iterative review and thematic analysis were performed on full-text articles to determine assessment themes. Studies were then grouped by assessment themes. A tailored quality assessment of the studies included in the final analysis was conducted. Assessment themes and validity were compared across surgical specialties and countries.

RESULTS

From an initial 7,059 articles, 91 studies (evaluating 6,563 surgical trainees) were included in the final analysis. Ten defined assessment themes were extracted. Ten studies (11.0%) were deemed low risk of bias based on the quality assessment tool used and thus were determined to be high quality. Minor differences in assessment themes were observed between specialties and countries. Assessment themes neglected by individual surgical specialties and assessment themes that need validated assessment tools were identified.

CONCLUSIONS

This review highlights the low quality of evidence and fragmented efforts to develop and optimize surgical assessments. The minor differences observed demonstrate a common approach, globally and across specialties, related to surgical assessments. A paradigm shift in assessment approaches, which will require national and international collaboration, is required to optimize design and validation so that a comprehensive assessment of surgical competence can be implemented.

The initiation, standardization and proficiency (ISP) phases of the learning curve for minimally invasive liver resection: comparison of a fellowship-trained surgeon with the pioneers and early adopters

BACKGROUND

Using the Ideal Development Exploration Assessment and Long-term study (IDEAL) paradigm, Halls et al. created risk-adjusted cumulative sum (RA-CUSUM) curves concluding that Pioneers (P) and Early Adopters (EA) of minimally invasive (MI) liver resection obtained similar results after fewer cases. In this study, we applied this framework to a MI Hepatic-Pancreatic and Biliary fellowship-trained surgeon (FT) in order to assess where along the curves this generation fell.

METHODS

The term FT was used to designate surgeons without previous independent operative experience who went from surgical residency directly into fellowship. Three phases of the learning curve were defined using published data on EAs and Ps of MI Hepatectomy, including phase 1 (initiation) (i.e., the first 17 or 50), phase 2 (standardization) (i.e., cases 18-46 or 1-50) and phase 3 (proficiency) (i.e., cases after 46, 50 or 135). Data analysis was performed using the Social Science Statistics software ( www.socscistatistics.com ). Statistical significance was defined as p < .05.

RESULTS

From November 2007 until April 2018, 95 MI hepatectomies were performed by a FT. During phase 1, the FT approached larger tumors than the EA group (p = 0.002), that were more often malignant (94.1%) when compared to the P group (52.5%) (p < 0.001). During phase 2, the FT operated on larger tumors and more malignancies (93.1%) when compared to the Ps (p = 0.004 and p = 0.017, respectively). However, there was no difference when compared to the EA. In the phase 3, the EAs tended to perform more major hepatectomies (58.7) when compared to either the FT (30.6%) (p = 0.002) or the P's cases 51-135 and after 135 (35.3% and 44.3%, respectively) (both p values < 0.001). When compared to the Ps cases from 51-135, the FT operated on more malignancies (p = 0.012), but this was no longer the case after 135 cases by the Ps (p = 0.164). There were no statistically significant differences when conversions; major complications or 30- and 90-day mortality were compared among these 3 groups.

DISCUSSION

Using the IDEAL framework and RA-CUSUM curves, a FT surgeon was found to have curves similar to EAs despite having no previous independent experience operating on the liver. As in our study, FTs may tend to approach larger and more malignant tumors and do more concomitant procedures in patients with higher ASA classifications than either of their predecessors, without statistically significant increases in major morbidity or mortality.

CONCLUSION

It is possible that the ISP (i.e., initiation, standardization, proficiency) model could apply to other innovative surgical procedures, creating different learning curves depending on where along the IDEAL paradigm surgeons fall.

[The Maastricht Education System in Aachen - a Few Miles Away or Worlds Apart?]

INTRODUCTION

Education for residents in surgery varies not only throughout the world, but also throughout Europe. Our clinic is well connected to Maastricht University Medical in Centre in the Netherlands (European Surgical Centre Aachen Maastricht). On the other hand, there are clear differences in resident programs. In the Netherlands, a structured feedback according to the OSATS concept (Objective Structured Assessment of Technical Skills) is mandatory after every operation performed by residents. The aim of the present study was to transfer the OSATS concept from Maastricht to Aachen and to evaluate the feasibility and benefits of this concept for surgical education.

MATERIAL AND METHODS

The OSATS concept was implemented for 3 months in our clinic within a prospective clinical trial. Seven out of 10 residents that were working in our clinic at that time participated in the study (70%). Half of these were assigned to structured written feedback after every autonomously performed operation. Additionally, all participants performed structured written proper feedback according to the OSATS concept. The primary endpoint was the feasibility of the OSATS concept in our clinic; secondary endpoints were the benefits for the residents and the differences between external and self-evaluation.

RESULTS

The OSATS-concept was easily implemented in our clinic and met wide acceptance. Evaluation was performed after a mean of 70% of operations. External evaluation was regarded as more beneficial for residents than self-evaluation. Structured written evaluation according to the OSATS concept was not time-consuming (< 3 minutes) and most residents (86%) supported permanent implementation of the OSATS concept in our clinic.

CONCLUSION

The OSATS concept is a suitable approach to provide structured feedback to residents in continuous education. It can easily be implemented in resident education in Germany. Structured, written feedback by senior physicians is perceived as beneficial by residents.

Applying the Delphi process for development of a hepatopancreaticobiliary robotic surgery training curriculum

BACKGROUND

Robotic hepatopancreaticobiliary (HPB) procedures are performed worldwide and establishing processes for safe adoption of this technology is essential for patient benefit. We report results of the Delphi process to define and optimize robotic training procedures for HPB surgeons.

METHODS

In 2019, a robotic HPB surgery panel with an interest in surgical training from the Americas and Europe was created and met. An e-consensus-finding exercise using the Delphi process was applied and consensus was defined as 80% agreement on each question. Iterations of anonymous voting continued over three rounds.

RESULTS

Members agreed on several points: there was need for a standardized robotic training curriculum for HPB surgery that considers experience of surgeons and based on a robotic hepatectomy includes a common approach for "basic robotic skills" training (e-learning module, including hardware description, patient selection, port placement, docking, troubleshooting, fundamentals of robotic surgery, team training and efficiency, and emergencies) and an "advanced technical skills curriculum" (e-learning, including patient selection information, cognitive skills, and recommended operative equipment lists). A modular approach to index procedures should be used with video demonstrations, port placement for index procedure, troubleshooting, and emergency scenario management information. Inexperienced surgeons should undergo training in basic robotic skills and console proficiency, transitioning to full procedure training of e-learning (video demonstration, simulation training, case observation, and final evaluation). Experienced surgeons should undergo basic training when using a new system (e-learning, dry lab, and operating room (OR) team training, virtual reality modules, and wet lab; case observations were unnecessary for basic training) and should complete the advanced index procedural robotic curriculum with assessment by wet lab, case observation, and OR team training.

CONCLUSIONS

Optimization and standardization of training and education of HPB surgeons in robotic procedures was agreed upon. Results are being incorporated into future curriculum for education in robotic surgery.

Radiation Safety Performance is More than Simply Measuring Doses! Development of a Radiation Safety Rating Scale

PURPOSE

Radiation safety performance is often evaluated using dose parameters measured by personal dosimeters and/or the C-arm, which provide limited information about teams' actual radiation safety behaviors. This study aimed to develop a rating scale to evaluate team radiation safety behaviors more accurately and investigate its reliability.

MATERIALS AND METHODS

A modified Delphi consensus was organized involving European vascular surgeons (VS), interventional radiologists, and interventional cardiologists. Initial items and anchors were drafted a priori and rated using five-point Likert scales. Participants could suggest additional items or adjustments. Consensus was defined as ≥ 80% agreement (rating ≥ 4) with Cronbach's alpha ≥ .80. Two VS with expertise in radiation safety evaluated 15 video-recorded endovascular repairs of infrarenal aortic aneurysms (EVAR) to assess usability, inter and intrarater reliability.

RESULTS

Thirty-one of 46 invited specialists completed three rating rounds to generate the final rating scale. Five items underwent major adjustments. In the final round, consensus was achieved for all items (alpha = .804; agreement > 87%): 'Pre-procedural planning', 'Preparation in angiosuite/operating room', 'Shielding equipment', 'Personal protective equipment', 'Position of operator/team', 'Radiation usage awareness', 'C-arm handling', 'Adjusting image quality', 'Additional dose reducing functions', 'Communication/leadership', and 'Overall radiation performance and ALARA principle'. All EVARs were rated, yielding excellent Cronbach's alpha (.877) with acceptable interrater and excellent intrarater reliability (ICC = .782; ICC = .963, respectively).

CONCLUSION

A reliable framework was developed to assess radiation safety behaviors in endovascular practice and provide teams with formative feedback. The final scale is provided in this publication.

The impact of surgical simulation on patient outcomes: a systematic review and meta-analysis

The use of simulation in surgical training is ever growing. Evidence suggests such training may have beneficial clinically relevant effects. The objective of this research is to investigate the effects of surgical simulation training on clinically relevant patient outcomes by evaluating randomized controlled trials (RCT). PubMed was searched using PRISMA guidelines: "surgery" [All Fields] AND "simulation" [All Fields] AND "patient outcome" [All Fields]. Of 119 papers identified, 100 were excluded for various reasons. Meta-analyses were conducted using the inverse-variance random-effects method. Nineteen papers were reviewed using the CASP RCT Checklist. Sixteen studies looked at surgical training, two studies assessed patient-specific simulator practice, and one paper focused on warming-up on a simulator before performing surgery. Median study population size was 22 (range 3-73). Most articles reported outcome measures such as post-intervention Global Rating Scale (GRS) score and/or operative time. On average, the intervention group scored 0.42 (95% confidence interval 0.12 to 0.71, P = 0.005) points higher on a standardized GRS scale of 1-10. On average, the intervention group was 44% (1% to 87%, P = 0.04) faster than the control group. Four papers assessed the impact of simulation training on patient outcomes, with only one finding a significant effect. We found a significant effect of simulation training on operative performance as assessed by GRS, albeit a small one, as well as a significant reduction to operative time. However, there is to date scant evidence from RCTs to suggest a significant effect of surgical simulation training on patient outcomes.

Deliberate Practice: Applying the Expert Performance Approach to Gynecologic Surgical Training

Surgical training is shifting from the historical Halstedian apprenticeship model to outcomes-based methods. Surgical residents can reach a higher level of performance when utilizing deliberate practice and the expert performance approach. This article discusses methods for implementing deliberate practice and the expert performance approach into gynecologic surgical training programs.

Development of an Assessment Tool for Surgeons in Their First Year of Independent Practice: The Junior Surgeon Performance Assessment Tool

OBJECTIVE

The purpose of this study was to create an assessment tool to evaluate newly practicing surgeons.

DESIGN

In this prospective mixed methods study, a needs assessment was performed by conducting focus groups with practicing general surgeons, asking questions regarding essential surgeon qualities, behaviors observed in inexperienced surgeons, current assessment methods, and desired assessment tool elements and attributes. A qualitative analysis was performed using a grounded theory methodology. The Junior Surgeon Performance Assessment Tool (JSPAT) was created using a 4-point scale for each category developed, with themes identified in the qualitative analysis used to create behavioral anchors. The JSPAT was evaluated by focus group participants and by members of the American College of Surgeons Advisory Council for Rural Surgery using an online survey.

SETTING

Rural and nonuniversity-based hospitals throughout the state of Oregon.

PARTICIPANTS

Practicing general surgeons.

RESULTS

Focus groups consisted of 31 surgeons (mean age 49, mean experience 17 years) from 11 different hospitals. Qualitative analysis revealed 91 different themes, which were grouped into 5 domains (technical skills, interaction with patients, interaction with surgeon colleagues, interactions with the greater medical community, and self-care) to create the assessment tool. Twenty online survey responses providing feedback on the assessment tool were obtained, with 75% rating the JSPAT useful or very useful and 69% satisfied or very satisfied with the time to complete the tool.

CONCLUSIONS

A mixed-methods model was used to create an assessment tool for surgeons in their first year of independent practice. Survey data demonstrated that practicing surgeons find value in the JSPAT.

The Role of Manual Dexterity and Cognitive Functioning in Enhancing Resident Operative Autonomy

INTRODUCTION

Autonomy, both operative and nonoperative, is one of the most critical aspects of successful surgical training. Both surgeon and resident share the responsibility of achieving this goal. We hypothesize that operative autonomy is distinct and depends, for the most part, on the resident's manual dexterity, knowledge of, and preparation for the procedure.

METHODS

Over a period of 4 academic years, between July 2014 and June 2018, a total of 958 Global Rating Scale of Operative Performance evaluations were completed by 32 general and subspecialty faculty surgeons for 35 residents. Elective procedures were evaluated, including 165 (17.2%) by postgraduate year (PGY)1 residents, 253 (26.4%) by PGY2, 199 (20.8%) by PGY3, 147 (15.3%) by PGY4, and 194 (20.3%) by PGY5. The procedures evaluated were: 261 (27.2%) hernia repairs; 178 (18.6%) cholecystectomies; 102 (10.6%) colorectal and anal procedures; 73 (7.6%) vascular procedures; 56 (5.8%) thyroid and parathyroidectomies; 39 (4.1%) foregut (esophagus and stomach) procedures; 38 (4%) skin, soft tissue, and breast; 92 (10%) hepatopancreatic; 20 (2.1%) pediatric procedures; and 99 (10.3%) other procedures including amputations, cardiothoracic, and solid organs procedures. Each resident was scored from 1 to 5 (1 lowest, 5 highest) in each of the following categories of Global Rating Scale of Operative Performance: respect for tissue (RT), time and motion (T&M), instrument handling (IH), knowledge of the instrument (KI), flow of operation (FO) and resident's preparation for the procedure (RP). Resident operative autonomy (ROA) was assessed using the Zwisch scale, a 4-point scale describing faculty supervision behaviors associated with different degrees of resident autonomy (1: Show and Tell, 2: Active Help, 3: Passive Help, and 4: Supervision Only).

RESULTS

Correlation and ordinal regression analyses were conducted to examine the relationship between ROA and manual dexterity (RT, T&M, IH, and FO), and cognitive functioning (knowledge of instruments and resident preparation). Results indicated a positive correlation between ROA and RT (r = 0.528, p < 0.001), T&M (r = 0.630, p < 0.001), IH (r = 0.597, p < 0.001), KI (r = 0.490, p < 0.001), FO (r = 0.637, p < 0.001), and RP (r = 0.525, p < 0.001). Additionally, there was a weak inverse correlation between ROA and the number of years the surgeon had been in practice (r = -0.127, p = 0.001). The significant predictors of resident autonomy found by the ordinal logistic regression include time and motion (p < 0.001), flow of operation (p < 0.001), and resident's preparation for the procedure (p < 0.001).

CONCLUSIONS

Resident operative autonomy is a product of shared responsibility between the faculty and resident. However, residents' inherent and/or acquired skills and preparation for the operative procedures play a critical role. Residents should be advised to use available resources such as simulation to augment their skills preoperatively and to enhance their autonomy in the operating room.

Simulation as tool for evaluating and improving technical skills in laparoscopic gynecological surgery

BACKGROUND

Simulation in laparoscopic surgery is nowadays recognized as a valid instrument for learning and training surgeons in different procedures. However, its role as evaluation test and self-assessment tool to verify basic surgical skills is still under discussion.

METHODS

Thirty-three residents in obstetrics and gynecology at University of Pisa, Italy were recruited, and they received a simulation program consisting of 5 tasks. They had to perform basic laparoscopic surgery maneuvers as creating pneumoperitoneum, positioning trocars under vision, demonstrating the appropriate use of dominant and non-dominant hand and making single stitch and knot. They were evaluated with a modified OSATs scale.

RESULTS

Senior trainees had better score than junior trainees (p value< 0,005) and after different sessions of simulation scores of both groups significantly improved (p < 0,001), especially for the junior group. All the trainees reported self-assessments that matched with the evaluation of external observers demonstrating the importance of simulation also as auto-evaluation test.

CONCLUSIONS

In this study, we demonstrated the role of simulation as powerful tool to evaluate and to self-assess surgical technical skills and to improve own capacities, with the use of a modified OSATs scale adapted to specific exercises.

Recruiting expertise: how surgical trainees engage supervisors for learning in the operating room

CONTEXT

For centuries now, the operating room (OR) has been the environment in which surgical trainees come to master procedures. Restricted working hours and insufficient levels of autonomy at the end of their training necessitate a shift towards alternative effective learning strategies. Self-regulated learning is a promising strategy by which surgical trainees can learn more with fewer exposures. However, the challenge is to understand how surgical trainees regulate their learning in the clinical context of the OR.

OBJECTIVES

The purpose of this study is to identify and understand the strategies of surgical trainees in engaging their supervisors for learning purposes and how these strategies contribute to effective learning.

METHODS

Total hip replacement procedures performed by four surgical trainees and their supervisors were videotaped. Using the iterative inductive process of conversation analysis, each verbal initiative to engage the supervisor was identified, analysed ('why that now') and categorised.

RESULTS

Surgical trainees used a range of practices to engage supervisors and recruit expertise, ranging from explicit recruitment to implicit hints. We identified four major categories. Surgical trainees: (i) invite the supervisor to provide an evaluation of the ongoing task; (ii) express an evaluation of the ongoing task and then explicitly invite the supervisor to provide an evaluation; (iii) express an evaluation of the ongoing task and then invite the supervisor to provide confirmation, and (iv) express an evaluation of the ongoing task without engaging the supervisor.

CONCLUSIONS

Surgical trainees recruit expertise from supervisors using practices of four different categories. Trainees' actions are provoked by the moment at which they experience insufficient expertise and are focused on the task at hand in the immediate present. Supervisors can and do elaborate on these requests to provide explicit teaching. Insight into these practices provides tools for reflection on OR learning, proficiency assessment and deliberation to adapt guidance in the real time of the procedure.

Evaluation tool for a gastroenterostomy simulated training1

PURPOSE

To create a checklist to evaluate the performance and systematize the gastroenterostomy simulated training.

METHODS

Experimental longitudinal study of a quantitative character. The sample consisted of twelve general surgery residents. The training was divided into 5 sessions and consisted of participation in 20 gastroenterostomys in synthetic organs. The training was accompanied by an experienced surgeon who was responsible for the feedback and the anastomoses evaluation. The anastomoses evaluated were the first, fourth, sixth, eighth and tenth. A 10 item checklist and the time to evaluate performance were used.

RESULTS

Residents showed a reduction in operative time and evolution in the surgical technique statistically significant (p<0.01). The correlation index of 0.545 and 0,295 showed a high linear correlation between time variables and Checklist. The average Checklist score went from 6.8 to 9 points.

CONCLUSION

The proposed checklist can be used to evaluate the performance and systematization of a simulated training aimed at configuring a gastroenterostomy.

Validation of the Omni: A Novel, Multimodality, and Longitudinal Surgical Skills Assessment

OBJECTIVE

The breadth of technical skills included in general surgery training continues to expand. The current competency-based training model requires assessment tools to measure acquisition, learning, and mastery of technical skill longitudinally in a reliable and valid manner. This study describes a novel skills assessment tool, the Omni, which evaluates performance in a broad range of skills over time.

DESIGN

The 5 Omni tasks, consisting of open bowel anastomosis, knot tying, laparoscopic clover pattern cut, robotic needle drive, and endoscopic bubble pop, were developed by general surgery faculty. Component performance metrics assessed speed, accuracy, and quality, which were scaled into an overall score ranging from 0 to 10 for each task. For each task, ANOVAs with Scheffé's post hoc comparisons and Pearson's chi-squared tests compared performance between 6 resident cohorts (clinical years (CY1-5) and research fellows (RF)). Paired samples t-tests evaluated changes in performance across academic years. Cronbach's alpha coefficient determined the internal consistency of the Omni as an overall assessment.

SETTING

The Omni was developed by the Department of Surgery at Duke University. Annual assessment and this research study took place in the Surgical Education and Activities Lab.

PARTICIPANTS

All active general surgery residents in 2 consecutive academic years spanning 2015 to 2017.

RESULTS

A total of 62 general surgery residents completed the Omni and 39 (67.2%) of those residents completed the assessment in 2 consecutive years. Based on data from all residents' first assessment, statistically significant differences (p < 0.05) were observed among CY cohorts for bowel anastomosis, robotic, and laparoscopic task metrics. By pair-wise comparisons, mean bowel anastomosis scores distinguished CY1 from CY3-5 and CY2 from CY5. Mean robotic scores distinguished CY1 from RF, and mean laparoscopic scores distinguished CY1 from RF, CY3, and CY5 in addition to CY2 from CY3. Mean scores in performance on the knot tying and endoscopic tasks were not significantly different. Statistically significant improvement in mean scores was observed for all tasks from year 1 to year 2 (all p < 0.02). The internal consistency analysis revealed an alpha coefficient of 0.656.

CONCLUSIONS

The Omni is a novel composite assessment tool for surgical technical skill that utilizes objective measures and scoring algorithms to evaluate performance. In this pilot study, 3 tasks demonstrated discriminative ability of performance by CY, and all 5 tasks demonstrated construct validity by showing longitudinal improvement in performance. Additionally, the Omni has adequate internal consistency for a formative assessment. These results suggest the Omni holds promise for the evaluation of resident technical skill and early identification of outliers requiring intervention.

Evaluating competency in video-assisted thoracoscopic surgery (VATS) lobectomy performance using a novel assessment tool and virtual reality simulation

BACKGROUND

Competency-based training has gained ground in surgical training and with it assessment tools to ensure that training objectives are met. Very few assessment tools are available for evaluating performance in thoracoscopic procedures. Video recordings would provide the possibility of blinded assessment and limited rater bias. This study aimed to provide validity evidence for a newly developed and dedicated tool for assessing competency in Video-Assisted Thoracoscopic Surgery (VATS) lobectomy.

METHODS

Participants with varying experience with VATS lobectomy were included from different countries. Video recordings from participants' performance of a VATS right upper lobe lobectomy on a virtual reality simulator were rated by three raters using a modified version of a newly developed VATS lobectomy assessment tool (the VATSAT) and analyzed in relation to the unitary framework (content, response process, internal structure, relation to other variables, and consequences of testing).

RESULTS

Fifty-three participants performed two consecutive simulated VATS lobectomies on the virtual reality simulator, leaving a total of 106 videos. Content established in previously published studies. Response process Standardized data collection was ensured by using an instructional element, uniform data collection, a special rating program, and automatic generation of the results to a database. Raters were carefully instructed in using the VATSAT, and tryout ratings were carried out. Internal structure Inter-rater reliability was calculated as intra-class correlation coefficients, to 0.91 for average measures (p < 0.001). Test/re-test reliability was calculated as Pearson's r of 0.70 (p < 0.001). G-coefficient was calculated to be 0.79 with two procedures and three raters. By performing D-theory was found that either three procedures rated by two raters or five procedures rated by one rater were enough to reach an acceptable G-coefficient of ≥ 0.8. Relation to other variables Significant differences between groups were found (p < 0.001). The participants' VATS lobectomy experience correlated significantly to their VATSAT score (p = 0.016). Consequences of testing The pass/fail score was found to be 14.9 points by the contrasting groups' method, leaving five false positive (29%) and six false negatives (43%).

CONCLUSION

Validity evidence was provided for the VATSAT according to the unitary framework. The VATSAT provides supervisors and assessors with a procedure-specific assessment tool for evaluating VATS lobectomy performance and aids with the decision of when the trainee is ready for unsupervised performance.

The Effectiveness of Bioskills Training for Simulated Lumbar Pedicle Screw Placement

STUDY DESIGN

Prospective randomized study.

OBJECTIVES

To define the impact of an inexpensive, user-friendly, and reproducible lumbar pedicle screw instrumentation bioskills training module and evaluation protocol.

METHODS

Participants were randomized to control (n = 9) or intervention (n = 10) groups controlling for level of experience (medical students, junior resident, or senior resident). The intervention group underwent a 20-minute bioskills training module while the control group spent the same time with self-directed study. Pre- and posttest performance was self-reported (Physician Performance Diagnostic Inventory Scale [PPDIS]). Objective outcome scores were obtained from a blinded fellowship-trained attending orthopedic spine surgeon using Objective Structured Assessment of Technical Skills (OSATS) and Objective Pedicle Instrumentation Score metrics. In addition, identification of pedicle breach and breach anatomic location was measured pre- and posttest in lumbar spine models.

RESULTS

The intervention group showed a 30.8% improvement in PPDIS scores, compared with 13.4% for the control group (P = .01). The intervention group demonstrated statistically significant 66% decrease in breaches (P = .001) compared with 28% decrease in the control group (P = .06). Breach identification demonstrated no change in accuracy of the control group (incorrect identification from 32.2% pre- to posttest 35%; P = .71), whereas the intervention group's improvement was statistically significant (42% pre- to posttest 36.5%; P = .0047).

CONCLUSIONS

We conclude that a concise lumbar pedicle screw instrumentation bioskills training session can be a useful educational tool to augment clinical education.

The impact of low-dose glucocorticoids on disease activity, bone mineral density, fragility fractures, and 10-year probability of fractures in patients with rheumatoid arthritis

This study aimed to investigate the effect of low-dose glucocorticoids (LDGs) on disease activity, bone density, and fractures in patients with rheumatoid arthritis (RA). This was an interim analysis of the RA Registry. Demographic data and clinical characteristics, including fracture risk assessment tool, were collected. 25(OH) Vitamin D, bone mineral density (BMD), and intact parathyroid hormone were measured at enrollment. The study group were those who took LDGs (2.5–7.5 mg/day prednisolone or equivalent dose), and the others were included as the control group. A total of 425 participants were enrolled, including 85 (20%) in the control group and 340 (80%) in the study group. The demographics and clinical characteristics were comparable between the two groups. Compared with the control group, the LDGs group had a significantly lower vertebral BMD (L 1–4) (g/cm²), (0.854 vs 0.896, p=0.046), significantly higher rate of previous fractures (103 (30.3%) vs 13 (15.3%), p=0.006), higher 10-year probability of major fractures (14 (15.5) vs 8 (8.6), p<0.0001), and higher 10-year probability of hip fractures (4.4 (8.4) vs 2 (3.9), p<0.0001). Disease activity appeared to be similar in the patients with RA regardless of whether or not they received LDG treatment. However, the patients with RA who received LDG treatment had a lower BMD at the spine (L1–4) and a higher rate of previous fractures that was associated with a significantly higher 10-year probability of fractures than those who did not receive LDG treatment.

Assessment of Surgical Skills and Competency

Evaluation of surgical skills and competency are important aspects of the medical education process. Measurable and reproducible methods of assessment with objective feedback are essential components of surgical training. Objective Structured Assessment of Technical Skills (OSATS) is widely used across the medical specialties and otolaryngology-specific tools have been developed and validated for sinus and mastoid surgery. Although assessment of surgical skills can be time-consuming and requires human and financial resources, new evaluation methods and emerging technology may alleviate these barriers while also improving data collection practices.

Assessment of Technical Skills Competence in the Operating Room: A Systematic and Scoping Review

PURPOSE

While academic accreditation bodies continue to promote competency-based medical education (CBME), the feasibility of conducting regular CBME assessments remains challenging. The purpose of this study was to identify evidence pertaining to the practical application of assessments that aim to measure technical competence for surgical trainees in a nonsimulated, operative setting.

METHOD

In August 2016, the authors systematically searched Medline, Embase, and the Cochrane Database of Systematic Reviews for English-language, peer-reviewed articles published in or after 1996. The title, abstract, and full text of identified articles were screened. Data regarding study characteristics, psychometric and measurement properties, implementation of assessment, competency definitions, and faculty training were extracted. The findings from the systematic review were supplemented by a scoping review to identify key strategies related to faculty uptake and implementation of CBME assessments.

RESULTS

A total of 32 studies were included. The majority of studies reported reasonable scores of interrater reliability and internal consistency. Seven articles identified minimum scores required to establish competence. Twenty-five articles mentioned faculty training. Many of the faculty training interventions focused on timely completion of assessments or scale calibration.

CONCLUSIONS

There are a number of diverse tools used to assess competence for intraoperative technical skills and a lack of consensus regarding the definition of technical competence within and across surgical specialties. Further work is required to identify when and how often trainees should be assessed and to identify strategies to train faculty to ensure timely and accurate assessment.

Developing and validating a simple and cost-effective strabismus surgery simulator

PURPOSE

To demonstrate that a nonbiologic strabismus surgery simulator is not inferior to a biologic wet lab for teaching the key steps of strabismus surgery.

METHODS

A total of 41 medical students were randomly assigned to one of two groups: biologic wet lab or nonbiologic simulator. The students trained according to the group's protocol then participated in a recorded final assessment using a realistic strabismus surgery model. Two independent reviewers, masked to training method, graded the video recordings using three scoring systems: the International Council of Ophthalmology Approved-Ophthalmology Surgical Competency Assessment Rubric for Strabismus Surgery (ICO-OSCAR), the Global Rating Scale of Objective Structured Assessment of Technical Skills (OSATS), and the Alphabetic Summary Scale (ASS).

RESULTS

The primary endpoint, total ICO-OSCAR score, was 36.7 ± 2.2 for the wet lab group and 36.0 ± 2.7 for the nonbiologic group (difference in means, -0.7; one-sided 95% CI, -2.0, ∞). The lower bound of the one-sided 95% confidence interval for the difference in mean scores was -2.0, which was greater than the a priori noninferiority margin of -5.0 points. The secondary outcome measure, mean total OSATS score and ASS score, revealed no statistical significant differences between training methods (P = 0.73 and P = 0.44, resp.).

CONCLUSIONS

The simple, nonbiologic strabismus surgery simulator is not inferior to the biologic wet lab with respect to total ICO-OSCAR score. It is a portable, inexpensive, and effective training tool for novice surgeons.

A novel assessment tool for evaluating competence in video-assisted thoracoscopic surgery lobectomy

BACKGROUND

Specific assessment tools can accelerate trainees' learning through structured feedback and ensure that trainees attain the knowledge and skills required to practice as competent, independent surgeons (competency-based surgical education). The objective was to develop an assessment tool for video-assisted thoracoscopic surgery (VATS) lobectomy by achieving consensus within an international group of VATS experts.

METHOD

The Delphi method was used as a structured process for collecting and distilling knowledge from a group of internationally recognized VATS experts. Opinions were obtained in an iterative process involving answering repeated rounds of questionnaires. Responses to one round were summarized and integrated into the next round of questionnaires until consensus was reached.

RESULTS

Thirty-one VATS experts were included and four Delphi rounds were conducted. The response rate for each round were 68.9% (31/45), 100% (31/31), 96.8% (30/31), and 93.3% (28/30) for the final round where consensus was reached. The first Delphi round contained 44 items and the final VATS lobectomy Assessment Tool (VATSAT) comprised eight items with rating anchors: (1) localization of tumor and other pathological tissue, (2) dissection of the hilum and veins, (3) dissection of the arteries, (4) dissection of the bronchus, (5) dissection of lymph nodes, (6) retrieval of lobe in bag, (7) respect for tissue and structures, and (8) technical skills in general.

CONCLUSION

A novel and dedicated assessment tool for VATS lobectomy was developed based on VATS experts' consensus. The VATSAT can support the learning of VATS lobectomy by providing structured feedback and help supervisors make the important decision of when trainees have acquired VATS lobectomy competencies for independent performance.

Developing an objective assessment of surgical performance from operating room video and surgical imagery

An unbiased, repeatable process for assessing operating room performance is an important step toward quantifying the relationship between surgical training and performance. Hip fracture surgeries offer a promising first target in orthopedic trauma because they are common and they offer quantitative performance metrics that can be assessed from video recordings and intraoperative fluoroscopic images. Hip fracture repair surgeries were recorded using a head-mounted point-of-view camera. Intraoperative fluoroscopic images were also saved. The following performance metrics were analyzed: duration of wire navigation, number of fluoroscopic images collected, degree of intervention by the surgeon's supervisor, and the tip-apex distance (TAD). Two orthopedic traumatologists graded surgical performance in each video independently using an Objective Structured Assessment of Technical Skill (OSATS). Wire navigation duration correlated with weeks into residency and prior cases logged. TAD correlated with cases logged. There was no significant correlation between the OSATS total score and experience metrics. Total OSATS score correlated with duration and number of fluoroscopic images. Our results indicate that two metrics of hip fracture wire navigation performance, duration and TAD, significantly differentiate surgical experience. The methods presented have the potential to provide truly objective assessment of resident technical performance in the OR.

The Effectiveness of Bioskills Training for Simulated Open Lumbar Laminectomy

STUDY DESIGN

Randomized, prospective study within an orthopedic surgery resident program at a large urban academic medical center.

OBJECTIVES

To develop an inexpensive, user-friendly, and reproducible lumbar laminectomy bioskills training module and evaluation protocol that can be readily implemented into residency training programs to augment the clinical education of orthopedic and neurosurgical physicians-in-training.

METHODS

Twenty participants comprising senior medical students and orthopedic surgical residents. Participants were randomized to control (n = 9) or intervention (n = 11) groups controlling for level of experience (medical students, junior resident, or senior resident). The intervention group underwent a 40-minute bioskills training module, while the control group spent the same time with self-directed study. Pre- and posttest performance was self-reported by each participant (Physician Performance Diagnostic Inventory Scale [PPDIS]). Objective outcome scores were obtained from a blinded fellowship-trained attending orthopedic spine surgeon using Objective Structured Assessment of Technical Skills (OSATS) and Objective Decompression Score metrics.

RESULTS

When compared with the control group, the intervention group yielded a significant mean improvement in OSATS (P = .022) and PPDIS (P = .0001) scores. The Objective Decompression Scores improved in the intervention group with a trend toward significance (P = .058).

CONCLUSIONS

We conclude that a concise lumbar laminectomy bioskills training session can be a useful educational tool for to augment clinical education. Although no direct clinical correlation can be concluded from this study, the improvement in trainee's technical and procedural skills suggests that Sawbones training modules can be an efficient and effective tool for teaching fundamental spine surgical skills outside of the operating room.