The effectiveness of video feedback in the acquisition of orthopedic technical skills

Development and preliminary validation of a new task-based objective procedure-specific assessment of inguinal hernia repair procedural safety

BACKGROUND

Surgical videos coupled with structured assessments enable surgical training programs to provide independent competency evaluations and align with the American Board of Surgery's entrustable professional activities initiative. Existing assessment instruments for minimally invasive inguinal hernia repair (IHR) have limitations with regards to reliability, validity, and usability. A cross-sectional study of six surgeons using a novel objective, procedure-specific, 8-item competency assessment for minimally invasive inguinal hernia repair (IHR-OPSA) was performed to assess inter-rater reliability using a "safe" vs. "unsafe" scoring rubric.

METHODS

The IHR-OPSA was developed by three expert IHR surgeons, field tested with five IHR surgeons, and revised based upon feedback. The final instrument included: (1) incision/port placement; (2) dissection of peritoneal flap (TAPP) or dissection of peritoneal flap (TEP); (3) exposure; (4) reducing the sac; (5) full dissection of the myopectineal orifice; (6) mesh insertion; (7) mesh fixation; and (8) operation flow. The IHR-OPSA was applied by six expert IHR surgeons to 20 IHR surgical videos selected to include a spectrum of hernia procedures (15 laparoscopic, 5 robotic), anatomy (14 indirect, 5 direct, 1 femoral), and Global Case Difficulty (easy, average, hard). Inter-rater reliability was assessed against Gwet's AC2.

RESULTS

The IHR-OPSA inter-rater reliability was good to excellent, ranging from 0.65 to 0.97 across the eight items. Assessments of robotic procedures had higher reliability with near perfect agreement for 7 of 8 items. In general, assessments of easier cases had higher levels of agreement than harder cases.

CONCLUSIONS

A novel 8-item minimally invasive IHR assessment tool was developed and tested for inter-rater reliability using a "safe" vs. "unsafe" rating system with promising results. To promote instrument validity the IHR-OPSA was designed and evaluated within the context of intended use with iterative engagement with experts and testing of constructs against real-world operative videos.

Comparison of self versus expert-assisted feedback for cricothyroidotomy training: a randomized trial

BACKGROUND

The self-video feedback method may have the potential to provide a low-cost alternative to physician-driven simulation-based training. This study aimed to assess the utility of two video feedback methods by comparing the improvement in performing cricothyroidotomy procedure following self video feedback (trainees review their performance by themselves) and expert-assisted video feedback (trainees review their performance while an emergency physician provides additional feedback).

METHODS

This study was pretest-posttest and two-group designed research performed at a university simulation center with 89 final-year medical students and used a cricothyroidotomy simulation model. After seeing an educational presentation and a best practice video, trainees were randomized into two groups; self video feedback group (SVFG) and expert-assisted video feedback group (EVFG). They performed the cricothyroidotomy before and after the feedback. The procedures were also recorded and scored by two emergency physicians.

RESULTS

There was a statistically significant improvement between pre-feedback and post-feedback assessments in terms of scores received and time needed for the procedures in both SVFG and EVFG groups (p < 0.05). Additionally, the post-feedback assessment scores were higher and time needed for the procedure was lower in the EVFG when compared with SVFG (p < 0.05 for both).

CONCLUSIONS

Results demonstrated significant improvement in cricothyroidotomy performance with both types of video feedback method. Even though the improvement was better in the EVFG compared to the SVFG, the self video feedback may have value especially in situations where expert-assisted feedback is not possible.

Video Coaching in Surgical Education: Utility, Opportunities, and Barriers to Implementation

OBJECTIVE

This review discusses the literature on Video-Based Coaching (VBC) and explores the barriers to widespread implementation.

DESIGN

A search was performed on Scopus and PubMed for the terms "operation," "operating room," "surgery," "resident," "house staff," "graduate medical education," "teaching," "coaching," "assessment," "reflection," "camera," and "video" on July 27, 2021, in English. This yielded 828 results. A single author reviewed the titles and abstracts and eliminated any results that did not pertain to operative VBC or assessment. All bibliographies were reviewed, and appropriate manuscripts were included in this study. This resulted in a total of 52 manuscripts included in this review.

SETTING/PARTICIPANTS

Original, peer-reviewed studies focused on VBC or assessment.

RESULTS

VBC has been both subjectively and objectively found to be a valuable educational tool. Nearly every study of video recording in the operating room found that subjects, including surgical residents and seasoned surgeons alike, overwhelmingly considered it a useful, non-redundant adjunct to their training. Most studies that evaluated skill acquisition via standardized assessment tools found that surgical residents who underwent a VBC program had significant improvements compared to their counterparts who did not undergo video review. Despite this evidence of effectiveness, fewer than 5% of residency programs employ video recording in the operating room. Barriers to implementation include significant time commitments for proposed coaching curricula and difficulty with integration of video cameras into the operating room.

CONCLUSIONS

VBC has significant educational benefits, but a scalable curriculum has not been developed. An optimal solution would ensure technical ease and expediency, simple, high-quality cameras, immediate review, and overcoming entrenched surgical norms and culture.

Correlation of Colorectal Surgical Skill With Patient Outcomes: A Cautionary Tale

BACKGROUND

Previous work has demonstrated a correlation between video ratings of surgical skill and clinical outcomes. Some have proposed the use of video review for technical skill assessment, credentialing, and quality improvement.

OBJECTIVE

Before its adoption as a quality measure for colorectal surgeons, we must first determine whether video-based skill assessments can predict patient outcomes among specialty surgeons.

DESIGN

Twenty-one surgeons submitted one representative video of a minimally invasive colectomy. Each video was edited to highlight key steps and then rated by 10 peer surgeons using a validated American Society of Colon and Rectal Surgeons assessment tool. Linking surgeons' ratings to a validated surgical outcomes registry, we assessed the relationship between skill and risk-adjusted complication rates.

SETTINGS

The study was conducted with the Michigan Surgical Quality Collaborative, a statewide collaborative including 70 community, academic, and tertiary hospitals.

PATIENTS

Patients included those who underwent minimally invasive colorectal resection performed by the participating surgeons.

MAIN OUTCOME MEASURES

Main outcome measures included 30-day risk-adjusted postoperative complications.

RESULTS

The average technical skill rating for each surgeon ranged from 2.6 to 4.6. Risk-adjusted complication rate per surgeon ranged from 9.9% to 33.1%. Patients of surgeons in the bottom quartile of overall skill ratings were older and more likely to have hypertension or to smoke; patients of surgeons in the top quartile were more likely to be immunosuppressed or have an ASA score of 3 or higher. After patient- and surgery-specific risk adjustment, there was no statistically significant difference in complication rates between the bottom and top quartile surgeons (17.5% vs 16.8%, respectively, p = 0.41).

LIMITATIONS

Limitations included retrospective cohort design with short-term follow-up of sampled cases. Videos were edited to highlight key steps, and reviewers did not undergo training to establish norms.

CONCLUSIONS

Our study demonstrates that video-based peer rating of minimally invasive colectomy was not correlated with postoperative complications among specialty surgeons. As such, the adoption of video review for use in credentialing should be approached with caution. See Video Abstract at http://links.lww.com/DCR/B802.CORRELACIÓN ENTRE LA HABILIDAD QUIRÚRGICA COLORRECTAL Y LOS RESULTADOS OBTENIDOS EN EL PACIENTE: RELATO PRECAUTORIOANTECEDENTES:Trabajos anteriores han demostrado una correlación entre la video-calificación de la habilidad quirúrgica y los resultados clínicos. Algunos autores han propuesto el uso de la revisión de videos para la evaluación de la habilidad técnica, la acreditación y la mejoría en la calidad quirúrgica.OBJETIVO:Antes de su adopción como medida de calidad entre los cirujanos colorrectales, primero debemos determinar si las evaluaciones de habilidades basadas en video pueden predecir los resultados clínicos de los pacientes entre cirujanos especializados.DISEÑO:Veintiún cirujanos enviaron un video representativo de una colectomía mínimamente invasiva. Cada video fue editado para resaltar los pasos clave y luego fué calificado por 10 cirujanos revisores utilizando una herramienta de evaluación validada por la ASCRS. Al vincular las calificaciones de los cirujanos al registro de resultados quirúrgicos aprobado, evaluamos la relación entre la habilidad y las tasas de complicaciones ajustadas al riesgo.AJUSTE:Colaboración en todo el estado incluyendo 70 hospitales comunitarios, académicos y terciarios, el Michigan Surgical Quality Collaborative.PACIENTES:Todos aquellos sometidos a resección colorrectal mínimamente invasiva realizada por los cirujanos participantes.MEDIDA DE RESULTADO PRINCIPAL:Complicaciones posoperatorias ajustadas al riesgo a los 30 días.RESULTADOS:La calificación de la habilidad técnica promedio de cada cirujano osciló entre 2.6 y 4.6. La tasa de complicaciones ajustada al riesgo por cirujano osciló entre el 9,9% y el 33,1%. Los pacientes operados por los cirujanos del cuartil inferior de las calificaciones generales de habilidades eran fumadores y añosos, y tambiés más propensos a la hipertensión arterial. Los pacientes operados por los cirujanos del cuartil superior tenían más probabilidades de ser inmunosuprimidos o tener una puntuación ASA> = 3. Después del ajuste de riesgo específico de la cirugía y el paciente, no hubo diferencias estadísticamente significativas en las tasas de complicaciones entre los cirujanos del cuartil inferior y superior (17,5% frente a 16,8%, respectivamente, p = 0,41).LIMITACIONES:Diseño de cohortes retrospectivo con seguimiento a corto plazo de los casos muestreados. Los videos se editaron para resaltar los pasos clave y los revisores no recibieron capacitación para establecer normas.CONCLUSIONES:Nuestro estudio demuestra que la evaluación realizada por los revisores basada en el video de la colectomía mínimamente invasiva no se correlacionó con las complicaciones post-operatorias entre los cirujanos especialistas. Por tanto, la adopción de la revisión del video quirúrgico para su uso en la acreditación profesional, debe abordarse con mucha precaución. Consulte Video Resumen en http://links.lww.com/DCR/B802. (Traducción-Dr. Xavier Delgadillo).

Analysis of Tools Used in Assessing Technical Skills and Operative Competence in Trauma and Orthopaedic Surgical Training: A Systematic Review

BACKGROUND

Robust assessment of skills acquisition and surgical performance during training is vital to ensuring operative competence among orthopaedic surgeons. A move to competency-based surgical training requires the use of tools that can assess surgical skills objectively and systematically. The aim of this systematic review was to describe the evidence for the utility of assessment tools used in evaluating operative performance in trauma and orthopaedic surgical training.

METHODS

We performed a comprehensive literature search of MEDLINE, Embase, and Google Scholar databases to June 2019. From eligible studies we abstracted data on study aim, assessment format (live theater or simulated setting), skills assessed, and tools or metrics used to assess surgical performance. The strengths, limitations, and psychometric properties of the assessment tools are reported on the basis of previously defined utility criteria.

RESULTS

One hundred and five studies published between 1990 and 2019 were included. Forty-two studies involved open orthopaedic surgical procedures, and 63 involved arthroscopy. The majority (85%) were used in the simulated environment. There was wide variation in the type of assessment tools in used, the strengths and weaknesses of which are assessor and setting-dependent.

CONCLUSIONS

Current technical skills-assessment tools in trauma and orthopaedic surgery are largely procedure-specific and limited to research use in the simulated environment. An objective technical skills-assessment tool that is suitable for use in the live operative theater requires development and validation, to ensure proper competency-based assessment of surgical performance and readiness for unsupervised clinical practice.

CLINICAL RELEVANCE

Trainers and trainees can gain further insight into the technical skills assessment tools that they use in practice through the utility evidence provided.

Success of Surgical Simulation in Orthopedic Training and Applications in Spine Surgery

STUDY DESIGN

This was a narrative review.

OBJECTIVE

This study aimed to review the current literature on surgical simulation in orthopedics and its application to spine surgery.

SUMMARY OF BACKGROUND DATA

As orthopedic surgery increases in complexity, training becomes more relevant. There have been mandates in the United States for training orthopedic residents the fundamentals of surgical skills; however, few studies have examined the various training options available. Lack of funding, availability, and time are major constraints to surgical simulation options.

METHODS

A PubMed review of the current literature was performed on all relevant articles that examined orthopedic trainees using surgical simulation options. Studies were examined for their thoroughness and application of simulation options to orthopedic surgery.

RESULTS

Twenty-three studies have explored orthopedic surgical simulation in a setting that objectively assessed trainee performance, most in the field of trauma and arthroscopy. However, there was a lack of consistency in measurements made and skills tested by these simulators. There has only been one study exploring surgical simulation in spine surgery.

CONCLUSIONS

While there has been a growing number of surgical simulators to train orthopedic residents the fundamentals of surgical skills, most of these simulators are not feasible, reproducible, or available to the majority of training centers. Furthermore, the lack of consistency in the objective measurements of these studies makes interpretation of their results difficult. There is a need for more simulation in spine surgery, and future simulators and their respective studies should be reproducible, affordable, applicable to the surgical setting, and easily assembled by various programs across the world.

The effect of verbal feedback, video feedback, and self-assessment on laparoscopic intracorporeal suturing skills in novices: a randomized trial

BACKGROUND

Laparoscopic skill acquisition involves a steep learning curve and laparoscopic suturing is an exceptionally challenging task. By improving the way feedback is given, trainees can learn these skills more effectively. This study aims to establish the most effective form of structured feedback on laparoscopic suturing skill acquisition in novices, by comparing the effects of expert verbal feedback, video review with expert feedback (video feedback), and video review with self-assessment.

METHODS

A prospective randomized blinded trial comparing verbal feedback, video feedback, and self-assessment. Novices in laparoscopic surgery were tasked with performing laparoscopic suturing with intracorporeal knot tying. Time was given for practice, and pre- and post-feedback assessments were undertaken. Suturing performance was measured using a task-specific checklist and global ratings. A post-study questionnaire was used to measure participant-perceived confidence, knowledge, and experience levels.

RESULTS

Fifty-one participants were randomized and allocated equally into the three groups. Performance in all three groups improved significantly from baseline. Video feedback had the largest improvement margin with checklist and global score improvements of 17.1% (± 9.9%) and 14.7% (± 9.3%), respectively. Performance improvements between groups were statistically significant in the global components (p = 0.004) but not the checklist components (p = 0.186). Global score improvement was significantly better in the video feedback group but was statistically insignificant between the self-assessment and verbal feedback groups. Questionnaire responses demonstrated positive results in confidence, knowledge, and experience levels, across all three study groups, with no differences between the groups (p > 0.05).

CONCLUSION

Structured video feedback facilitates reflection and self-directed learning, which improves the ability to develop proficiency in surgical skills. Combining both self-assessment and video feedback may be beneficial over verbal feedback alone due to the advantages of video review. These techniques should therefore be considered for implementation into surgical education curricula.

Improving junior doctor medicine prescribing and patient safety: An intervention using personalised, structured, video-enhanced feedback and deliberate practice

AIMS

This research investigated the effectiveness of an intervention for improving the prescribing and patient safety behaviour among Foundation Year doctors. The intervention consisted of simulated clinical encounters with subsequent personalised, structured, video-enhanced feedback and deliberate practice, undertaken at the start of four-month sub-specialty rotations.

METHODS

Three prospective, non-randomised control intervention studies were conducted, within two secondary care NHS Trusts in England. The primary outcome measure, error rate per prescriber, was calculated using daily prescribing data. Prescribers were grouped to enable a comparison between experimental and control conditions using regression analysis. A break-even analysis evaluated cost-effectiveness.

RESULTS

There was no significant difference in error rates of novice prescribers who received the intervention when compared with those of experienced prescribers. Novice prescribers not participating in the intervention had significantly higher error rates (P = .026, 95% confidence interval [CI] Wald 0.093 to 1.436; P = .026, 95% CI 0.031 to 0.397) and patients seen by them experienced significantly higher prescribing error rates (P = .007, 95% CI 0.025 to 0.157). Conversely, patients seen by the novice prescribers who received the intervention experienced a significantly lower rate of significant errors compared to patients seen by the experienced prescribers (P = .04, 95% CI -0.068 to -0.001). The break-even analysis demonstrates cost-effectiveness for the intervention.

CONCLUSION

Simulated clinical encounters using personalised, structured, video-enhanced feedback and deliberate practice improves the prescribing and patient safety behaviour of Foundation Year doctors. The intervention is cost-effective with potential to reduce avoidable harm.

Video Modeling and Video Feedback to Reduce Time to Perform Intravenous Cannulation in Medical Students: A Randomized-Controlled Mixed-Methods Study

PURPOSE

Combined video modeling (VM) and video feedback (VF) may be more beneficial than traditional feedback when teaching procedural skills. This study examined whether repeated VM and VF compared with VM alone reduced the time required for medical students to perform peripheral intravenous (IV) cannulation.

METHODS

Twenty-five novice medical students were randomly assigned to groups in a one-way blinded embedded mixed-methods study to perform IV cannulation. Participants received standardized instruction and performed IV cannulation on each other while being audio-video recorded. They were assigned to review a video of an expert performing IV cannulation (VM alone), or both the expert video and a video of their own most recent IV cannulation (VM+VF), before returning to perform another IV cannulation. This was repeated for a total of four IV cannulation encounters and three video reviews. A post-test interview was also conducted and analyzed qualitatively using thematic content analysis.

RESULTS

The median [interquartile range] time required to perform IV cannulation in the final encounter was significantly different between the VM+VF group vs VM alone group (126 [93-226] sec vs 345 [131-537] sec, respectively; median difference, 111 sec; 95% confidence interval, 8 to 391; P = 0.02). There was no significant difference in IV cannulation success between VM alone and VM+VF in the final encounter (75% vs 85% respectively; P = 0.65). For the VM+VF group, the time to perform IV cannulation was reduced after the final encounter compared with the baseline encounter (P = 0.002), which was not true of the VM alone group (P = 0.35).

CONCLUSION

Video modeling and feedback shortened time to IV skill completion, reduced complications, and improved satisfaction in novice medical students.

Simulation in Neonatal Resuscitation

Approximately 1 in 10 newborns will require basic resuscitation interventions at birth. Some infants progress to require more advanced measures including the provision of positive pressure ventilation, chest compressions, intubation and administration of volume/cardiac medications. Although advanced resuscitation is infrequent, it is crucial that personnel adequately trained in these techniques are available to provide such resuscitative measures. In 2000, Louis Halmalek et al. called for a "New Paradigm in Pediatric Medical Education: Teaching Neonatal Resuscitation in a Simulated Delivery Room Environment." This was one of the first articles to highlight simulation as a method of teaching newborn resuscitation. The last decades have seen an exponential growth in the area of simulation in newborn care, in particular in newborn resuscitation and stabilization. Simulation is best defined as an instructional strategy "used to replace or amplify real experiences with guided experiences that evoke or replicate substantial aspects of the real world in a fully interactive manner." Simulation training has now become an important point of how we structure training and deliver improved healthcare to patients. Some of the key aspects of simulation training include feedback, deliberate practice, outcome measurement, retention of skills and curriculum integration. The term "Train to win" is often used in sporting parlance to define how great teams succeed. The major difference between sports teams is that generally their game day comes once a week, whereas in newborn resuscitation every day is potentially "game day." In this review we aim to summarize the current evidence on the use of simulation based education and training in neonatal resuscitation, with particular emphasis on the evidence supporting its effectiveness. We will also highlight recent advances in the development of simulation based medical education in the context of newborn resuscitation to ensure we "train to win."

Video-Based Assessment in Surgical Education: A Scoping Review

BACKGROUND AND OBJECTIVE

Video-based assessment of residents' surgical skills may offer several advantages over direct observations of clinical performance in terms of objectivity, time-efficiency, and feasibility. Although video-based assessment is becoming more common in surgical training, a broad understanding of its utility is lacking. This scoping review explores video-based assessment in surgical training and presents the evidence supporting its use.

DESIGN

A literature search was conducted using the Web of Science database with key words related to video-based assessment and surgical training. Exclusion criteria included articles not published in English and articles on undergraduate medical education, continuing professional development, or non-surgical disciplines. Initially, 702 articles were identified; after title, abstract, and full-text screening by two independent reviewers (SM and VM), 199 articles remained.

RESULTS

We present the benefits of video-based assessment, including the ability to capture clinical ability in the operating room without decreasing intraoperative efficiency, as well as the potential to improve formative assessment and feedback practices. We describe the validity, reliability, and challenges of video-based assessment, as well as the use of video-based methods in clinical and simulated settings. We conclude by discussing questions that remain to be addressed.

CONCLUSIONS

Although further research and cost-benefit analyses are required, greater adoption of video-based assessment into surgical training may help meet increased assessment demands in an era of competency-based medical education.

Instant replay: Evaluation of instant video feedback in surgical novices for a laparoscopic gallbladder dissection

BACKGROUND

Athletes often use video to improve their technique. We hypothesized that surgical novices given feedback using video-replay would outperform surgical novices given verbal feedback in the performance of a laparoscopic task.

METHODS

Our study used a prospective, randomized control design. The surgical task involved the laparoscopic dissection of a pig gallbladder. Our participants performed a dissection, pre- and post-traditional or video feedback. Each recording was independently scored by two staff surgeons using the previously validated rating tools.

RESULTS

There was no significant difference between video feedback or traditional feedback groups in their mean overall or task specific scores. Both traditional and video-feedback groups had a trend towards improved performance post-feedback.

CONCLUSIONS

No significant difference in performance by both our global assessment metrics or task-specific metrics was observed. Video feedback requires further study to investigate its impact on surgical training.

Utilizing Intraprocedural Interactive Video Capture With Google Glass for Immediate Postprocedural Resident Coaching

BACKGROUND

Video coaching has been found to be an effective teaching method because it incorporates many of the established principles of successful adult learning. The goal of our study was to assess the feasibility and effectiveness of using a point-of-view video camera (Google Glass) to improve the surgical skills education of orthopaedic surgery residents.

METHODS

Forty-two residents from 4 institutions participated in a partially blinded randomized control trial performing an intra-articular distal tibial fracture reduction task while wearing Google Glass to record the performance. Participants underwent a structured coaching session with 20 participants (intervention group) using the recorded video to augment this session, and 22 participants (control group) receiving verbal coaching alone. The task was repeated again immediately after the coaching session. Performance was scored using an Objective Structured Assessment of Technical Skills checklist, Global Rating Scale, fluoroscopic usage, and reduction quality. A semistructured interview was then performed to assess experience of participants.

RESULTS

There was no significant difference (p > 0.05) seen in score improvement in the Objective Structured Assessment of Technical Skills checklist, Global Rating Scale, fluoroscopic usage, or reduction quality between the control and intervention groups. Thematic analysis of interview showed majority of participants found video coaching increased effectiveness in understanding of goals, developing techniques and strategies, and process of self-reflection. Their involvement was seen overall as a positive experience, with participants wanting to see more inclusion of video coaching within surgical education.

CONCLUSIONS

No difference in performance improvement between the 2 groups was seen, but majority of participants found the video coaching sessions valuable and could have potential beneficial role in education.

Personalized Video Feedback and Repeated Task Practice Improve Laparoscopic Knot-Tying Skills: Two Controlled Trials

PURPOSE

Compare the effect of personalized feedback (PF) vs. task demonstration (TD), both delivered via video, on laparoscopic knot-tying skills and perceived workload; and evaluate the effect of repeated practice.

METHOD

General surgery interns and research fellows completed four repetitions of a simulated laparoscopic knot-tying task at one-month intervals. Midway between repetitions, participants received via e-mail either a TD video (demonstration by an expert) or a PF video (video of their own performance with voiceover from a blinded senior surgeon). Each participant received at least one video per format, with sequence randomly assigned. Outcomes included performance scores and NASA Task Load Index (NASA-TLX) scores. To evaluate the effectiveness of repeated practice, scores from these trainees on a separate delayed retention test were compared against historical controls who did not have scheduled repetitions.

RESULTS

Twenty-one trainees completed the randomized study. Mean change in performance scores was significantly greater for those receiving PF (difference = 23.1 of 150 [95% confidence interval (CI): 0, 46.2], P = .05). Perceived workload was also significantly reduced (difference = -3.0 of 20 [95% CI: -5.8, -0.3], P = .04). Compared with historical controls (N = 93), the 21 with scheduled repeated practice had higher scores on the laparoscopic knot-tying assessment two weeks after the final repetition (difference = 1.5 of 10 [95% CI: 0.2, 2.8], P = .02).

CONCLUSIONS

Personalized video feedback improves trainees' procedural performance and perceived workload compared with a task demonstration video. Brief monthly practice sessions support skill acquisition and retention.

Effective Training and Assessment of Surgical Skills, and the Correlates of Performance

This report briefly describes the University of Toronto Surgical Skills Centre and summarizes research in technical skills conducted at that site. This includes work on curriculum evaluation, the development and validation of assessment instruments, the retention of technical skills after training, and the prediction of success in surgery. These studies benefited from the large number of participants made available by the Surgical Skills Centre, allowing for randomized controlled studies or correlation studies where larger numbers are necessary for adequate statistical power. Recent emphasis has been on the further development of objective means of assessment and the exploration of correlates of surgical performance. Ongoing research is also aimed at simulator validation.

Evaluating Intraoperative Laparoscopic Skill: Direct Observation Versus Blinded Videotaped Performances

The Global Operative Assessment of Laparoscopic Skill (GOALS) has been shown to meet high standards for direct observation. The purpose of this study was to investigate the reliability and validity of GOALS when applied to blinded, videotaped performances. Five novice surgeons and 5 experienced surgeons were each evaluated by 2 observers during a laparoscopic cholecystectomy. Subsequently, 4 laparoscopists (V1 to V4) evaluated the videotaped procedures using GOALS. Two of the raters (V1 and V3) had prior experience using GOALS. The interrater reliabilities between video raters (VRs) and between VRs and direct raters (DRs) were calculated using the intraclass correlation coefficient. Construct validity was assessed using 2-way analysis of variance. Interrater reliability between the 4 VRs and the 2 DRs was 0.72. The intraclass correlation coefficient for the 4 VRs was 0.68 and for each VR compared with the mean DR was 0.86, 0.39, 0.94, and 0.76, respectively. All raters, except V2, differentiated between novice and experienced groups ( P values ranged from .01 to .05). These data suggest that GOALS can be used to assess laparoscopic skill based on videotaped performances but that rater training may play an important role in ensuring the reliability and validity of the instrument. Experience with the tool in the operating room may improve the reliability of video rating and could be of value in training evaluators.

Video-augmented feedback for procedural performance

BACKGROUND

Resident programs must assess residents' achievement of core competencies for clinical and procedural skills.

AIMS

Video-augmented feedback may facilitate procedural skill acquisition and promote more accurate self-assessment.

METHODS

A randomized controlled study to investigate whether video-augmented verbal feedback leads to increased procedural skill and improved accuracy of self-assessment compared to verbal only feedback. Participants were evaluated during procedural training for ultrasound guided internal jugular central venous catheter (US IJ CVC) placement. All participants received feedback based on a validated 30-point checklist for US IJ CVC placement and validated 6-point procedural global rating scale.

RESULTS

Scores in both groups improved by a mean of 9.6 points (95% CI: 7.8-11.4) on the 30-point checklist, with no difference between groups in mean score improvement on the global rating scale. In regards to self-assessment, participant self-rating diverged from faculty scoring, increasingly so after receiving feedback. Residents rated highly by faculty underestimated their skill, while those rated more poorly demonstrated increasing overestimation.

CONCLUSIONS

Accuracy of self-assessment was not improved by addition of video. While feedback advanced the skill of the resident, video-augmented feedback did not enhance skill acquisition or improve accuracy of resident self-assessment compared to standard feedback.

Feedback in formative OSCEs: comparison between direct observation and video-based formats

INTRODUCTION

Medical students at the Faculty of Medicine, University of Geneva, Switzerland, have the opportunity to practice clinical skills with simulated patients during formative sessions in preparation for clerkships. These sessions are given in two formats: 1) direct observation of an encounter followed by verbal feedback (direct feedback) and 2) subsequent review of the videotaped encounter by both student and supervisor (video-based feedback). The aim of the study was to evaluate whether content and process of feedback differed between both formats.

METHODS

In 2013, all second- and third-year medical students and clinical supervisors involved in formative sessions were asked to take part in the study. A sample of audiotaped feedback sessions involving supervisors who gave feedback in both formats were analyzed (content and process of the feedback) using a 21-item feedback scale.

RESULTS

Forty-eight audiotaped feedback sessions involving 12 supervisors were analyzed (2 direct and 2 video-based sessions per supervisor). When adjusted for the length of feedback, there were significant differences in terms of content and process between both formats; the number of communication skills and clinical reasoning items addressed were higher in the video-based format (11.29 vs. 7.71, p=0.002 and 3.71 vs. 2.04, p=0.010, respectively). Supervisors engaged students more actively during the video-based sessions than during direct feedback sessions (self-assessment: 4.00 vs. 3.17, p=0.007; active problem-solving: 3.92 vs. 3.42, p=0.009). Students made similar observations and tended to consider that the video feedback was more useful for improving some clinical skills.

CONCLUSION

Video-based feedback facilitates discussion of clinical reasoning, communication, and professionalism issues while at the same time actively engaging students. Different time and conceptual frameworks may explain observed differences. The choice of feedback format should depend on the educational goal.

Randomized trial to assess the effect of supervised and unsupervised video feedback on teaching practical skills

BACKGROUND

Feedback is a vital component of the learning process; however, great variation exists in the quality, quantity, and method of delivery. Video feedback is not commonly used in the teaching of surgical skills. The aim of this trial was to evaluate the benefit of 2 types of video feedback-individualized video feedback (IVF), with the student reviewing their performance with an expert tutor, and unsupervised video-enhanced feedback (UVF), where the student reviews their own performance together with an expert teaching video-to determine if these improve performance when compared with a standard lecture feedback.

METHODS

A prospective blinded randomized control trial comparing lecture feedback with IVF and UVF was carried out. Students were scored by 2 experts directly observing the performance and 2 blinded experts using a validated pro forma. Participants were recorded on video when performing a suturing task. They then received their feedback via any of the 3 methods before being invited to repeat the task.

RESULTS

A total of 32 students were recruited between the 3 groups. There was no significant difference in suturing skill performance scores given by those directly observing the students and those blinded to the participant. There was no statistically significant difference between the 2 video feedback groups (p = 1.000), but there was significant improvement between standard lecture feedback and UVF (p = 0.047) and IVF (p = 0.001).

CONCLUSION

Video feedback can facilitate greater learning of clinical skills. Students can attain a similar level of surgical skills improvement with UVF as with teacher-intensive IVF.

Determining the Role of a National Objective Assessment of Surgical Skills in Gynecological Oncology: An e-Delphi Methodology

Objective

The aim of this study is to determine a national consensus on the role of an objective assessment of technical surgical skills in gynecological oncology (GO).

Methods

After approval was obtained from Society of Gynecologic Oncology of Canada, A panel of 20 GO leaders was assembled, representing all GO fellowship programs, and was asked to participate in an anonymous group and respond to an online 49-item questionnaire using a modified Delphi methodology.

Results

Ninety-five percent (n = 19) of those invited to participate did so. Seventeen of the panelists (89.5%) believed there was no sufficiently standardized technical skills assessment for GO fellows, whereas 18 responders (95%) believed that fellows should be objectively assessed on more than 1 occasion during their training. Consensus was predefined as Cronbach α greater than 0.8. The panel agreed on what procedures should be objectively assessed with a Cronbach α of 0.967. An overall Cronbach α of 0.993 was achieved after a single Delphi round.

Conclusions

We achieved consensus on the possible components and logistics of a skills assessment process among a group of highly experienced GO trainers in Canada. This study provides the basis for further investigation and debate on the potential value, necessity, and feasibility of an assessment of advanced surgical and nonsurgical skills of GO trainees.

Randomized trial of the effect of video feedback on the acquisition of surgical skills

Background

Constructive feedback provides a mechanism for reinforcing learning during the acquisition of surgical skills. Feedback is usually given verbally, and sometimes documented, after direct observation by a trained assessor. The aim was to evaluate video recording as an effective modality for enhancing feedback, in comparison with standard verbal feedback alone.

Methods

This was a prospective, blinded, randomized clinical trial comparing standard verbal feedback plus video with standard verbal feedback alone. Validated pro formas for assessment were used and quality control was performed by independent expert assessors. Trial participants were recorded on video performing the surgical skill, and returned the next day to perform the skill again following video and standard verbal feedback (group 1) or standard verbal feedback alone (group 2).

Results

Forty-eight participants were divided equally between the two groups. There was a significant improvement in the mean overall procedure score for group 1 of 2·875 from a maximum achievable score of 20 (P = 0·003), but not for group 2. There were significant improvements in the specific domains of instrument familiarity, needle handling, skin handling and accurate apposition, again all in group 1. The only significant improvement in group 2 was in an organized approach to the task, also observed in group 1. Knot-tying security deteriorated after feedback in group 2 but not in group 1.

Conclusion

The addition of video feedback can improve the acquisition of surgical skills, and could be incorporated into formal surgical curricula.

A comparison C1-C2 transarticular screw placement after self-education and mentored education of orthopaedic residents

STUDY DESIGN

Prospective randomized trial.

OBJECTIVE

This study will provide preliminary data on whether residents can be "self-taught" and to what extent a lecture, demonstration, and coaching can improve skills and knowledge.

BACKGROUND DATA

Practice-based learning is an essential competency in Accreditation Council for graduate Medical Education-accredited residencies. Little has been done to demonstrate whether residents can be self-taught or benefit from mentoring in understanding and performing difficult surgical tasks.

METHODS

A written test was given to orthopedic residents on C1-C2 transarticular screw placement. They were then provided reading on C1-C2 transarticular screw placement. Residents were then divided into a "self-directed learning" group and a "mentored learning" group. All residents then performed the technique on models, with the "mentored" group receiving a lecture and coaching from the mentor. The models were analyzed for technique errors and the previous test was administered again as a posttest. The test and screw placement were repeated 4 months later.

RESULTS

Residents without mentoring had an average improvement of 4.5 points, those with mentoring had average improvement of 8.6 points (P=0.0068). The screw placement technique error rate for the nonmentored group (n=8) was 2.55, and for the mentored group (n=9) was 1.47 (P=0.004). Sixteen residents completed the delayed test, 7 from the nonmentored groups and 8 from the mentored group. Nine residents were able to repeat the screw placement technique 4 months after the initial test and screw placement, 3 nonentored, and 6 mentored. Although there were some trends toward the mentored group having better retention, neither knowledge nor skill was statistically different.

CONCLUSIONS

This preliminary trial seems to indicate that residents provided a lecture and guided technical instruction will obtain knowledge and perform procedures better than those that do not. Conclusions based upon Post Graduate year, motivation, and interest in spine surgery could not be made.

Keynote address: the focus on competencies and individual learner assessment as emerging themes in medical education research

This article describes opportunities for scholarship in medical education, based on a brief overview of recent changes in medical education. The implications arising from these changes are discussed, with recommendations for focus, and suggestions and examples for making progress in this field. The author discusses 1) the historical context of the current shift toward competency-based medical education, 2) the potential contribution of social and behavioral sciences to medical education scholarship, 3) methods and approaches for supporting scholarship in medical education, and very briefly 4) trends in simulation. The author concludes with a call for quality in medical education scholarship and argues that the most promising and fruitful area of medical education scholarship for the future lies in the field of assessment of individual competence.

Objective structured assessment of technical skill in upper extremity surgery

PURPOSE

Objective assessment of technical skills in hand surgery has been lacking. This article reports on an Objective Structured Assessment of Technical Skills format of a multiple bench-station evaluation of orthopedic surgery residents' technical skills for 3 common upper extremity surgeries.

METHODS

Twenty-seven residents (6 postgraduate year [PGY] 2, 8 PGY 3, 8 PGY 4, and 5 PGY 5) participated in the examination. Each resident performed surgery on a cadaveric specimen at 3 stations, trigger finger release (TFR), open carpal tunnel release, and distal radius fracture fixation. A board-certified hand surgeon evaluated trainee performance at each station, using a procedure-specific detailed checklist, a validated global rating scale, and pass/fail assessment. A resident post-testing evaluation was collected.

RESULTS

Construct validity with correlation between year in training and detailed checklist scores was demonstrated for TFR and carpal tunnel release; between year in training and global rating scores for TFR and distal radius fracture fixation; and between year in training and pass/fail assessment for TFR. Criterion validity was demonstrated by the correlation between global rating scale scores, detailed checklist scores, and pass/fail assessment for TFR, carpal tunnel release, and distal radius fracture fixation. Time to complete the surgery was not correlated with surgical performance. Residents rated the multiple-station Objective Structured Assessment of Technical Skills format as highly educational.

CONCLUSIONS

This study reports that a surgeon's ability to release a trigger finger does not correlate specifically to his or her ability to perform a carpal tunnel release or to perform plate fixation of a radius fracture. The results of this study would indicate that, for 3 different surgical simulations representing procedures of varying complexity, assessments by a single assessment tool is not adequate. To completely understand a resident's abilities, assessment by checklist (understanding the steps of the surgery), global rating scales (assessment of basic surgical skills in light of lesser or greater complexity surgeries), and pass/fail assessment (examination of adverse events) are all necessary components.

TYPE OF STUDY/LEVEL OF EVIDENCE

Therapeutic II.

Improving the accuracy of self-assessment of practical clinical skills using video feedback--the importance of including benchmarks

INTRODUCTION

Isolated video recording has not been demonstrated to improve self-assessment accuracy. This study examines if the inclusion of a defined standard benchmark performance in association with video feedback of a student's own performance improves the accuracy of student self-assessment of clinical skills.

METHODS

Final year medical students were video recorded performing a standardised suturing task in a simulated environment. After the exercise, the students self-assessed their performance using global rating scales (GRSs). An identical self-assessment process was repeated following video review of their performance. Students were then shown a video-recorded 'benchmark performance', which was specifically developed for the study. This demonstrated the competency levels required to score full marks (30 points). A further self-assessment task was then completed. Students' scores were correlated against expert assessor scores.

RESULTS

A total of 31 final year medical students participated. Student self-assessment scores before video feedback demonstrated moderate positive correlation with expert assessor scores (r = 0.48, p < 0.01) with no change after video feedback (r = 0.49, p < 0.01). After video feedback with benchmark performance demonstration, self-assessment scores demonstrated a very strong positive correlation with expert scores (r = 0.83, p < 0.0001).

CONCLUSIONS

The demonstration of a video-recorded benchmark performance in combination with video feedback may significantly improve the accuracy of students' self-assessments.

The impact of nontechnical skills on technical performance in surgery: a systematic review

BACKGROUND

Failures in nontechnical and teamwork skills frequently lie at the heart of harm and near-misses in the operating room (OR). The purpose of this systematic review was to assess the impact of nontechnical skills on technical performance in surgery.

STUDY DESIGN

MEDLINE, EMBASE, PsycINFO databases were searched, and 2,041 articles were identified. After limits were applied, 341 articles were retrieved for evaluation. Of these, 28 articles were accepted for this review. Data were extracted from the articles regarding sample population, study design and setting, measures of nontechnical skills and technical performance, study findings, and limitations.

RESULTS

Of the 28 articles that met inclusion criteria, 21 articles assessed the impact of surgeons' nontechnical skills on their technical performance. The evidence suggests that receiving feedback and effectively coping with stressful events in the OR has a beneficial impact on certain aspects of technical performance. Conversely, increased levels of fatigue are associated with detriments to surgical skill. One article assessed the impact of anesthesiologists' nontechnical skills on anesthetic technical performance, finding a strong positive correlation between the 2 skill sets. Finally, 6 articles assessed the impact of multiple nontechnical skills of the entire OR team on surgical performance. A strong relationship between teamwork failure and technical error was empirically demonstrated in these studies.

CONCLUSIONS

Evidence suggests that certain nontechnical aspects of performance can enhance or, if lacking, contribute to deterioration of surgeons' technical performance. The precise extent of this effect remains to be elucidated.

Identifying orthopaedic surgeons of the future

The aim of this study was to investigate the effect of training on the arthroscopic performance of a group of medical students and to determine whether all students could be trained to competence. Thirty-three medical students with no previous experience of arthroscopy were randomised to a ‘Trained’ or an ‘Untrained’ cohort. They were required to carry out 30 episodes of two simulated arthroscopic tasks (one shoulder and one knee). The primary outcome variable was task success at each episode. Individuals achieved competence when their learning curve stabilised. The secondary outcome was technical dexterity, assessed objectively using a validated motion analysis system. Six subjects in the ‘Untrained’ cohort failed to achieve competence in the shoulder task, compared with one in the ‘Trained’ cohort. During the knee task, two subjects in each cohort failed to achieve competence. Based on the objective motion analysis parameters, the ‘Trained’ cohort performed better on the shoulder task (p < 0.05) but there was no significant difference for the knee task (p > 0.05).

Although specific training improved the arthroscopic performance of novices, there were individuals who could not achieve competence despite focused training.These findings may have an impact on the selection process for trainees and influence individual career choices.

The assessment of suturing skills of dental undergraduates

INTRODUCTION

To improve the assessment of suturing skills, we firstly developed a checklist based assessment. As an aid to the teaching of suturing skills a video was subsequently developed.

MATERIALS AND METHODS

To validate the checklist a cohort of 57 final year students were videoed performing the formative suturing assessment which was assessed by two examiners independently. Two cohorts of third year students took part in the formal assessment but only the second cohort of students would have access to the video.

RESULTS

The first cohort of 58 third year students' success rate was 93% and 1 year later 94% of 53 third year students were successful. One hundred per cent of the first cohort and 98% of the second cohort thought that the exam was a fair assessment of suturing skills. The majority of students thought that the checklist was helpful (94% and 93%). However, 62% and 55% thought that the assessment was a stressful experience. However, only 80% of the first cohort compared with 98% of the second cohort thought that they could now place sutures in intra-oral wounds.

CONCLUSION

Our findings suggest that checklist based assessments of suturing skills were well received by the students and improved the objectivity and transparency of the assessment process. Further work is required to determine if teaching initially on a tabletop model can be transferred to the dental clinical situation.

Workplace-based assessment: assessing technical skill throughout the continuum of surgical training

The Royal Colleges of Surgeons and Surgical Specialty Associations in the UK have introduced competence-based syllabi and curricula for surgical training. The syllabi of the Intercollegiate Surgical Curriculum Programme (ISCP) and Orthopaedic Curriculum and Assessment Programme (OCAP) define the core competencies, that is, the observable and measureable behaviours required of a surgical trainee. The curricula define when, where and how these will be assessed. Procedure-based assessment (PBA) has been adopted as the principal method of assessing surgical skills. It combines competencies specific to the procedure with generic competencies such as safe handling of instruments. It covers the entire procedure, including preoperative and postoperative planning. A global summary of the level at which the trainee performed the assessed elements of the procedure is also included. The form has been designed to be completed quickly by the assessor (clinical supervisor) and fed-back to the trainee between operations. PBA forms have been developed for all index procedures in all surgical specialties. The forms are intended to be used as frequently as possible when performing index procedures, as their primary aim is to aid learning. At the end of a training placement the aggregated PBA forms, together with the logbook, enable the Educational Supervisor and/or Programme Director to make a summary judgement about the competence of a trainee to perform index procedures to a given standard.

Validation of orthopaedic bench models for trauma surgery

The aim of this study was to validate the use of three models of fracture fixation in the assessment of technical skills. We recruited 21 subjects (six experts, seven intermediates, and eight novices) to perform three procedures: application of a dynamic compression plate on a cadaver porcine model, insertion of an unreamed tibial intramedullary nail, and application of a forearm external fixator, both on synthetic bone models. The primary outcome measures were the Objective Structural Assessment of technical skills global rating scale on video recordings of the procedures which were scored by two independent expert observers, and the hand movements of the surgeons which were analysed using the Imperial College Surgical Assessment Device. The video scores were significantly different for the three groups in all three procedures (p < 0.05), with excellent inter-rater reliability (alpha = 0.88). The novice and intermediate groups specifically were significantly different in their performance with dynamic compression plate and intramedullary nails (p < 0.05). Movement analysis distinguished between the three groups in the dynamic compression plate model, but a ceiling effect was demonstrated in the intramedullary nail and external fixator procedures, where intermediates and experts performed to comparable standards (p > 0.6). A total of 85% (18 of 21) of the subjects found the dynamic compression model and 57% (12 of 21) found all the models acceptable tools of assessment. This study has validated a low-cost, high-fidelity porcine dynamic compression plate model using video rating scores for skills assessment and movement analysis. It has also demonstrated that Synbone models for the application of and intramedullary nail and an external fixator are less sensitive and should be improved for further assessment of surgical skills in trauma. The availability of valid objective tools of assessment of surgical skills allows further studies into improving methods of training.

Checklists for assessment and certification of clinical procedural skills omit essential competencies: a systematic review

OBJECTIVE

To develop generic criteria for the global assessment of clinical procedural competence and to quantify the extent to which existing checklists allow for holistic assessment of procedural competencies.

METHODS

We carried out a systematic review and qualitative analysis of published clinical procedural skills assessment checklists and enumerated the contents of each. Source materials included all English-language papers published from 1990 to June 2005, identified from 18 databases, which described or referred to an assessment document for any clinical procedural skill. A pair of reviewers identified key generic themes and sub-themes through in-depth analysis of a subset of 20 checklists with iterative agreement and independent retesting of a coding framework. The resulting framework was independently applied to all checklists by pairs of reviewers checking for the emergence of new themes and sub-themes. Main outcome measures were identification of generic clinical procedural skills and the frequency of occurrence of each in the identified checklists.

RESULTS

We identified 7 themes ('Procedural competence', represented in 85 [97%] checklists; 'Preparation', 65 [74%]; 'Safety', 45 [51%]; 'Communication and working with the patient', 32 [36%]; 'Infection control', 28 [32%]; 'Post-procedural care', 24 [27%]; 'Team working', 13 [15%]) and 37 sub-themes, which encapsulated all identified checklists. Of the sub-themes, 2 were identified after the initial coding framework had been finalised.

CONCLUSIONS

It is possible to develop generic criteria for the global assessment of clinical procedural skills. A third and a half of checklists, respectively, do not enable explicit assessment of the key competencies 'Infection control' and 'Safety'. Their assessment may be inconsistent in assessments which use such checklists.

Predicting the technical competence of surgical residents

I describe an approach to predicting competence in technical skills for the purposes of resident selection. To demonstrate a predictive relationship, it is necessary to use measures that exhibit variation, reliability, and validity. There is little evidence that such measures are routinely used in the process of selecting residents. I argue that the selection of assessment instruments in the predictor domain must be guided by relevant theoretical considerations, while assessment in the surgical domain must make use of more objective and reliable instruments than is currently the practice. I present a brief summary of research on predicting operative technical competence.

Simulator assessment of innate endovascular aptitude versus empirically correct performance

OBJECTIVE

Prior work has established that performance on an endovascular simulator incorporating tactile feedback (haptics) correlates with previous endovascular experience and can be improved by training. This study was designed to test the ability to define and measure innate endovascular aptitude and empirically correct performance and to determine whether these are two different things.

METHODS

Subjects ranging in endovascular skill level from novice to expert were surveyed to determine video game experience and skill, endovascular level of training, and endovascular experience. They were then tested by using a standard protocol requiring timed advancement of a catheter and wire sequentially into each of three vessels arising from a simulated type I arch. Recorded trials were independently and blindly scored by two experienced endovascular faculty members by using a modification of a previously validated scale (Modified Reznick Scale; MRS). Summed scores were analyzed by frequency analysis and categorized as satisfactory and unsatisfactory on the basis of a clear bimodal distribution. Categorical outcome, time to task completion, and other variables were analyzed by means of linear regression, analysis of variance, and Welch modified two-sample t tests, as indicated.

RESULTS

A total of 61 subjects were enrolled: 42% students, 8% technicians, 19% surgeons, 13% cardiologists, and 18% radiologists. Of these, 62% were considered novices and 30% experts on the basis of previous experience; 56% of subjects worked in an endovascular-related occupation. MRS scores were highly correlated between raters (P < .0001) and showed a clear bimodal distribution, with subjects in any endovascular occupation (including technicians) scoring significantly better than all others (P < .0001). Hours of video games played per week were correlated highly with completion times (P < .001) and MRS scores (P < .001). Measures of formal training (number of endovascular cases and occupation) correlated highly with completion times (all P < .03) and MRS scores (all P < .008). In comparing completion times vs MRS scores, three groups were apparent: unskilled-inexperienced, skilled-inexperienced, and skilled-experienced, corresponding primarily to senior subjects without endovascular experience, younger subjects without endovascular experience, and formally trained endovascular physicians, respectively. Those judged intermediate in aptitude reduced times to the lowest possible level before improving their MRS scores.

CONCLUSIONS

Although inherently subjective, the MRS yields reproducible scores that correlate with endovascular experience and formal training. Experts and novices with extensive video game experience achieve short completion times, whereas high MRS scores are achieved only by formally trained subjects. Innate endovascular aptitude and empirically correct performance may be two separate things, and aptitude may be acquirable through (or identified by) extensive nonmedical video game experience.