Validity of scoring systems for the assessment of technical and non-technical skills in ophthalmic surgery-a systematic review

Case study and pilot results: Stepwise approach to teach a resident tube shunt surgery

Purpose

To illustrate the utility of a previously published stepwise rubric for evaluating a resident's progress learning aqueous tube shunt surgery.

Method

Using a stepwise rubric, a single PGY3 ophthalmology resident and attending glaucoma surgeon evaluated the resident's performance after each aqueous tube shunt surgery. The rubric subdivides the surgery into 12 consecutive steps and scores the resident's proficiency in each step with either a 0 (observation), 2 (novice), 3 (beginner), 4 (advanced beginner), or 5 (competent).

Results

The resident's cumulative score increased significantly throughout the 17 surgeries performed, with the resident's self-evaluated score and attending's score increasing from 12 to 27 and 14 to 27 from the first to last surgery, respectively. Scores were consistent between the resident and attending; for any given surgery, the resident's own score never deviated from the attending's score by more than 1 point. The resident completed at least 50% of the steps in 11 of the 17 cases. While some surgical steps were mastered earlier on ("tube tying" and "suture implantation"), other steps were more challenging to master ("tunnel in sclera and enter the AC" and "close conjunctiva", as demonstrated by fewer overall attempts or never attaining a score of '5' despite multiple attempts.

Conclusions and Importance

This study demonstrates the utility of the stepwise rubric in tracking resident surgical scores chronologically via self and attending assessment. The ability to compare their own scores to that of an attending allows the resident to learn how to effectively evaluate their own performance. Most importantly, statistics obtained for each step provides the resident with personalized and real-time feedback for learning specific surgical steps. In conclusion, the stepwise rubric is a useful add-on to a resident's aqueous tube shunt surgery education.

The importance of non-technical skills in robot-assisted surgery in gynaecology

Robot-assisted surgery (RAS) in gynaecology has undergone exponential growth in recent decades, with utility in treating both benign and malignant gynaecological conditions. The technological complexities and amended theatre dynamics that RAS demands mean that effective non-technical skills (NTS) are vitally important to overcome these unique challenges. However, NTS have been neglected in RAS-training programmes with focus placed instead on the exclusive acquisition of technical skills (TS). NTS include teamwork, communication, leadership, situational awareness, decision-making and stress management. Communication is the most frequently cited NTS impacted during RAS, as the physical limitations imposed by the robotic hardware make communication exchange difficult. The full immersion that RAS enables can contribute to situational awareness deficits. However, RAS can complement communication and teamwork when multidisciplinary (MDT) surgeries (such as complex endometriosis excisions) are undertaken; dual-console capabilities facilitate the involvement of specialties such as general surgery and urology. The development of NTS in RAS cannot be achieved with in-situ experience alone, and current training is poorly standardised. RAS-training programmes and curricula for gynaecology do exist, however the integration of NTS remain limited. Simulation is a viable tool to facilitate enhanced-NTS integration, yet cost implications form a barrier to its wider implementation. However, given that RAS will continue to occupy a greater proportion of the gynaecological caseload, integration of NTS within gynaecological RAS training curricula is necessary. Patients undergoing gynaecological RAS would benefit from the improved safety standards and enhanced surgical outcomes that would result.

Simulation-based training for intraoperative posterior capsule rupture management: an analysis of nontechnical skills development and the relationship between technical and nontechnical skills

OBJECTIVES

To analyse development of individual nontechnical skills (NTS) domains after undertaking a previously developed simulation-based training model and analyse the relationship between technical skills (TS) and NTS in ophthalmic surgery.

METHODS

The simulation-based training model involved a cataract surgery case complicated by intraoperative posterior capsule rupture. Cataract surgeons underwent the simulation twice, separated by a training intervention. Two blinded independent experts assessed participants' NTS using HUFOES, NOn-Technical Skills for Surgeons (NOTSS), and the OSATS global rating scale for TS. Paired t-tests assessed differences in individual NTS domains, with p < 0.05 indicating significance. The Pearson Product Moment Correlation Coefficient was used to assess the correlation between scores from each scoring system.

RESULTS

All NTS domains within HUFOES and NOTSS demonstrated statistically significant improvements secondary to the training intervention. Positive correlations were demonstrated between HUFOES and OSATS scores in the pre- and post-training simulations, r = 0.870 (p < 0.001) and r = 0.861 (p < 0.001), respectively. Positive correlations were also demonstrated between NOTSS and OSATS scores in pre- and post-training simulations, r = 0.849 (p < 0.001) and r = 0.757 (p = 0.001), respectively. Positive correlations were demonstrated between HUFOES and NOTSS scores; r = 0.979 (p < 0.001) (n = 17) and r = 0.959 (p < 0.001) for pre- and post-training simulations, respectively.

CONCLUSION

All NTS domains contained within HUFOES and NOTSS demonstrated significant increases following the completion of the simulation-based training model. Positive correlations exist between an ophthalmic surgeon's TS and NTS. This is the first study to report these findings within ophthalmic surgery.

The making of a (new) surgeon: Opportunities for repeated messaging on NOTSS in the Canadian medical system

Proficiency in nontechnical skills (NTS) contributes to reduction in critical safety incidents and improvement in patient safety outcomes. Despite evidence demonstrating the importance of NTS in patient safety, there remains limited NTS specific curricula and formal teaching in Canadian surgical programs. We propose a three-stage longitudinal approach to education surrounding NTS using the Nontechnical skills for surgeons (NOTSS) framework.

Cataract surgical training in Europe: European Board of Ophthalmology survey

PURPOSE

To survey recently graduated European ophthalmologists concerning cataract surgery (CS) training opportunities.

SETTING

Countries affiliated to the European Board of Ophthalmology (EBO).

DESIGN

Cross-sectional study of anonymous survey results.

METHODS

A 23-question online survey was emailed to candidates who sat the EBO Diploma Examination as residents between 2018 and 2022.

RESULTS

821 ophthalmologists from 30 countries completed the survey. The mean residency duration was 4.73 (SD 0.9) years. The mean reported number of entire CS procedures performed was 80.7 (SD 100.6) at the end of residency, but more than 25% of respondents (n = 210) had received no live CS training during their residency. The self-confidence (scale, 1 to 10) to perform a simple case or challenging case, manage posterior capsular rupture, and realize a corneal stitch were rated 4.1, 3.2, 4.2, 2.4, respectively. We observed extensive variation in clinical exposure to CS and self-reported confidence to perform CS between European trainees. Females reported a mean of 18% fewer entire procedures than their male colleagues and were also less confident in their surgical skills (P < .05). Trainees in residency programs longer than 5 years performed fewer procedures and were less confident than trainees in residences of shorter duration (P < .001). The importance of fellowships to complete surgical education was rated 7.7 out of 10.

CONCLUSIONS

CS training across European countries lacks harmony. Female ophthalmology trainees continue, as in other specialties, to experience apparent gender bias. European level recommendations seem necessary to raise and harmonize competency-based CS training programs and promote post-residency fellowship training programs.

Non-technical skills simulation-based training model for managing intraoperative posterior capsule rupture during cataract surgery

OBJECTIVES

To develop and implement a simulation-based training model for the management of posterior capsule rupture (PCR) from a non-technical skills (NTS) perspective, and analyse changes in participant's NTS and technical skills (TS).

METHODS

The simulation-based training model consisted of two identical PCR simulations with NTS stressors applied, separated by a predominantly NTS focussed training intervention. Participants' TS and NTS were evaluated by two blinded assessors using the Objective Structured Assessment of Technical Skill (OSATS) global rating scale and the HUman Factors in intraoperative Ophthalmic Emergencies Scoring System (HUFOES) respectively. Paired t-tests were used to establish the difference in mean HUFOES and OSATS scores between initial and repeat simulations; p < 0.05 indicated statistical significance. McGaghie's model of translational outcomes for simulation-based learning was used to establish the simulation model's educational status.

RESULTS

Seventeen cataract surgeons of varying training grades participated in the simulation-based training model. NTS improved with statistical significance; mean HUFOES scores increased from 48.7 ± 16.6 to 59.2 ± 14.8 (p < 0.001). Mean OSATS scores increased without statistical significance from 16.0 ± 7.3 to 17.9 ± 8.3 (p = 0.07). This simulation model achieved Level 1 (internal acceptability) and Level 2 (contained effects) according to McGaghie's model.

CONCLUSIONS

This novel simulation-based training model was designed to improve the NTS required for managing intraoperative PCR, through the provision of an interactive training session. Statistically significant improvements in participants' NTS in combination with statistically insignificant improvements in TS demonstrate that the simulation-based training model has specificity within the NTS domain.

The Ophthalmology Surgical Competency Assessment Rubric (OSCAR) for Open Globe Surgical Management

Background/Aims

To produce an internationally developed rubric to assess surgical competency in open globe management.

Methods

An international expert panel of seven ophthalmologist educators developed a standardized competency-based rubric. The steps to perform an open globe repair were outlined. Based on a modified Dreyfus model, the experts agreed on the steps of surgery and certain global indices. They then assigned descriptors for the competency expected of a novice, beginner, advanced beginner, and competent surgeon. The tool was then vetted by another panel of ten ophthalmologists. The main outcome measure was the final version of the tool as agreed upon by the expert review panel.

Results

The steps of open globe repair and key global indices were established. Descriptive wording for each step and global indices for novice, beginner, advanced beginner, and competent surgery were listed. All the expert comments were integrated to establish face and content validity.

Conclusion

This standardized rubric to evaluate resident competency should be used globally in training programs to assess open globe repair surgical skills. Using a modified Dreyfus model, four different levels of training competency were defined allowing a non-biased, objective, numerical and simple assessment.

Development of a 3D Motion Tracking System for the Analysis of Skills in Microsurgery

Microsurgical skills of trainee surgeons have been subjectively evaluated due to the lack of technological tools. The objective of this investigation is to present the construct validity of the mitracks3D, which is a system designed to help in the objective evaluation of microsurgery trainees. To achieve this, a stereoscopic vision system records the 3D motion of two tweezers manipulated by surgeons during microsurgery training. Using motion analysis parameters (MAPs), quantitative information about their microsurgical skills and performance is obtained. For validation, 14 participants were enrolled and divided into two groups: expert microsurgeons (3 female, 2 male) and trainee surgeons (4 female, 5 male). The 3D motion tracking was acquired while the surgeons practiced with two training models: transferring objects and suture. Twelve MAPs were used to objective assessment the skill levels of each participant. Subsequent, statistical analysis was computed to compare the scores of both groups. Validation results showed statistically significant differences in 8 of the 12 MAPs and in 10 of the 12 MAPs using the transferring and the suturing models, respectively. The capability of mitracks3D to differentiate the performance of microsurgeons by analyzing their movements was shown. The mitracks3D system was successfully validated. With this system was possible to differentiate the psychomotor microsurgical skills between the two groups of surgeons. The mitracks3D system is a suitable device for the evaluation of microsurgical skills in a variety of surgical specialties that require it during the training of their residents.