Development and validation of a simulation-based assessment of operative competence for higher specialist trainees in general surgery

BACKGROUND

Simulation is increasingly being explored as an assessment modality. This study sought to develop and collate validity evidence for a novel simulation-based assessment of operative competence. We describe the approach to assessment design, development, pilot testing, and validity investigation.

METHODS

Eight procedural stations were generated using both virtual reality and bio-hybrid models. Content was identified from a previously conducted Delphi consensus study of trainers. Trainee performance was scored using an equally weighted Objective Structured Assessment of Technical Skills (OSATS) tool and a modified Procedure-Based Assessment (PBA) tool. Validity evidence was analyzed in accordance with Messick's validity framework. Both 'junior' (ST2-ST4) and 'senior' trainees (ST 5-ST8) were included to allow for comparative analysis.

RESULTS

Thirteen trainees were assessed by ten assessors across eight stations. Inter-station reliability was high (α = 0.81), and inter-rater reliability was acceptable (inter-class correlation coefficient 0.77). A significant difference in mean station score was observed between junior and senior trainees (44.82 vs 58.18, p = .004), while overall mean scores were moderately correlated with increasing training year (rs = .74, p = .004, Kendall's tau-b .57, p = 0.009). A pass-fail score generated using borderline regression methodology resulted in all 'senior' trainees passing and 4/6 of junior trainees failing the assessment.

CONCLUSION

This study reports validity evidence for a novel simulation-based assessment, designed to assess the operative competence of higher specialist trainees in general surgery.

Generating a Prioritized List of Operative Procedures for Simulation-based Assessment of General Surgery Trainees Through Consensus

OBJECTIVE

To develop appropriate content for high-stakes simulation-based assessments of operative competence in general surgery training through consensus.

BACKGROUND

Valid methods of summative operative competence assessment are required by competency-based training programs in surgery.

METHOD

An online Delphi consensus study was conducted. Procedures were derived from the competency expectations outlined by the Joint Committee on Surgical Training Curriculum 2021, and subsequent brainstorming. Procedures were rated according to their perceived importance, perceived procedural risk, how frequently they are performed, and simualtion feasibility by a purposive sample of 30 surgical trainers and a 5-person steering group. A modified Copenhagen Academy for Medical Education and Simulation Needs Assessment Formula was applied to the generated data to produce ranked procedural lists, which were returned to participants for re-prioritization.

RESULTS

Prioritized lists were generated for simulation-based operative competence assessments at 2 key stages of training; the end of 'phase 2' prior to the development of a sub-specialty interest, and the end of 'phase 3', that is, end-of-training certification. A total of 21 and 16 procedures were deemed suitable for assessments at each of these stages, respectively.

CONCLUSIONS

This study describes a national needs assessment approach to content generation for simulation-based assessments of operative competence in general surgery using Delphi consensus methodology. The prioritized procedural lists generated by this study can be used to further develop operative skill assessments for use in high-stakes scenarios, such as trainee progression, entrustment, and end-of-training certification, before subsequent validity testing.

Gaps in plastic surgery training: A comparative literature review of assessment tools in plastic surgery and general surgery

Assessment tools for grading technical and nontechnical skills, such as operative technique and professionalism, are well established in general surgery. Less is known regarding the application of these tools in plastic surgery training. This study is a comparative review of the most prevalent assessment tools and rubrics utilized in general and plastic surgery. Two parallel systematic reviews of the literature utilizing PubMed and Cochrane were conducted for articles published between 1990 and 2022. Searches used Boolean operators specific to assessment tools in general and plastic surgery. Fourteen studies met the inclusion criteria for general surgery assessment tools, and 21 studies were included for plastic surgery assessment tools. Seven studies (50%) evaluated technical skills in general surgery, whereas 15 studies (71%) assessed technical skills in plastic surgery with commonality found in the evaluation of principles, such as tissue and instrument handling and operative flow. Task-specific evaluation tools were described for both general and plastic surgeries. Five studies evaluated nontechnical skills, such as communication and leadership in general surgery, whereas no plastic surgery studies solely examined nontechnical assessment tools. Our literature review demonstrates that standardized skill assessments in plastic surgery are lacking compared with those available in general surgery. Plastic surgery programs should consider implementing competency-based assessment tools in surgical coaching and training for technical and nontechnical skills. More research is necessary in plastic surgery to optimize the evaluation of nontechnical skills.

Assessment of laparoscopic intracorporeal intestinal anastomosis training using simulation-based 3D printed models: exploring surgical performance and learning curves

BACKGROUND AND AIMS

Intestinal anastomosis is a clinical procedure widely used to reconstruct the digestive tract, but authentic laparoscopic intracorporeal intestinal anastomosis (LIIA) models are lacking. However, three-dimensional (3D) printing can enable authentic and reusable models. In this paper, a novel cost-effective 3D-printing training model of LIIA is designed and the authenticity and validity of the model are tested.

METHODS

A fused deposition modeling 3D printing and an assembled lab model were built to test LIIA. Fifteen surgeons were required to perform LIIA, and their operation score and time were recorded and analyzed. Five experts were invited to assess the face and content validity of the models. A study was also performed to further evaluate and validate the learning curve of surgeons.

RESULTS

The difference in modified anastomosis objective structured assessment of technical skills (MAOSATS) scores between the expert, intermediate, and novice groups were significant (64.1±1.8: 48.5±1.7: 29.5±3.1, P <0.001). In addition, the operation time of the procedure was statistically different for all three groups (21.5±1.9: 30.6±2.8:70.7±4.0, P<0.001 ). The five experts rated the face and content validity of the model very highly, with the median being four out of five. Surgeons who underwent repeated training programs showed improved surgical performance. After eight training sessions, the novices' performance was similar to that of the average level of untrained intermediates, while the operation scores of the intermediates were close to that of the average level of experts.

CONCLUSIONS

In this study, it is found that the LIIA model exhibits excellent face, content, and construct validity. Repeated simulation training of the LIIA training program improved the surgeon's operative performance, so the model is considered one of the most effective methods for LIIA training and assessment of surgical quality in the future and for reducing healthcare costs.

Perspectives on simulation-based assessment of operative skill in surgical training

INTRODUCTION

The perspectives of the wider surgical community toward simulation-based assessment (SBA) in training is a gap in the literature. This study aims to explore the factors associated with the acceptable use of SBA in surgical training, through the perceptions and experiences of a broad range of stakeholder representatives, building on findings from a review of the published literature.

MATERIALS AND METHODS

Ten semi-structured interviews were conducted, using a sequential transformative qualitative methods approach, with representatives from identified key stakeholder groups; executive management, risk management, a practicing surgeon, an anaesthesiologist, a theatre-nursing representative, a representative from simulation industry, a patient, a medical student, a junior surgical trainee, and a senior surgical trainee. Interview transcripts underwent reflexive thematic analysis using an inductive and constructivist framework (NVIVO software, NVIVO 12, QSR International).

RESULTS

Four themes emerged: the 'need' for SBA, the concept of a 'minimum standard', the 'optimum design' of an SBA framework, and 'fairness'. SBA is a potential solution to challenges in the current training environment. It emerged that it should not replace trainer judgement, but could ensure that trainees meet a minimum operative competency standard. SBA should be used to identify underperforming trainees early in training to provide targeted remediation. The application of SBA in high-stakes settings such as trainee selection, autonomy granting, and end-of training certification has perceived benefits over current assessment methods.

CONCLUSIONS

This study builds on findings from prior research to explore factors regarding the acceptable use of simulation as an assessment method in surgical training, including perspectives from a broad range of stakeholder representatives. Findings can inform the development of simulation-based assessment curricula in surgical training.

Working Towards Competence: A Novel Application of Borderline Regression to a Task-Specific Checklist for Technical Skills in Novices

OBJECTIVE

Demonstrated competence through frequent assessment is an expected goal for progressive development in competency-based medical education curricula. The Objective Structured Assessment of Technical Skill (OSATS) is considered a valid method of formative assessment, but in few instances have standards been set for determining competence. The present study used borderline regression methods to examine standard setting of performance on a complex technical task with novices assessed using an OSATS checklist.

METHODS

This was a single institution prospective single arm experimental design study. Participants were 58 non-medical undergraduate students with no previous surgical experience, who observed a computer-based training module on end-to-side vascular anastomosis. Subsequently, participants were provided two 20-minute training sessions, two weeks apart where they received expert feedback whilst performing the task on a low-fidelity model. After each training session, participants completed the task unaided. Sessions were recorded and assessed using an OSATS checklist retrospectively by experts.

RESULTS

Paired t-test analyses indicate that for both the checklist total score (t(52) = 8.05, p < 0.001) and the global rating score (t(53) = 8.15, p < 0.001), individuals performed significantly better in Phase 2. Borderline regression analyses indicated that in Phase 1 (R² = .60) and Phase 2 (R² = .75), the OSATS checklist could adequately capture variation in performance in novices. Further, the checklist could reliably classify novices at three of the five global rating performance levels. Pass rates determined by regression equations improved from Phase 1 to Phase 2 on all global rating levels.

CONCLUSIONS

With the increasing focus on competency-based medical education, it is imperative that training programs have the capacity to accurately assess outcomes and set minimum performance standards. Borderline regression methods can accurately differentiate novice learners of varying performance levels before and after training on a complex technical skill task using an OSATS checklist.

Value of Standardized Testing in Surgical Training

Standardized testing remains a cornerstone of assessment in surgical education. Summative standardized tests make up a bulk of the certification requirements that encompasses demonstration of efficient, safe application of clinically relevant surgical knowledge and skills. Formative standardized tests serve similar role to guide teaching endeavors for the programs and comparison of individual trainees on a national level. Ongoing rigorous psychometric evaluations of the standardized tests ensure reliability and validity; however, standardized tests are not without their limitations and biases.

Priorities for peri-operative research in Africa

Deaths following surgery are the third largest contributor to deaths globally, and in Africa are twice the global average. There is a need for a peri-operative research agenda to ensure co-ordinated, collaborative research efforts across Africa in order to decrease peri-operative mortality. The objective was to determine the top 10 research priorities for peri-operative research in Africa. A Delphi technique was used to establish consensus on the top research priorities. The top 10 research priorities identified were (1) Develop training standards for peri-operative healthcare providers (surgical, anaesthesia and nursing) in Africa; (2) Develop minimum provision of care standards for peri-operative healthcare providers (surgical, anaesthesia and nursing) in Africa; (3) Early identification and management of mothers at risk from peripartum haemorrhage in the peri-operative period; (4) The role of communication and teamwork between surgical, anaesthetic, nursing and other teams involved in peri-operative care; (5) A facility audit/African World Health Organization situational analysis tool audit to assess emergency and essential surgical care, which includes anaesthetic equipment available and level of training and knowledge of peri-operative healthcare providers (surgeons, anaesthetists and nurses); (6) Establishing evidence-based practice guidelines for peri-operative physicians in Africa; (7) Economic analysis of strategies to finance access to surgery in Africa; (8) Establishment of a minimum dataset surgical registry; (9) A quality improvement programme to improve implementation of the surgical safety checklist; and (10) Peri-operative outcomes associated with emergency surgery. These peri-operative research priorities provide the structure for an intermediate-term research agenda to improve peri-operative outcomes across Africa.