Technical skills assessment toolbox: a review using the unitary framework of validity

Evaluation of a high-fidelity cleft alveolar bone graft simulator

INTRODUCTION

Cleft alveolar bone graft surgery is technically challenging to perform as well as difficult to learn and teach. A high-fidelity cleft alveolar bone graft simulator was previously developed. However, further evaluation of the simulator is necessary to assess its efficacy.

METHODS

Two cleft simulation workshops were conducted in which participants were led through a simulated cleft alveolar bone graft. The first simulation workshop involved six plastic surgery trainees. The second workshop involved 43 practicing cleft surgeons. The participants were provided with a Likert-type questionnaire assessing the simulators' features, realism, and value as a training tool. The change in self-reported confidence was assessed by providing each participant with a pre- and post-simulation confidence questionnaire.

RESULTS

There was overall agreement in the realism of the simulators' features (average score of 4.67 and 3.80 out of 5 for the trainees and surgeons, respectively). There was overall agreement to strong agreement in the simulators value as a training tool (average score of 5 and 4.43 out of 5 for the trainees and surgeons, respectively). The self-reported confidence increased for all questionnaire items for both the trainees and surgeons. This was significant (p < 0.05) for five out of eight and all questionnaire items for the trainees and surgeons, respectively. The magnitude of the confidence increase was generally greater for less experienced participants.

CONCLUSION

The cleft alveolar bone graft simulator was found to be realistic and valuable as a training platform. Use of the simulator improved self-reported confidence in cleft alveolar bone graft surgery.

Objective assessment tools in laparoscopic or robotic-assisted gynecological surgery: A systematic review

INTRODUCTION

There is a growing emphasis on proficiency-based progression within surgical training. To enable this, clearly defined metrics for those newly acquired surgical skills are needed. These can be formulated in objective assessment tools. The aim of the present study was to systematically review the literature reporting on available tools for objective assessment of minimally invasive gynecological surgery (simulated) performance and evaluate their reliability and validity.

MATERIAL AND METHODS

A systematic search (1989-2022) was conducted in MEDLINE, Embase, PubMed, Web of Science in accordance with PRISMA. The trial was registered with the Prospective Register of Systematic Reviews (PROSPERO) ID: CRD42022376552. Randomized controlled trials, prospective comparative studies, prospective single-group (with pre- and post-training assessment) or consensus studies that reported on the development, validation or usage of assessment tools of surgical performance in minimally invasive gynecological surgery, were included. Three independent assessors assessed study setting and validity evidence according to a contemporary framework of validity, which was adapted from Messick's validity framework. Methodological quality of included studies was assessed using the modified medical education research study quality instrument (MERSQI) checklist. Heterogeneity in data reporting on types of tools, data collection, study design, definition of expertise (novice vs. experts) and statistical values prevented a meaningful meta-analysis.

RESULTS

A total of 19 746 titles and abstracts were screened of which 72 articles met the inclusion criteria. A total of 37 different assessment tools were identified of which 13 represented manual global assessment tools, 13 manual procedure-specific assessment tools and 11 automated performance metrices. Only two tools showed substantive evidence of validity. Reliability and validity per tool were provided. No assessment tools showed direct correlation between tool scores and patient related outcomes.

CONCLUSIONS

Existing objective assessment tools lack evidence on predicting patient outcomes and suffer from limitations in transferability outside of the research environment, particularly for automated performance metrics. Future research should prioritize filling these gaps while integrating advanced technologies like kinematic data and AI for robust, objective surgical skill assessment within gynecological advanced surgical training programs.

Training and assessment for colorectal surgery and appendicectomy- a systematic review

AIM

There is currently an increased focus on competency-based training, in which training and assessment play a crucial role. The aim of this systematic review is to create an overview of hands-on training methods and assessment tools for appendicectomy and colon and rectal surgery procedures using either an open, laparoscopic or robot-assisted approach.

METHOD

A systematic review of Medline, Embase, Cochrane and Scopus databases was conducted following the PRISMA guidelines. We conducted the last search on 9 March 2023. All published papers describing hands-on training, evaluation of performance data and development of assessment tools were eligible. The quality of studies and the validity evidence of assessment tools are reported.

RESULTS

Fifty-one studies were identified. Laparoscopic assessment tools are abundant, but the literature still lacks good-quality assessment tools for open appendicectomy, robotic colectomy and open rectal surgery. Overall, there is a lack of discussion regarding the establishment of pass/fail standards and the consequences of assessment. Virtual reality simulation is used more for appendicectomy than colorectal procedures. Only a few of the studies investigating training were of acceptable quality. There is a need for high-quality studies in open and robotic-assisted colon surgery and all approaches to rectal surgery.

CONCLUSION

This review provides an overview of current training methods and assessment tools and identifies where more research is needed based on the quality of the studies and the current validity evidence.

Surgical skills training for practicing surgeons founded on established educational theories and frameworks

Following completion of structured training in residency or fellowship programs, surgeons need to acquire new skills throughout their careers to address changes in their surgical practices, and to perform new procedures and use new technologies as these are developed. The opportunities available to surgeons to acquire the new skills, safely introduce these skills into surgical practice, and then maintain the skills, vary greatly and may not fully support a surgeon's needs. The article shares background information relating to the vexing problems, and describes established educational theories and frameworks as well as specific frameworks relating to surgical skills training and verification that may be used in designing and implementing comprehensive and impactful surgical skills training programs aimed at practicing surgeons. A number of practical strategies based on these theories and frameworks are highlighted. National professional organizations and academic medical centers need to play a critical role in pursuing such efforts, which will remain pivotal in supporting the professional work of surgeons and in providing optimal surgical care well into the future.

Appraisal of surgical outcomes and oncological efficiency of intraoperative adverse events in robotic radical gastrectomy for gastric cancer

BACKGROUND

Surgical quality control is a crucial determinant of evaluating the tumor efficacy.

OBJECTIVE

To assess the ClassIntra grade for quality control and oncological outcomes of robotic radical surgery for gastric cancer (GC).

METHODS

Data of patients undergoing robotic radical surgery for GC at a high-volume center were retrospectively analyzed. Patients were categorized into two groups, the intraoperative adverse event (iAE) group and the non-iAE group, based on the occurrence of intraoperative adverse events. The iAEs were further classified into five sublevels (ranging from I to V according to severity) based on the ClassIntra grade. Surgical performance was assessed using the Objective Structured Assessment of Technical Skill (OSATS) and the General Error Reporting Tool.

RESULTS

This study included 366 patients (iAE group: n = 72 [19.7%] and non-iAE group: n = 294 [80.3%]). The proportion of ClassIntra grade II patients was the highest in the iAE group (54.2%). In total and distal gastrectomies, iAEs occurred most frequently in the suprapancreatic area (50.0% and 54.8%, respectively). In total gastrectomy, grade IV iAEs were most common during lymph node dissection in the splenic hilum area (once for bleeding [grade IV] and once for injury [grade IV]). The overall survival (OS) and disease-free survival of the non-iAE group were significantly better than those of the iAE group (Log rank P < 0.001). Uni- and multi-variate analyses showed that iAEs were key prognostic indicators, independent of tumor stage and adjuvant chemotherapy (P < 0.001).

CONCLUSION

iAEs in patients who underwent robotic radical gastrectomy significantly correlated with the occurrence of postoperative complications and a poor long-term prognosis. Therefore, utilization and inclusion of ClassIntra grading as a crucial surgical quality control and prognostic indicator in the routine surgical quality evaluation system are recommended.

Development and Testing of Step, Error, and Event Frameworks to Evaluate Technical Performance in Peripheral Endovascular Interventions

OBJECTIVE

Tools for endovascular performance assessment are necessary in competency based education. This study aimed to develop and test a detailed analysis tool to assess steps, errors, and events in peripheral endovascular interventions (PVI).

METHODS

A modified Delphi consensus was used to identify steps, errors, and events in iliac-femoral-popliteal endovascular interventions. International experts in vascular surgery, interventional radiology, cardiology, and angiology were identified, based on their scientific track record. In an initial open ended survey round, experts volunteered a comprehensive list of steps, errors, and events. The items were then rated on a five point Likert scale until consensus was reached with a pre-defined threshold (Cronbach's alpha > 0.7) and > 70% expert agreement. An experienced endovascular surgeon applied the finalised frameworks on 10 previously videorecorded elective PVI cases.

RESULTS

The expert consensus panel was formed by 28 of 98 invited proceduralists, consisting of three angiologists, seven interventional radiologists, five cardiologists, and 13 vascular surgeons, with 29% from North America and 71% from Europe. The Delphi process was completed after three rounds (Cronbach's alpha; αsteps = 0.79; αerrors = 0.90; αevents = 0.90), with 15, 26, and 18 items included in the final step (73 - 100% agreement), error (73 - 100% agreement), and event (73 - 100% agreement) frameworks, respectively. The median rating time per case was 4.3 hours (interquartile range [IQR] 3.2, 5 hours). A median of 55 steps (IQR 40, 67), 27 errors (IQR 21, 49), and two events (IQR 1, 6) were identified per case.

CONCLUSION

An evaluation tool for the procedural steps, errors, and events in iliac-femoral-popliteal endovascular procedures was developed through a modified Delphi consensus and applied to recorded intra-operative data to identify hazardous steps, common errors, and events. Procedural mastery may be promoted by using the frameworks to provide endovascular proceduralists with detailed technical performance feedback.

Simulating mitral repair: lessons learned

PURPOSE OF REVIEW

With the growing complexity of cardiac surgical cases, increased focus on patient safety, and minimally invasive techniques, simulation-based training has experienced a renaissance. This review highlights important elements of simulation-based training, focusing specifically on available simulators for mitral valve repair and the uses for simulation.

RECENT FINDINGS

Referring to simulators as being high or low fidelity is oversimplified. Fidelity is a multifactorial concept, and for surgical task trainers, structural and functional fidelity should be discussed. For mitral valve repair, there are a spectrum of simulators, including tissue-based models, bench-top models, and hybrid models. All these simulator modalities serve a role in training if they align with predetermined objectives. There have been advancements in mitral valve repair simulation, notably patient-specific 3D printed silicone replicas of disease.

SUMMARY

There is evidence to support that simulation improves performance in the simulated environment, but future investigation should look to determine whether simulation improves performance in the clinical setting and ultimately patient outcomes.

Development and preliminary validation of a new task-based objective procedure-specific assessment of inguinal hernia repair procedural safety

BACKGROUND

Surgical videos coupled with structured assessments enable surgical training programs to provide independent competency evaluations and align with the American Board of Surgery's entrustable professional activities initiative. Existing assessment instruments for minimally invasive inguinal hernia repair (IHR) have limitations with regards to reliability, validity, and usability. A cross-sectional study of six surgeons using a novel objective, procedure-specific, 8-item competency assessment for minimally invasive inguinal hernia repair (IHR-OPSA) was performed to assess inter-rater reliability using a "safe" vs. "unsafe" scoring rubric.

METHODS

The IHR-OPSA was developed by three expert IHR surgeons, field tested with five IHR surgeons, and revised based upon feedback. The final instrument included: (1) incision/port placement; (2) dissection of peritoneal flap (TAPP) or dissection of peritoneal flap (TEP); (3) exposure; (4) reducing the sac; (5) full dissection of the myopectineal orifice; (6) mesh insertion; (7) mesh fixation; and (8) operation flow. The IHR-OPSA was applied by six expert IHR surgeons to 20 IHR surgical videos selected to include a spectrum of hernia procedures (15 laparoscopic, 5 robotic), anatomy (14 indirect, 5 direct, 1 femoral), and Global Case Difficulty (easy, average, hard). Inter-rater reliability was assessed against Gwet's AC2.

RESULTS

The IHR-OPSA inter-rater reliability was good to excellent, ranging from 0.65 to 0.97 across the eight items. Assessments of robotic procedures had higher reliability with near perfect agreement for 7 of 8 items. In general, assessments of easier cases had higher levels of agreement than harder cases.

CONCLUSIONS

A novel 8-item minimally invasive IHR assessment tool was developed and tested for inter-rater reliability using a "safe" vs. "unsafe" rating system with promising results. To promote instrument validity the IHR-OPSA was designed and evaluated within the context of intended use with iterative engagement with experts and testing of constructs against real-world operative videos.

Development and Evaluation of a High-Fidelity Rhinoplasty Simulator

SUMMARY

Rhinoplasty is a challenging procedure with a steep learning curve. Surgical simulators provide a safe platform to gain hands-on experience without compromising patient outcomes. Therefore, rhinoplasty is an ideal procedure to benefit from an effective surgical simulator. A high-fidelity rhinoplasty simulator was developed using three-dimensional computer modeling, three-dimensional printing, and polymer techniques. The simulator was tested by six surgeons with experience in rhinoplasty to assess realism, anatomic accuracy, and value as a training tool. The surgeons performed common rhinoplasty techniques and were provided a Likert-type questionnaire assessing the anatomic features of the simulator. A variety of surgical techniques were performed successfully using the simulator, including open and closed approaches. Bony techniques performed included endonasal osteotomies and rasping. Submucous resection with harvest of septal cartilage, cephalic trim, and tip suturing, as well as grafting techniques including alar rim, columellar strut, spreader, and shield grafts, were performed successfully. Overall, there was agreement on the simulator's anatomic accuracy of bony and soft-tissue features. There was strong agreement on the simulator's overall realism and value as a training tool. The simulator provides a high-fidelity, comprehensive training platform to learn rhinoplasty techniques to augment real operating experience without compromising patient outcomes.

Bilateral Cleft lip Simulation

OBJECTIVE

To evaluate the features, anatomic accuracy, and educational value of a high-fidelity bilateral cleft lip simulator.

DESIGN

Evaluation of the simulator by expert cleft surgeons after performing a simulated bilateral cleft lip repair.

SETTING

The simulator was evaluated by the surgeons during the Latin American Craniofacial Association meeting.

PARTICIPANTS

Eleven experienced cleft surgeons evaluated the simulator. The cleft surgeons were selected based on their availability during the meeting.

INTERVENTIONS

The participants performed a simulated bilateral cleft lip repair. They were each provided with a questionnaire assessing the simulator's features, realism and value as a training tool.

MAIN OUTCOME MEASURE (S)

The main outcome measure are the scores obtained from a Likert-type questionnaire assessing the simulators features, realism and value.

RESULTS

Overall, the surgeons agreed with the simulator's realism and anatomic accuracy (average score of 3.7 out of 5). Overall, the surgeons strongly agreed with the value of the simulator as a training tool (average score of 4.6 out of 5).

CONCLUSIONS

A high-fidelity bilateral cleft lip simulator was developed that is realistic and valuable as a training tool. The simulator provides a comprehensive training platform to gain hands-on experience in bilateral cleft lip repair before operating on real patients.

Validity Evidence for Procedure-specific Competence Assessment Tools in Orthopaedic Surgery: A Scoping Review

INTRODUCTION

Competency-based training requires frequent assessment of residents' skills to determine clinical competence. This study reviews existing literature on procedure-specific competence assessment tools in orthopaedic surgery.

METHODS

A systematic search of eight databases up to May 2023 was conducted. Two reviewers independently assessed validity evidence and educational utility of each assessment tool and evaluated studies' methodological quality.

RESULTS

Database searching identified 2,556 unique studies for title and abstract screening. Full texts of 290 studies were reviewed; 17 studies met the inclusion criteria. Bibliography review identified another five studies, totaling 22 studies examining 24 assessment tools included in the analysis. These tools assessed various orthopaedic surgery procedures within trauma, sports medicine, spine, and upper extremity. Overall validity evidence was low across all studies, and was lowest for consequences and highest for content. Methodological quality of studies was moderate. Educational utility assessment was not explicitly done for most tools.

DISCUSSION

The paucity of current procedure-specific assessment tools in orthopaedic surgery lacks the validity evidence required to be used reliably in high-stake summative assessments. Study strengths include robust methodology and use of an evidence-based validity evidence framework. Poor-quality existing evidence is a limitation and highlights the need for evidence-based tools across more subspecialties.

Assessment of a Novel, Adjustable Task Trainer for Cardiac Surgical Skills

INTRODUCTION

A recent needs assessment in Canadian cardiac surgery programs identified the desire for a coronary artery bypass (CABG) and aortic valve replacement (AVR) simulation model for home practice. We aimed to develop and assess a portable, adjustable task trainer for cardiac surgical skills with high functional task alignment.

METHODS

Intraoperative measurements were taken from patients undergoing elective CABG and AVR (N = 30). Measurements were taken in 3 axes and used to create a chest cavity that resembles the mediastinal constraints of a patient undergoing CABG and AVR. The task trainer is adjustable on the following 3 levels: (1) size of the incision, (2) depth of the chest, and (3) relative position of coronary artery or aortic valve model within the chest. Three groups (novices, intermediates, and experts) of cardiac surgery members evaluated the task trainer for functional task alignment and construct validity.

RESULTS

The CABG and AVR model had high functional task alignment. There was a high satisfaction for both models and all participants would recommend the AVR and CABG model as an educational tool. Performance time significantly differed between the groups for both models (CABG: P = 0.032 and AVR: P = 0.001), as well as number of errors (CABG: P = 0.04 and AVR: P = 0.043).

CONCLUSIONS

Using real patient data, we were able to develop an adjustable task trainer for training principles of CABG and AVR. Our pilot study provides preliminary sources of evidence for validity and future study will look to assess transferability of skill to the operating room.

On the use and abuse of metaphors in assessment

This paper is motivated by a desire to advance assessment in the health professions through encouraging the judicious and productive use of metaphors. Through five specific examples (pixels, driving lesson/test, jury deliberations, signal processing, and assessment as a toolbox), we interrogate how metaphors are being used in assessment to consider what value they add to understanding and implementation of assessment practices. By unpacking these metaphors in action, we probe each metaphor's rationale and function, the gains each metaphor makes, and explore the unintended meanings they may carry. In summarizing common uses of metaphors, we elucidate how there may be both advantages and/or disadvantages. Metaphors can play important roles in simplifying, complexifying, communicating, translating, encouraging reflection, and convincing. They may be powerfully rhetorical, leading to intended consequences, actions, and other pragmatic outcomes. Although metaphors can be extremely helpful, they do not constitute thorough critique, justified evidence or argumentation. We argue that although metaphors have utility, they must be carefully considered if they are to serve assessment needs in intended ways. We should pay attention to how metaphors may be misinterpreted, what they ignore or unintentionally signal, and perhaps mitigate against this with anticipated corrections or nuanced qualifications. Failure to do so may lead to implementing practices that miss underlying and relevant complexities for assessment science and practice. Using metaphors requires careful attention with respect to their role, contributions, benefits and limitations. We highlight the value that comes from critiquing metaphors, and demonstrate the care required to ensure their continued utility.

Validity evidence for observational ERCP competency assessment tools: a systematic review

BACKGROUND : Assessment of competence in endoscopic retrograde cholangiopancreatography (ERCP) is critical for supporting learning and documenting attainment of skill. Validity evidence supporting ERCP observational assessment tools has not been systematically evaluated. METHODS : We conducted a systematic search using electronic databases and hand-searching from inception until August 2021 for studies evaluating observational assessment tools of ERCP performance. We used a unified validity framework to characterize validity evidence from five sources: content, response process, internal structure, relations to other variables, and consequences. Each domain was assigned a score of 0-3 (maximum score 15). We assessed educational utility and methodological quality using the Accreditation Council for Graduate Medical Education framework and the Medical Education Research Quality Instrument, respectively. RESULTS : From 2769 records, we included 17 studies evaluating 7 assessment tools. Five tools were studied for clinical ERCP, one for simulated ERCP, and one for simulated and clinical ERCP. Validity evidence scores ranged from 2 to 12. The Bethesda ERCP Skills Assessment Tool (BESAT), ERCP Direct Observation of Procedural Skills Tool (ERCP DOPS), and The Endoscopic Ultrasound (EUS) and ERCP Skills Assessment Tool (TEESAT) had the strongest validity evidence, with scores of 10, 12, and 11, respectively. Regarding educational utility, most tools were easy to use and interpret, and required minimal additional resources. Overall methodological quality (maximum score 13.5) was strong, with scores ranging from 10 to 12.5. CONCLUSIONS : The BESAT, ERCP DOPS, and TEESAT had strong validity evidence compared with other assessments. Integrating tools into training may help drive learners' development and support competency decision making.

Factors affecting the quality of laparoscopic D2 lymph node dissection for gastric cancer: a cohort study from two randomized controlled trials

BACKGROUND

Laparoscopic D2 lymph node dissection (LND) for gastric cancer has a wide range and high difficulty. In the past, the quality of surgery was often judged by the time of the operation or the amount of blood loss, but the analysis based on surgical video was rarely reported. The purpose of this study was to analyze the relationship between the quality of laparoscopic D2 LND for gastric cancer and postoperative complications.

METHODS

The surgical video and clinicopathological data of 610 patients in two randomized controlled trials in our center from 2013 to 2016 were retrospectively analyzed. Klass-02-QC LND scale and general error score tool were used to quantitatively evaluate the intraoperative performance of D2 LND. Logistic regression was used to analyze the influencing factors of postoperative complications.

RESULTS

The overall incidence of complications (CD classification≥2) was 20.6%; the incidence of surgical complications was 6.9%. According to whether the LND score reached 44, patients were divided into a qualified group (73%) and a not-qualified group (27%). Event score (ES) by quartile was divided into grade 1 (21.7%), grade 2 (26%), grade 3 (28%), and grade 4 (24.3%) from low to high. Univariate logistic regression analysis showed that ES greater than or equal to 3, tumor size greater than or equal to 35 mm, and cTNM >II were independent risk factors for not-qualified LND. Male,tumor size greater than or equal to 35 mm and cTNM >II were independent risk factors for grade 4 ES. Not-qualified LND (OR=1.62, 95% CI: 1.16-3.89, P =0.021), grade 4 ES (OR=3.21, 95% CI: 1.52-3.90, P =0.035), and cTNM >II (OR=1.74, 95% CI: 1.39-7.33, P =0.041) were independent risk factors for postoperative surgical complications.

CONCLUSIONS

The qualification of LND and intraoperative events based on surgical video are the independent influencing factors of postoperative complications of laparoscopic gastric cancer surgery. Specialist training and teaching based on surgical video may help to improve the surgical skills of specialists and improve the postoperative outcome of patients.

Video-based tools for surgical quality assessment of technical skills in laparoscopic procedures: a systematic review

BACKGROUND

Quality of surgery has substantial impact on both short- and long-term clinical outcomes. This stresses the need for objective surgical quality assessment (SQA) for education, clinical practice and research purposes. The aim of this systematic review was to provide a comprehensive overview of all video-based objective SQA tools in laparoscopic procedures and their validity to objectively assess surgical performance.

METHODS

PubMed, Embase.com and Web of Science were systematically searched by two reviewers to identify all studies focusing on video-based SQA tools of technical skills in laparoscopic surgery performed in a clinical setting. Evidence on validity was evaluated using a modified validation scoring system.

RESULTS

Fifty-five studies with a total of 41 video-based SQA tools were identified. These tools were used in 9 different fields of laparoscopic surgery and were divided into 4 categories: the global assessment scale (GAS), the error-based assessment scale (EBAS), the procedure-specific assessment tool (PSAT) and artificial intelligence (AI). The number of studies focusing on these four categories were 21, 6, 31 and 3, respectively. Twelve studies validated the SQA tool with clinical outcomes. In 11 of those studies, a positive association between surgical quality and clinical outcomes was found.

CONCLUSION

This systematic review included a total of 41 unique video-based SQA tools to assess surgical technical skills in various domains of laparoscopic surgery. This study suggests that validated SQA tools enable objective assessment of surgical performance with relevance for clinical outcomes, which can be used for training, research and quality improvement programs.

The American College of Surgeons Entering Resident Readiness Assessment (ACS ERRA): Validity Evidence From a Three-Year National Study

OBJECTIVE

To gather validity evidence supporting the use and interpretation of scores from the American College of Surgeons Entering Resident Readiness Assessment (ACS ERRA) Program.

SUMMARY AND BACKGROUND DATA

ACS ERRA is an online formative assessment program developed to assess entering surgery residents' ability to make critical clinical decisions, and includes 12 clinical areas and 20 topics identified by a national panel of surgeon educators and residency program directors.

METHODS

Data from 3 national testing administrations of ACS ERRA (2018-2020) were used to gather validity evidence regarding content, response process, internal structure (reliability), relations to other variables, and consequences.

RESULTS

Over the 3 administrations, 1975 surgery residents participated from 125 distinct residency programs. Overall scores [Mean = 64% (SD = 7%)] remained consistent across the 3 years ( P = 0.670). There were no significant differences among resident characteristics (gender, age, international medical graduate status). The mean case discrimination index was 0.54 [SD = 0.15]. Kappa inter-rater reliability for scoring was 0.87; the overall test score reliability (G-coefficient) was 0.86 (Ф-coefficient = 0.83). Residents who completed residency readiness programs had higher ACS ERRA scores (66% versus 63%, Cohen's d = 0.23, P < 0.001). On average, 15% of decisions made (21/140 per test) involved potentially harmful actions. Variability in scores from graduating medical schools (7%) carried over twice as much weight than from matched residency programs (3%).

CONCLUSIONS

ACS ERRA scores provide valuable information to entering surgery residents and surgery program directors to aid in development of individual and group learning plans.

Serious gaming and virtual reality in the multimodal training of laparoscopic inguinal hernia repair: a randomized crossover study

BACKGROUND

The aim of this study was to assess the transferability of surgical skills for the laparoscopic hernia module between the serious game Touch Surgery™ (TS) and the virtual reality (VR) trainer Lap Mentor™. Furthermore, this study aimed to collect validity evidence and to discuss "sources of validity evidence" for the findings using the laparoscopic inguinal hernia module on TS.

METHODS

In a randomized crossover study, medical students (n = 40) in their clinical years performed laparoscopic inguinal hernia modules on TS and the VR trainer. TS group started with "Laparoscopic Inguinal Hernia Module" on TS (phase 1: Preparation, phase 2: Port Placement and Hernia Repair), performed the module first in training, then in test mode until proficiency was reached. VR group started with "Inguinal Hernia Module" on the VR trainer (task 1: Anatomy Identification, task 2: Incision and Dissection) and also performed the module until proficiency. Once proficiency reached in the first modality, the groups performed the other training modality until reaching proficiency. Primary endpoint was the number of attempts needed to achieve proficiency for each group for each task/phase.

RESULTS

Students starting with TS needed significantly less attempts to reach proficiency for task 1 on the VR trainer than students who started with the VR trainer (TS = 2.7 ± 0.6 vs. VR = 3.2 ± 0.7; p = 0.028). No significant differences for task 2 were observed between groups (TS = 2.3 ± 1.1 vs. VR = 2.1 ± 0.8; p = 0.524). For both phases on TS, no significant skill transfer from the VR trainer to TS was observed. Aspects of validity evidence for the module on TS were collected.

CONCLUSION

The results show that TS brought additional benefit to improve performances on the VR trainer for task 1 but not for task 2. Skill transfer from the VR trainer to TS could not be shown. VR and TS should thus be used in combination with TS first in multimodal training to ensure optimal training conditions.

Development of a procedure-specific tool for skill assessment in left- and right-sided laparoscopic complete mesocolic excision

AIM

To (1) develop an assessment tool for laparoscopic complete mesocolic excision (LCME) and (2) report evidence of its content validity.

METHOD

Assessment statements were revealed through (1) semi-structured expert interviews and (2) consensus by the Delphi method, both involving an expert panel of five LCME surgeons. All experts were interviewed and then asked to rate LCME describing statements from 1 (strongly disagree) to 5 (strongly agree). Responses were returned anonymously to the panel until consensus was reached. Statements were directly included as content in the assessment tool if ≥60% of the experts responded "agree" or "strongly agree" (ratings 4 and 5), with the remaining responses being "neither agree nor disagree" (rating 3). Interclass correlation coefficient (ICC) was calculated for expert agreement evaluation. All included statements were subsequently reformulated as tool items and approved by the experts.

RESULTS

Four Delphi rounds were performed to reach consensus. Disagreement was reported for statements describing instrument handling around pancreas; visualisation of landmarks before inferior mesenteric artery ligation; lymphadenectomy around the inferior mesenteric artery, and division of the terminal ileum and transverse colon. ICC in the last Delphi-round was 0.84. The final tool content included 73 statements, converted to 48 right- and 40 left-sided items for LCME assessment.

CONCLUSION

A procedure-specific, video-based tool, named complete mesocolic excision competency assessment tool (CMECAT), has been developed for LCME skill assessment. In the future, we hope it can facilitate assessment of LCME surgeons, resulting in improved patient outcome after colon cancer surgery.

Validation study of a skill assessment tool for education and outcome prediction of laparoscopic distal gastrectomy

BACKGROUND

The Japanese operative-rating scale for laparoscopic distal gastrectomy (JORS-LDG) was developed through cognitive task analysis together with the Delphi method to measure intraoperative performance during laparoscopic distal gastrectomy. This study aimed to investigate the value of this rating scale as an educational tool and a surgical outcome predictor in laparoscopic distal gastrectomy.

METHODS

The surgical performance of laparoscopic distal gastrectomy was assessed by the first assistant, through self-evaluation in the operating room and by video raters blind to the case. We evaluated inter-rater reliability, internal consistency, and correlations between the JORS-LDG scores and the evaluation methods, patient characteristics, and surgical outcomes.

RESULTS

Fifty-four laparoscopic distal gastrectomy procedures performed by 40 surgeons at 16 institutions were evaluated in the operating room and with video recordings using the proposed rating scale. The video inter-rater reliability was > 0.8. Participating surgeons were divided into the low, intermediate, and high groups based on their total scores. The number of laparoscopic surgeries and laparoscopic gastrectomy procedures performed differed significantly among the groups according to laparoscopic distal gastrectomy skill levels. The low, intermediate, and high groups also differed in terms of median operating times (311, 266, and 229 min, respectively, P < 0.001), intraoperative complication rates (27.8, 11.8, and 0%, respectively, P = 0.01), and postoperative complication rates (22.2, 0, and 0%, respectively, P = 0.002).

CONCLUSIONS

The JORS-LDG is a reliable and valid measure for laparoscopic distal gastrectomy training and could be useful in predicting surgical outcomes.

Impact of Rater Training on Residents Technical Skill Assessments: A Randomized Trial

OBJECTIVE

The ACS/APDS Resident Skills Curriculum's Objective Structured Assessment of Technical Skills (OSATS) consists of task-specific checklists and a global rating scale (GRS) completed by raters. Prior work demonstrated a need for rater training. This study evaluates the impact of a rater-training curriculum on scoring discrimination, consistency, and validity for handsewn bowel anastomosis (HBA) and vascular anastomosis (VA).

DESIGN/ METHODS

A rater training video model was developed, which included a GRS orientation and anchoring performances representing the range of potential scores. Faculty raters were randomized to rater training or no rater training and were asked to score videos of resident HBA/VA. Consensus scores were assigned to each video using a modified Delphi process (Gold Score). Trained and untrained scores were analyzed for discrimination and score spread and compared to the Gold Score for relative agreement.

RESULTS

Eight general and eight vascular surgery faculty were randomized to score 24 HBA/VA videos. Rater training increased rater discrimination and decreased rating scale shrinkage for both VA (mean trained score: 2.83, variance 1.88; mean untrained score: 3.1, variance 1.14, p = 0.007) and HBA (mean trained score: 3.52, variance 1.44; mean untrained score: 3.42, variance 0.96, p = 0.033). On validity analyses, a comparison between each rater group vs Gold Score revealed a moderate training impact for VA, trained κ=0.65 vs untrained κ=0.57 and no impact for HBA, R1 κ = 0.71 vs R2 κ = 0.73.

CONCLUSION

A rater-training curriculum improved raters' ability to differentiate performance levels and use a wider range of the scoring scale. However, despite rater training, there was persistent disagreement between faculty GRS scores with no groups reaching the agreement threshold for formative assessment. If technical skill exams are incorporated into high stakes assessments, consensus ratings via a standard setting process are likely a more valid option than individual faculty ratings.

The Validation of Surgical Simulators: A Technical Report on Current Validation Terminology as a Reference for Future Research

In recent years, surgical trainees have been exposed to a lower volume of operative procedures. In part, this is due to the reduction in working hours and further disruption by the coronavirus disease 2019 pandemic. Much has been done to develop the techniques of surgical skill training outside of the operating theatre. Simulation-based interventions must undergo a process of validation to assess their appropriateness and effectiveness for use in training. The terminology of validation within current literature, however, has not evolved in line with the education community, resulting in varying definitions for the same phrase across domains. This can result in confusion and misinterpretation among researchers and surgeons working within this domain. This technical report describes the "types of validity" definitions used in the traditional framework of surgical simulation literature and the contemporary, unitary framework of validity adopted by educationalist theorists. There is a clear overlap between the traditional "types of validity" and the contemporary, unitary framework. The divergence in the use of those definitions seems, at least partly, influenced by the context of the investigations being conducted. By utilising the contemporary definitions, authors may have struggled to provide the evidence required to justify the use of the multitude of surgical skill simulators developed in the recent past. This report has provided an overview of the current terminology within the validation frameworks and can be used as a reference for future surgical simulation research.

An Adaptive Force Guidance System for Computer-Guided Laparoscopy Training

We present an adaptive force guidance system for laparoscopic surgery skills training. This system consists of self-adjusting fuzzy sliding-mode controllers and switching mode controllers to provide proper force feedback. Using virtual fixtures, the proposed system restricts motions or guides a trainee to navigate a surgical instrument in a 3-D space in a manner that mimics a human instructor who would teach the trainees by holding their hands. The self-adjusting controllers incorporate human factors, such as different force sensitivity and proficiency levels. The proposed system was implemented and evaluated using the computer-assisted surgical trainer (CAST). The effects of the force guidance system are presented based on the experimental test results.

Validity Evidence for Procedure-Specific Competency Assessment Tools in Cardiovascular and Thoracic Surgery: A Scoping Review

OBJECTIVE

Surgical education has shifted from a time-based approach to the achievement and demonstration of procedural competency. High quality, objective assessment instruments are required to support this new approach. This study comprehensively reviewed the literature to identify and evaluate available procedure-specific assessment instruments in cardiothoracic and vascular surgery.

DESIGN

A systematic search of 8 databases identified studies containing procedure-specific operative assessment instruments in cardiothoracic and vascular surgery. Generic global rating scales were excluded, unless modified to be procedure-specific. Two reviewers independently evaluated the validity evidence, methodological rigour and educational utility of each instrument using objective scoring criteria. Validity evidence was evaluated with a scoring tool aligned with the contemporary framework of validity. Methodological rigour was evaluated using the Medical Education Research Study Quality Instrument. Educational utility was evaluated according to the Accreditation Council for Graduate Medical Education (ACGME) framework.

RESULTS

There were 2130 unique studies describing procedure-specific assessment in surgery. Of these, 9 studies evaluating 8 procedure-specific assessment instruments met inclusion criteria for cardiothoracic and vascular surgery. Four instruments were identified in thoracic surgery, 2 in cardiac surgery, and 2 in vascular surgery. Only 1 instrument was designed to evaluate surgeon performance, with the remainder designed to evaluate residents. No single instrument scored the maximum score of 15 for validity evidence. The highest score was 11, with 62.5% (n = 5) of instruments scoring greater than 10. All tools attained high scores in content validity, with minimal evidence generally presented regarding the consequences of assessment using a particular instrument. All but 1 instrument scored greater than 11 out of a maximum 16.5 points for methodological rigour. Very few studies reported on the ACGME domains of educational utility.

CONCLUSIONS

In an era where surgical education is shifting towards the demonstration of procedural competency, objective procedure-specific assessment is critical. This review identified that few procedure-specific assessment instruments in cardiothoracic and vascular surgery exist, emphasizing the need for such instruments to ensure the success of competency-based education models.

Tools for assessing the scalability of innovations in health: a systematic review

BACKGROUND

The last decade has seen growing interest in scaling up of innovations to strengthen healthcare systems. However, the lack of appropriate methods for determining their potential for scale-up is an unfortunate global handicap. Thus, we aimed to review tools proposed for assessing the scalability of innovations in health.

METHODS

We conducted a systematic review following the COSMIN methodology. We included any empirical research which aimed to investigate the creation, validation or interpretability of a scalability assessment tool in health. We searched Embase, MEDLINE, CINAHL, Web of Science, PsycINFO, Cochrane Library and ERIC from their inception to 20 March 2019. We also searched relevant websites, screened the reference lists of relevant reports and consulted experts in the field. Two reviewers independently selected and extracted eligible reports and assessed the methodological quality of tools. We summarized data using a narrative approach involving thematic syntheses and descriptive statistics.

RESULTS

We identified 31 reports describing 21 tools. Types of tools included criteria (47.6%), scales (33.3%) and checklists (19.0%). Most tools were published from 2010 onwards (90.5%), in open-access sources (85.7%) and funded by governmental or nongovernmental organizations (76.2%). All tools were in English; four were translated into French or Spanish (19.0%). Tool creation involved single (23.8%) or multiple (19.0%) types of stakeholders, or stakeholder involvement was not reported (57.1%). No studies reported involving patients or the public, or reported the sex of tool creators. Tools were created for use in high-income countries (28.6%), low- or middle-income countries (19.0%), or both (9.5%), or for transferring innovations from low- or middle-income countries to high-income countries (4.8%). Healthcare levels included public or population health (47.6%), primary healthcare (33.3%) and home care (4.8%). Most tools provided limited information on content validity (85.7%), and none reported on other measurement properties. The methodological quality of tools was deemed inadequate (61.9%) or doubtful (38.1%).

CONCLUSIONS

We inventoried tools for assessing the scalability of innovations in health. Existing tools are as yet of limited utility for assessing scalability in health. More work needs to be done to establish key psychometric properties of these tools. Trial registration We registered this review with PROSPERO (identifier: CRD42019107095).

Validity Evidence for Procedure-specific Competence Assessment Tools in General Surgery: A Scoping Review

OBJECTIVE

This study aims to provide a focused and detailed assessment of the validity evidence supporting procedure-specific operative assessment tools in general surgery.

SUMMARY OF BACKGROUND DATA

Competency-based assessment tools should be supported by robust validity evidence to be used reliably for evaluation of operative skills. The contemporary framework of validity relies on five sources of evidence: content, response process, internal structure, relation to other variables, and consequences.

METHODS

A systematic search of 8 databases was conducted for studies containing procedure-specific operative assessment tools in general surgery. The validity evidence supporting each tool was assessed and scored in alignment with the contemporary framework of validity. Methodological rigour of studies was assessed with the Medical Education Research Study Quality Instrument. The educational utility of each tool was assessed with the Accreditation Council for Graduate Medical Education framework.

RESULTS

There were 28 studies meeting inclusion criteria and 23 unique tools were assessed. Scores for validity evidence varied widely between tools, ranging from 3 - 14 (maximum 15). Medical Education Research Study Quality Instrument scores assessing the quality of study methodology were also variable (8.5-15.5, maximum 16.5). Direct reporting of educational utility criteria was limited.

CONCLUSIONS

This study has identified a small group of procedure-specific operative assessment tools in general surgery. Many of these tools have limited validity evidence and have not been studied sufficiently to be used reliably in high-stakes summative assessments. As general surgery transitions to competency-based training, a more robust library of operative assessment tools will be required to support resident education and evaluation.

Training and assessment of musculoskeletal ultrasound and injection skills-a systematic review

OBJECTIVES

To examine how residents are trained and assessed in musculoskeletal ultrasound (MSUS), MSUS-guided and landmark-guided joint aspiration and injection. Additionally, to present the available assessment tools and examine their supporting validity evidence.

METHODS

A systematic search of PubMed, Cochrane Library, and Embase was conducted in accordance with the PRISMA guidelines and studies published from January 1, 2000 to May 31, 2021 were included. Two independent reviewers performed the search and data extraction. The studies were evaluated using the Medical Education Research Quality Instrument (MERSQI).

RESULTS

9,884 articles were screened, and 43 were included; 3 were randomized studies, 21 pre- and post-test studies, 16 descriptive studies, and 3 studies developing assessment tools. The studies used various theoretical training modalities e.g. lectures, anatomical quizzes and e-learning. The practical training models varied from mannequins and cadavers to healthy volunteers and patients. The majority of studies used subjective "comfort level" as assessment, others used practical examination and/or theoretical examination. All training programs increased trainees' self-confidence, theoretical knowledge, and/or practical performance, however few used validated assessment tools to measure the effect. Only one study met the MERSQI high methodical quality cut-off score of 14.

CONCLUSION

The included studies were heterogeneous, and most were of poor methodological quality and not based on contemporary educational theories. This review highlights the need for educational studies using validated theoretical and practical assessment tools to ensure optimal MSUS training and assessment in rheumatology.

How can surgical skills in laparoscopic colon surgery be objectively assessed?-a scoping review

Background

In laparoscopic colorectal surgery, higher technical skills have been associated with improved patient outcome. With the growing interest in laparoscopic techniques, pressure on surgeons and certifying bodies is mounting to ensure that operative procedures are performed safely and efficiently. The aim of the present review was to comprehensively identify tools for skill assessment in laparoscopic colon surgery and to assess their validity as reported in the literature.

Methods

A systematic search was conducted in EMBASE and PubMed/MEDLINE in May 2021 to identify studies examining technical skills assessment tools in laparoscopic colon surgery. Available information on validity evidence (content, response process, internal structure, relation to other variables, and consequences) was evaluated for all included tools.

Results

Fourteen assessment tools were identified, of which most were procedure-specific and video-based. Most tools reported moderate validity evidence. Commonly not reported were rater training, assessment correlation with variables other than training level, and validity reproducibility and reliability in external educational settings.

Conclusion

The results of this review show that several tools are available for evaluation of laparoscopic colon cancer surgery, but few authors present substantial validity for tool development and use. As we move towards the implementation of new techniques in laparoscopic colon surgery, it is imperative to establish validity before surgical skill assessment tools can be applied to new procedures and settings. Therefore, future studies ought to examine different aspects of tool validity, especially correlation with other variables, such as patient morbidity and pathological reports, which impact patient survival.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00464-021-08914-z.

Colonoscopy competence assessment tools: a systematic review of validity evidence

BACKGROUND

Assessment tools are essential for endoscopy training, being required to support feedback provision, optimize learner capabilities, and document competence. We aimed to evaluate the strength of validity evidence that supports the available colonoscopy direct observation assessment tools using the unified framework of validity.

METHODS

We systematically searched five databases for studies investigating colonoscopy direct observation assessment tools from inception until 8 April 2020. We extracted data outlining validity evidence (content, response process, internal structure, relations to other variables, and consequences) from the five sources and graded the degree of evidence, with a maximum score of 15. We assessed educational utility using an Accreditation Council for Graduate Medical Education framework and methodological quality using the Medical Education Research Quality Instrument (MERSQI).

RESULTS

From 10 841 records, we identified 27 studies representing 13 assessment tools (10 adult, 2 pediatric, 1 both). All tools assessed technical skills, while 10 each assessed cognitive and integrative skills. Validity evidence scores ranged from 1-15. The Assessment of Competency in Endoscopy (ACE) tool, the Direct Observation of Procedural Skills (DOPS) tool, and the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT) had the strongest validity evidence, with scores of 13, 15, and 14, respectively. Most tools were easy to use and interpret, and required minimal resources. MERSQI scores ranged from 9.5-11.5 (maximum score 14.5).

CONCLUSIONS

The ACE, DOPS, and GiECAT have strong validity evidence compared with other assessments. Future studies should identify barriers to widespread implementation and report on the use of these tools in credentialing examinations.

Current Evidence for Simulation-Based Training and Assessment of Myringotomy and Ventilation Tube Insertion: A Systematic Review

OBJECTIVE

Myringotomy and ventilation tube insertion (MT) is a key procedure in otorhinolaryngology and can be trained using simulation models. We aimed to systematically review the literature on models for simulation-based training and assessment of MT and supporting educational evidence.

DATABASES REVIEWED

PubMed, Embase, Cochrane Library, Web of Science, Directory of Open Access Journals.

METHODS

Inclusion criteria were MT training and/or skills assessment using all types of training modalities and learners. Studies were divided into 1) descriptive and 2) educational interventional/observational in the analysis. For descriptive studies, we provide an overview of available models including materials and cost. Educational studies were appraised using Kirkpatrick's level of educational outcomes, Messick's framework of validity, and a structured quality assessment tool.

RESULTS

Forty-six studies were included consisting of 21 descriptive studies and 25 educational studies. Thirty-one unique physical and three virtual reality simulation models were identified. The studies report moderate to high realism of the different simulators and trainees and educators perceive them beneficial in training MT skills. Overall, simulation-based training is found to reduce procedure time and errors, and increase performance as measured using different assessment tools. None of the studies used a contemporary validity framework and the current educational evidence is limited.

CONCLUSION

Numerous simulation models and assessment tools have been described in the literature but educational evidence and systematic implementation into training curricula is scarce. There is especially a need to establish the effect of simulation-based training of MT in transfer to the operating room and on patient outcomes.

Applications of Extended Reality in Ophthalmology: Systematic Review

BACKGROUND

Virtual reality, augmented reality, and mixed reality make use of a variety of different software and hardware, but they share three main characteristics: immersion, presence, and interaction. The umbrella term for technologies with these characteristics is extended reality. The ability of extended reality to create environments that are otherwise impossible in the real world has practical implications in the medical discipline. In ophthalmology, virtual reality simulators have become increasingly popular as tools for surgical education. Recent developments have also explored diagnostic and therapeutic uses in ophthalmology.

OBJECTIVE

This systematic review aims to identify and investigate the utility of extended reality in ophthalmic education, diagnostics, and therapeutics.

METHODS

A literature search was conducted using PubMed, Embase, and Cochrane Register of Controlled Trials. Publications from January 1, 1956 to April 15, 2020 were included. Inclusion criteria were studies evaluating the use of extended reality in ophthalmic education, diagnostics, and therapeutics. Eligible studies were evaluated using the Oxford Centre for Evidence-Based Medicine levels of evidence. Relevant studies were also evaluated using a validity framework. Findings and relevant data from the studies were extracted, evaluated, and compared to determine the utility of extended reality in ophthalmology.

RESULTS

We identified 12,490 unique records in our literature search; 87 met final eligibility criteria, comprising studies that evaluated the use of extended reality in education (n=54), diagnostics (n=5), and therapeutics (n=28). Of these, 79 studies (91%) achieved evidence levels in the range 2b to 4, indicating poor quality. Only 2 (9%) out of 22 relevant studies addressed all 5 sources of validity evidence. In education, we found that ophthalmic surgical simulators demonstrated efficacy and validity in improving surgical performance and reducing complication rates. Ophthalmoscopy simulators demonstrated efficacy and validity evidence in improving ophthalmoscopy skills in the clinical setting. In diagnostics, studies demonstrated proof-of-concept in presenting ocular imaging data on extended reality platforms and validity in assessing the function of patients with ophthalmic diseases. In therapeutics, heads-up surgical systems had similar complication rates, procedural success rates, and outcomes in comparison with conventional ophthalmic surgery.

CONCLUSIONS

Extended reality has promising areas of application in ophthalmology, but additional high-quality comparative studies are needed to assess their roles among incumbent methods of ophthalmic education, diagnostics, and therapeutics.

Tools for measuring technical skills during gynaecologic surgery: a scoping review

BACKGROUND

Standardised assessment is key to structured surgical training. Currently, there is no consensus on which surgical assessment tool to use in live gynaecologic surgery. The purpose of this review is to identify assessment tools measuring technical skills in gynaecologic surgery and evaluate the measurement characteristics of each tool.

METHOD

We utilized the scoping review methodology and searched PubMed, Medline, Embase and Cochrane. Inclusion criteria were studies that analysed assessment tools in live gynaecologic surgery. Kane's validity argument was applied to evaluate the assessment tools in the included studies.

RESULTS

Eight studies out of the 544 identified fulfilled the inclusion criteria. The assessment tools were categorised as global rating scales, global and procedure rating scales combined, procedure-specific rating scales or as a non-procedure-specific error assessment tool.

CONCLUSION

This scoping review presents the current different tools for observational assessment of technical skills in intraoperative, gynaecologic surgery. This scoping review can serve as a guide for surgical educators who want to apply a scale or a specific tool in surgical assessment.

Isolating steps instead of learners: Use of deliberate practice and validity evidence in coronavirus disease (COVID)-era procedural assessment

Background

In surgical training, assessment tools based on strong validity evidence allow for standardized evaluation despite changing external circumstances. At a large academic institution, surgical interns undergo a multimodal curriculum for central line placement that uses a 31-item binary assessment at the start of each academic year. This study evaluated this practice within increased in-person learning restrictions. We hypothesized that external constraints would not affect resident performance nor assessment due to a robust curriculum and assessment checklist.

Methods

From 2018 to 2020, 81 residents completed central line training and assessment. In 2020, this curriculum was modified to conform to in-person restrictions and social distancing guidelines. Resident score reports were analyzed using multivariate analyses to compare performance, objective scoring parameters, and subjective assessments among “precoronavirus disease” years (2018 and 2019) and 2020.

Results

There were no significant differences in average scores or objective pass rates over 3 years. Significant differences between 2020 and precoronavirus disease years occurred in subjective pass rates and in first-time success for 4 checklist items: patient positioning, draping, sterile ultrasound probe cover placement, and needle positioning before venipuncture.

Conclusion

Modifications to procedural training within current restrictions did not adversely affect residents’ overall performance. However, our data suggest that in 2020, expert trainers may not have ensured learner acquisition of automated procedural steps. Additionally, although 2020 raters could have been influenced by logistical barriers leading to more lenient grading, the assessment tool ensured training and assessment integrity.

Development and validation of a simulator for teaching minimally invasive thoracic surgery in Brazil

Purpose

To develop and validate a chest cavity simulator for teaching video-assited thoracic surgery (VATS).

Methods

The first phase of the study consisted of developing a chest cavity simulator. A quasi-experimental study was performed in the second phase, and 25 surgeons and residents participated in a three-stage pulmonary suture experiment. The videos were recorded and timed. Generalized linear regression models for repeated measures were used to analyze the outcome change over time.

Results

The chest cavity simulator consists of a console simulating the left hemithorax. Among the participants, 96% rated the design, visual aspect, positioning ergonomics, and triangulation of the portals as very good or excellent (face validity). There was a decrease in suturing time in step 1 from 435.7 ± 105 to 355.6 ± 76.8 seconds compared to step 3 (p = 0.001). The evaluation of the simulation effectiveness and performance (content validity) was rated as very good or excellent by 96% ofparticipants. The most experienced surgeon showed significant reduction in procedure time (p = 0.021) (construct validity).

Conclusions

The thoracic cavity simulator is realistic, showing content and construct validity, and can be used in VATS training. The simulation model allowed skill gain in the endoscopic suture.

Validation of a Novel Simulation-Based Test in Robot-Assisted Radical Prostatectomy

Purpose: To investigate validity evidence for a simulator-based test in robot-assisted radical prostatectomy (RARP). Materials and Methods: The test consisted of three modules on the RobotiX Mentor VR-simulator: Bladder Neck Dissection, Neurovascular Bundle Dissection, and Ureterovesical Anastomosis. Validity evidence was investigated by using Messick's framework by including doctors with different RARP experience: novices (who had assisted for RARP), intermediates (robotic surgeons, but not RARP surgeons), or experienced (RARP surgeons). The simulator metrics were analyzed, and Cronbach's alpha and generalizability theory were used to explore reliability. Intergroup comparisons were done with mixed-model, repeated measurement analysis of variance and the correlation between the number of robotic procedures and the mean test score were examined. A pass/fail score was established by using the contrasting groups' method. Results: Ten novices, 11 intermediates, and 6 experienced RARP surgeons were included. Six metrics could discriminate between groups and showed acceptable internal consistency reliability, Cronbach's alpha = 0.49, p < 0.001. Test-retest reliability was 0.75, 0.85, and 0.90 for one, two, and three repetitions of tests, respectively. Six metrics were combined into a simulator score that could discriminate between all three groups, p = 0.002, p < 0.001, and p = 0.029 for novices vs intermediates, novices vs experienced, and intermediates vs experienced, respectively. Total number of robotic operations and the mean score of the three repetitions were significantly correlated, Pearson's r = 0.74, p < 0.001. Conclusion: This study provides validity evidence for a simulator-based test in RARP. We determined a pass/fail level that can be used to ensure competency before proceeding to supervised clinical training.

Performance Assessment for Total Laparoscopic Hysterectomy in the Operating Room: Validity Evidence of a Procedure-specific Rating Scale

STUDY OBJECTIVE

The technical conduct of total laparoscopic hysterectomy (LH) is critical to surgical outcomes. This study explored the validity evidence of an objective scale specific to the assessment of technical skills (H-OSATS) for 7 tasks of an LH with salpingo-oophorectomy procedure performed in the operating room.

DESIGN

Observational cohort study.

SETTING

Two academic hospitals in Marseille and Montpellier, France.

PATIENTS

Three groups of operators (novice, intermediate, and experienced surgeons) were video recorded during their live performances of LH on a simple case. For each group, a dozen unedited videos were obtained for the following tasks: division of the round ligament, division of the infundibulopelvic ligament, creation of the bladder flap, opening of the posterior peritoneum, division of the uterine vessels, colpotomy, and closure of the vault.

INTERVENTIONS

Two qualified raters blindly assessed each video using the H-OSATS rating scale. Inter-rater reliability and test-retest reliability were calculated as measures of internal structure. In a separate round of evaluations, the raters provided a global competent/noncompetent decision for each performance. As a measure of consequential validity, a pass/fail score was set for each task using the contrasting group method.

MEASUREMENTS AND MAIN RESULTS

Three tasks (creation of the bladder flap, colpotomy, and closure of the vault) displayed sound validity evidence: a meaningful total score difference among the 3 groups of experience as well as between the intermediate and experienced surgeons, reliability outcomes of >0.7, and a pass/fail score with a theoretical false-positive rate of <10%.

CONCLUSION

The validity evidence of the H-OSATS rating scale differed for separate evaluations of the 7 tasks. Three tasks (i.e., creation of the bladder flap, colpotomy, and closure of the vault) revealed sound validity evidence, including at the level of the attending surgeon, whereas other tasks were more consistent with low-stakes formative evaluation standards.

Assessing Performance in Simulated Cleft Palate Repair Using a Novel Video Recording Setup

OBJECTIVE

To test the feasibility of implementing a high-fidelity cleft palate simulator during a workshop in Santiago, Chile, using a novel video endoscope to assess technical performance.

DESIGN

Sixteen cleft surgeons from South America participated in a 2-day cleft training workshop. All 16 participants performed a simulated repair, and 13 of them performed a second simulated repair. The repairs were recorded using a low-cost video camera and a newly designed camera mouth retractor attachment. Twenty-nine videos were assessed by 3 cleft surgeons using a previously developed cleft palate objective structured assessment of technical skill (CLOSATS with embedded overall score assessment) and global rating scale. The reliability of the ratings and technical performance in relation to minimum acceptable scores and previous experience was assessed.

RESULTS

The video setup provided acceptable recording quality for the purpose of assessment. Average intraclass correlation coefficient for the CLOSATS, global, and overall performance score was 0.69, 0.75, and 0.82, respectively. None of the novice surgeons passed the CLOSATS and global score for both sessions. One participant in the intermediate group, and 2 participants in the advanced group passed the CLOSATS and global score for both sessions. There were highly experienced participants who failed to pass the CLOSATS and global score for both sessions.

CONCLUSIONS

The cleft palate simulator can be practically implemented with video-recording capability to assess performance in cleft palate repair. This technology may be of assistance in assessing surgical competence in cleft palate repair.

Simulation of advanced cataract surgery - validation of a newly developed test

PURPOSE

To develop and investigate an Eyesi simulator-based test for the more experienced cataract surgeon for evidence of validity.

METHODS

The study was a prospective interventional cohort study and carried out at the Copenhagen Academy for Medical Education and Simulation. The Eyesi Simulator was used for the test which was developed by three expert cataract surgeons. Ten cataract surgeons (>250 surgeries performed) and ten ophthalmic residents performed two repetitions of the test. The test consisted of four modules: Iris Expansion Ring insertion - level 1, Iris Expansion Ring extraction - level 2, Capsulorhexis - level 3 and Anterior Vitrectomy - level 6.

RESULTS

Internal consistency reliability showed Cronbach's alpha of 0.63. Test-retest reliabilities were significant for Iris Expansion Ring extraction - level 2 (p = 0.012) and Capsulorhexis - level 3 (p = 0.018). Differences between the two groups were only significant in both repetitions for the Iris Expansion Ring extraction - level 2 (p < 0.001 and p = 0.041, respectively). Furthermore, we found a statistically significant difference between the mean module scores for novices and the more experienced surgeons for Iris Expansion Ring insertion - level 1 (p = 0.021) and Capsulorhexis - level 3 (p = 0.019) in the first repetition.

CONCLUSION

The investigated modules show evidence of validity within several aspects of Messick's framework. However, the evidence is not strong enough to apply the test for certification purposes of cataract surgeons, but the modules may still be relevant in the training of advanced cataract surgical procedures.

Use of a Low-Cost Portable 3D Virtual Reality Simulator for Psychomotor Skill Training in Minimally Invasive Surgery: Task Metrics and Score Validity

Background

The high cost and low availability of virtual reality simulators in surgical specialty training programs in low- and middle-income countries make it necessary to develop and obtain sources of validity for new models of low-cost portable simulators that enable ubiquitous learning of psychomotor skills in minimally invasive surgery.

Objective

The aim of this study was to obtain validity evidence for relationships to other variables, internal structure, and consequences of testing for the task scores of a new low-cost portable simulator mediated by gestures for learning basic psychomotor skills in minimally invasive surgery. This new simulator is called SIMISGEST-VR (Simulator of Minimally Invasive Surgery mediated by Gestures - Virtual Reality).

Methods

In this prospective observational validity study, the authors looked for multiple sources of evidence (known group construct validity, prior videogaming experience, internal structure, test-retest reliability, and consequences of testing) for the proposed SIMISGEST-VR tasks. Undergraduate students (n=100, reference group), surgical residents (n=20), and experts in minimally invasive surgery (n=28) took part in the study. After answering a demographic questionnaire and watching a video of the tasks to be performed, they individually repeated each task 10 times with each hand. The simulator provided concurrent, immediate, and terminal feedback and obtained the task metrics (time and score). From the reference group, 29 undergraduate students were randomly selected to perform the tasks 6 months later in order to determine test-retest reliability.

Results

Evidence from multiple sources, including strong intrarater reliability and internal consistency, considerable evidence for the hypothesized consequences of testing, and partial confirmation for relations to other variables, supports the validity of the scores and the metrics used to train and teach basic psychomotor skills for minimally invasive surgery via a new low-cost portable simulator that utilizes interaction technology mediated by gestures.

Conclusions

The results obtained provided multiple sources of evidence to validate SIMISGEST-VR tasks aimed at training novices with no prior experience and enabling them to learn basic psychomotor skills for minimally invasive surgery.

The effect of simulation training on resident proficiency in thoracolumbar pedicle screw placement using computer-assisted navigation

OBJECTIVE

Residency work-hour restrictions necessitate efficient, reproducible training. Simulation training for spinal instrumentation placement shows significant benefit to learners' subjective and objective proficiency. Cadaveric laboratories are most effective but have high cost and low availability. The authors' goal was to create a low-cost, efficient, reproducible spinal instrumentation placement simulation curriculum for neurosurgery and orthopedic surgery residents using synthetic models and 3D computer-assisted navigation, assessing subjective and objective proficiency with placement of thoracolumbar pedicle screws.

METHODS

Fifteen neurosurgery and orthopedic surgery residents participated in a standardized curriculum with lecture followed by two separate sessions of thoracolumbar pedicle screw placement in a synthetic spine model utilizing 3D computer-assisted navigation. Data were collected on premodule experience, time and accuracy of screw placement, and both subjective and objective ratings of proficiency.

RESULTS

Fifteen of 15 residents demonstrated improvement in subjective (Physician Performance Diagnostic Inventory Scale [PPDIS]) and 14 in objective (Objective Structured Assessment of Technical Skills [OSATS]) measures of proficiency in navigated screw placement with utilization of this curriculum (p < 0.001 for both), regardless of the number of cases of previous experience using thoracolumbar spinal instrumentation. Fourteen of 15 residents demonstrated decreased time per screw placement from session 1 to session 2 (p = 0.006). There was no significant difference in pedicle screw accuracy between session 1 and session 2.

CONCLUSIONS

A standardized curriculum using synthetic simulation training for navigated thoracolumbar pedicle screw placement results in significantly improved resident subjective and objective proficiency. Development of a nationwide competency curriculum using simulation training for spinal instrumentation placement should be considered for safe, efficient resident training.

Practicing on the Advanced Training in Laparoscopic Suturing Curriculum (ATLAS): Is Mastery Learning in Residency Feasible to Achieve Expert-Level Performance in Laparoscopic Suturing?

BACKGROUND

Mastery learning assumes that given enough time and appropriate instructional strategies, most trainees will be able to achieve proficiency. Expert-level performance requires numerous hours of intensive and focus practice. We aimed to study whether it was possible for surgical trainees to achieve expert-derived proficiency level in laparoscopic suturing using the Advanced Training in Laparoscopic Suturing (ATLAS) curriculum over a short period of time.

STUDY DESIGN

A multicenter IRB approved prospective study included surgery residents and minimally invasive fellows. Participants underwent weekly supervised instruction and assessments of ATLAS skills and self-directed practice between sessions over 12 weeks. Participants were asked to practice until they achieved previously established proficiency benchmarks of expert laparoscopic surgeons.

RESULTS

Fifteen participants, PGY2 to PGY6, from 3 institutions practiced on the ATLAS curriculum. Three participants were able to achieve proficiency on the entire curriculum, with a cumulative practice time varying between 3.4 and 7.6 hours. Individual tasks had varying degrees of difficulty ranging from 85% proficiency on task 1 to 33.3% proficiency for task 6. Using a mixed-method model, the mean cumulative hours of practice to reach the benchmark threshold was estimated for each task and varied from 4.5 to 13.2 hours. The improved performance was associated with higher PGY level and proficiency in FLS.

CONCLUSIONS

This study demonstrates that it is possible for some senior surgical trainees to achieve proficiency in an expert-level laparoscopic suturing curriculum. This study establishes a learning curve for each ATLAS individual task. Some learners may not be able to achieve proficiency on the entire curriculum over a short period of practice. Additional studies are needed to assess how to shorten the learning curve with effective instructional methods such as expert-guided training with immediate feedback.

Development and validation of a transanal endoscopic rectal purse string simulator

BACKGROUND

Rectal purse string placement for transanal mesorectal excision is challenging, and practice is difficult. The objective of this study is to build an endoscopic rectal purse string simulator and provide evidence for the validity of its use.

METHODS

A low-cost transanal endoscopic rectal purse string simulator was created and used to measure the performance of participants. Participants included general surgery residents, fellows, and staff surgeons from several Canadian university-affiliated institutions across Canada. The performance of the rectal purse string placement was measured by the time to completion of the task as well scoring with a modified objective structured assessment of technical skills (OSATS) score and a modified advanced laparoscopic suturing (ALS) score.

RESULTS

Thirty-nine participants were recruited into the study. Participants were split into three groups, based on prior experience with laparoscopic suturing, for the analysis of suturing a rectal purse string on the simulator based on three performance measures. There was a significant difference found in all three measures of performance in the three groups (time to completion p = 0.014, mean blinded OSATS score p = 0.007, mean blinded ALS score p = 0.020). Participants with previous laparoscopic suturing had significantly faster times to completion and higher skills scores when compared to residents (time: 5.1 ± 1.4 min vs 9.0 ± 4.8 min, p = 0.005; OSATS: 19.7 ± 2.8 vs 13.0 ± 5.8, p = 0.00398, ALS: 27.4 ± 4.0 vs 18.9 ± 8.5, p = 0.0151).

CONCLUSIONS

A transanal endoscopic purse string simulator was constructed and preliminary testing has shown variable performance based on prior laparoscopic suturing experience.

Towards Development of an Open Surgery Competency Assessment for Residents (OSCAR) Tool - A Systematic Review of the Literature and Delphi Consensus

OBJECTIVE

Competency-based education has mandated accurate intra-operative assessment tools. We aimed to define consensus-based open surgical skills perceived by experts as critical for assessment.

DESIGN

A mixed-method design was employed: systematic review and e-Delphi methodology.

SETTING

The study was performed at McGill University-affiliated large tertiary academic centers in Montreal, Quebec, Canada.

PARTICIPANTS

Per PRISMA guidelines, a peer-reviewed search strategy was employed. Studies published in English and those describing technical skill assessment of open abdominal surgery were included; subspecialty-specific skills, conference abstracts, academic memoirs were excluded. Most-cited skills were subjected to e-Delphi methodology to identify those deemed essential by experts, based a 3-point Likert scale. Eighteen McGill University-affiliated general surgeons, representing a variety of subspecialties of General Surgery, were invited to answer the questionnaire.

RESULTS

Around 120 of 4285 references were retained for analysis. The 12 most cited skills included suturing, tissue and instrument handling, movement economy, instrument knowledge, knot tying, flow, knowledge of procedure, completion time, dissection technique, knowledge of anatomy and sterile technique; 6 of these achieved high or perfect scores and agreement after 2 rounds of survey: suturing, sterile technique, knot tying, knowledge of anatomy, knowledge of procedure, and tissue handling. Median standard deviation decreased (0.495 to 0.450) from first to second round, indicating improvement in consensus.

CONCLUSION

These results will help develop and validate the OSCAR (objective structured clinical assessment rubric) assessment tool for immediate intra-operative feedback of open technical skills for surgical trainees.

Technical Performance as a Predictor of Clinical Outcomes in Laparoscopic Gastric Cancer Surgery

OBJECTIVE

The purpose of this study was to evaluate the relationship between technical performance and patient outcomes in laparoscopic gastric cancer surgery.

BACKGROUND

Laparoscopic gastrectomy for cancer is an advanced procedure with high rate of postoperative morbidity and mortality. Many variables including patient, disease, and perioperative management factors have been shown to impact postoperative outcomes; however, the role of surgical performance is insufficiently investigated.

METHODS

A retrospective review was performed for all patients who had undergone laparoscopic gastrectomy for cancer at 3 teaching institutions between 2009 and 2015. Patients with available, unedited video-recording of their procedure were included in the study. Video files were rated for technical performance, using Objective Structured Assessments of Technical Skills (OSATS) and Generic Error Rating Tool instruments. The main outcome variable was major short-term complications. The effect of technical performance on patient outcomes was assessed using logistic regression analysis with backward selection strategy.

RESULTS

Sixty-one patients with available video recordings were included in the study. The overall complication rate was 29.5%. The mean Charlson comorbidity index, type of procedure, and the global OSATS score were included in the final predictive model. Lower performance score (OSATS ≤29) remained an independent predictor for major short-term outcomes (odds ratio 6.49), while adjusting for comorbidities and type of procedure.

CONCLUSIONS

Intraoperative technical performance predicts major short-term outcomes in laparoscopic gastrectomy for cancer. Ongoing assessment and enhancement of surgical skills using modern, evidence-based strategies might improve short-term patient outcomes. Future work should focus on developing and studying the effectiveness of such interventions in laparoscopic gastric cancer surgery.

Evaluation methods and impact of simulation-based training in pediatric surgery: a systematic review

PURPOSE

The aim of this study was to identify (1) the type of skill evaluation methods and (2) how the effect of training was evaluated in simulation-based training (SBT) in pediatric surgery.

METHODS

Databases of PubMed, Cochrane Library, and Web of Science were searched for articles published from January 2000 to January 2017. Search concepts of Medical Subject Heading terms were "surgery," "pediatrics," "simulation," and "training, evaluation."

RESULTS

Of 5858 publications identified, 43 were included. Twenty papers described simulators as assessment tools used to evaluate technical skills. Reviewers differentiated between experts and trainees using a scoring system (45%) and/or a checklist (25%). Simulators as training tools were described in 23 papers. While the training's effectiveness was measured using performance assessment scales (52%) and/or surveys (43%), no study investigated the improvement of the clinical outcomes after SBT.

CONCLUSION

Scoring, time, and motion analysis methods were used for the evaluation of basic techniques of laparoscopic skills. Only a few SBT in pediatric surgery have definite goals with clinical effect. Future research needs to demonstrate the educational effect of simulators as assessment or training tools on SBT in pediatric surgery.

How Do Thresholds of Principle and Preference Influence Surgeon Assessments of Learner Performance?

OBJECTIVE

The present study asks whether intraoperative principles are shared among faculty in a single residency program and explores how surgeons' individual thresholds between principles and preferences might influence assessment.

BACKGROUND

Surgical education continues to face significant challenges in the implementation of intraoperative assessment. Competency-based medical education assumes the possibility of a shared standard of competence, but intersurgeon variation is prevalent and, at times, valued in surgical education. Such procedural variation may pose problems for assessment.

METHODS

An entire surgical division (n = 11) was recruited to participate in video-guided interviews. Each surgeon assessed intraoperative performance in 8 video clips from a single laparoscopic radical left nephrectomy performed by a senior learner (>PGY5). Interviews were audio recorded, transcribed, and analyzed using the constant comparative method of grounded theory.

RESULTS

Surgeons' responses revealed 5 shared generic principles: choosing the right plane, knowing what comes next, recognizing normal and abnormal, making safe progress, and handling tools and tissues appropriately. The surgeons, however, disagreed both on whether a particular performance upheld a principle and on how the performance could improve. This variation subsequently shaped their reported assessment of the learner's performance.

CONCLUSIONS

The findings of the present study provide the first empirical evidence to suggest that surgeons' attitudes toward their own procedural variations may be an important influence on the subjectivity of intraoperative assessment in surgical education. Assessment based on intraoperative entrustment may harness such subjectivity for the purpose of implementing competency-based surgical education.

Gathering Validity Evidence for Surgical Simulation: A Systematic Review

OBJECTIVE

To identify current trends in the use of validity frameworks in surgical simulation, to provide an overview of the evidence behind the assessment of technical skills in all surgical specialties, and to present recommendations and guidelines for future validity studies.

SUMMARY OF BACKGROUND DATA

Validity evidence for assessment tools used in the evaluation of surgical performance is of paramount importance to ensure valid and reliable assessment of skills.

METHODS

We systematically reviewed the literature by searching 5 databases (PubMed, EMBASE, Web of Science, PsycINFO, and the Cochrane Library) for studies published from January 1, 2008, to July 10, 2017. We included original studies evaluating simulation-based assessments of health professionals in surgical specialties and extracted data on surgical specialty, simulator modality, participant characteristics, and the validity framework used. Data were synthesized qualitatively.

RESULTS

We identified 498 studies with a total of 18,312 participants. Publications involving validity assessments in surgical simulation more than doubled from 2008 to 2010 (∼30 studies/year) to 2014 to 2016 (∼70 to 90 studies/year). Only 6.6% of the studies used the recommended contemporary validity framework (Messick). The majority of studies used outdated frameworks such as face validity. Significant differences were identified across surgical specialties. The evaluated assessment tools were mostly inanimate or virtual reality simulation models.

CONCLUSION

An increasing number of studies have gathered validity evidence for simulation-based assessments in surgical specialties, but the use of outdated frameworks remains common. To address the current practice, this paper presents guidelines on how to use the contemporary validity framework when designing validity studies.

Simulation platforms to assess laparoscopic suturing skills: a scoping review

BACKGROUND

Laparoscopic suturing (LS) has become a common technique used in a variety of advanced laparoscopic procedures. However, LS is a challenging skill to master, and many trainees may not be competent in performing LS at the end of their training. The purpose of this review is to identify simulation platforms available for assessment of LS skills, and determine the characteristics of the platforms and the LS skills that are targeted.

METHODS

A scoping review was conducted between January 1997 and October 2018 for full-text articles. The search was done in various databases. Only articles written in English or French were included. Additional studies were identified through reference lists. The search terms included "laparoscopic suturing" and "clinical competence."

RESULTS

Sixty-two studies were selected. The majority of the simulation platforms were box trainers with inanimate tissue, and targeted basic suturing and intracorporeal knot-tying techniques. Most of the validation came from internal structure (rater reliability) and relationship to other variables (compare training levels/case experience, and various metrics). Consequences were not addressed in any of the studies.

CONCLUSION

We identified many types of simulation platforms that were used for assessing LS skills, with most being for assessment of basic skills. Platforms assessing the competence of trainees for advanced LS skills were limited. Therefore, future research should focus on development of LS tasks that better reflect the needs of the trainees.

Efficacy of Trauma Surgery Technical Skills Training Courses

OBJECTIVE

Because open surgical skills training for trauma is limited in clinical practice, trauma skills training courses were developed to fill this gap, The aim of this report is to find supporting evidence for efficacy of these courses. The questions addressed are: What courses are available and is there robust evidence of benefit?

DESIGN

We performed a systematic review of the training course literature on open trauma surgery procedural skills courses for surgeons using Kirkpatrick's framework for evaluating complex educational interventions. Courses were identified using Pubmed, Google Scholar and other databases.

SETTING AND PARTICIPANTS

The review was carried out at the University of Maryland, Baltimore with input from civilian and military trauma surgeons, all of whom have taught and/or developed trauma skills courses.

RESULTS

We found 32 course reports that met search criteria, including 21 trauma-skills training courses. Courses were of variable duration, content, cost and scope. There were no prospective randomized clinical trials of course impact. Efficacy for most courses was with Kirkpatrick level 1 and 2 evidence of benefit by self-evaluations, and reporting small numbers of respondents. Few courses assessed skill retention with longitudinal data before and after training. Three courses, namely: Advanced Trauma Life Support (ATLS), Advanced Surgical Skills for Exposure in Trauma (ASSET) and Advanced Trauma Operative Management (ATOM) have Kirkpatrick's level 2-3 evidence for efficacy. Components of these 3 courses are included in several other courses, but many skills courses have little published evidence of training efficacy or skills retention durability.

CONCLUSIONS

Large variations in course content, duration, didactics, operative models, resource requirements and cost suggest that standardization of content, duration, and development of metrics for open surgery skills would be beneficial, as would translation into improved trauma patient outcomes. Surgeons at all levels of training and experience should participate in these trauma skills courses, because these procedures are rarely performed in routine clinical practice. Faculty running courses without evidence of training benefit should be encouraged to study outcomes to show their course improves technical skills and subsequently patient outcomes. Obtaining Kirkpatrick's level 3 and 4 evidence for benefits of ASSET, ATOM, ATLS and for other existing courses should be a high priority.