Technical skills assessment toolbox: a review using the unitary framework of validity

Efficacy of Trauma Surgery Technical Skills Training Courses

OBJECTIVE

Because open surgical skills training for trauma is limited in clinical practice, trauma skills training courses were developed to fill this gap, The aim of this report is to find supporting evidence for efficacy of these courses. The questions addressed are: What courses are available and is there robust evidence of benefit?

DESIGN

We performed a systematic review of the training course literature on open trauma surgery procedural skills courses for surgeons using Kirkpatrick's framework for evaluating complex educational interventions. Courses were identified using Pubmed, Google Scholar and other databases.

SETTING AND PARTICIPANTS

The review was carried out at the University of Maryland, Baltimore with input from civilian and military trauma surgeons, all of whom have taught and/or developed trauma skills courses.

RESULTS

We found 32 course reports that met search criteria, including 21 trauma-skills training courses. Courses were of variable duration, content, cost and scope. There were no prospective randomized clinical trials of course impact. Efficacy for most courses was with Kirkpatrick level 1 and 2 evidence of benefit by self-evaluations, and reporting small numbers of respondents. Few courses assessed skill retention with longitudinal data before and after training. Three courses, namely: Advanced Trauma Life Support (ATLS), Advanced Surgical Skills for Exposure in Trauma (ASSET) and Advanced Trauma Operative Management (ATOM) have Kirkpatrick's level 2-3 evidence for efficacy. Components of these 3 courses are included in several other courses, but many skills courses have little published evidence of training efficacy or skills retention durability.

CONCLUSIONS

Large variations in course content, duration, didactics, operative models, resource requirements and cost suggest that standardization of content, duration, and development of metrics for open surgery skills would be beneficial, as would translation into improved trauma patient outcomes. Surgeons at all levels of training and experience should participate in these trauma skills courses, because these procedures are rarely performed in routine clinical practice. Faculty running courses without evidence of training benefit should be encouraged to study outcomes to show their course improves technical skills and subsequently patient outcomes. Obtaining Kirkpatrick's level 3 and 4 evidence for benefits of ASSET, ATOM, ATLS and for other existing courses should be a high priority.

Surgery Tutor for Computational Assessment of Technical Proficiency in Soft-Tissue Tumor Resection in a Simulated Setting

BACKGROUND

In competency-based medical education, progression between milestones requires reliable and valid methods of assessment. Surgery Tutor is an open-source motion tracking platform developed to objectively assess technical proficiency during open soft-tissue tumor resections in a simulated setting. The objective of our study was to provide evidence in support of construct validity of the scores obtained by Surgery Tutor. We hypothesized that Surgery Tutor would discriminate between novice, intermediate, and experienced operators.

METHODS

Thirty participants were assigned to novice, intermediate, or experienced groups, based on the number of prior soft-tissue resections performed. Each participant resected 2 palpable and 2 nonpalpable lesions from a soft-tissue phantom. Surgery Tutor was used to track hand and instrument motions, number of tumor breaches, and time to perform each resection. Mass of excised specimens and margin status were also recorded.

RESULTS

Surgery Tutor scores demonstrated "moderate" to "good" internal structure (test-retest reliability) for novice, intermediate, and experienced groups (interclass correlation coefficient = 0.596, 0.569, 0.737; p < 0.001). Evidence in support of construct validity (consequences) was demonstrated by comparing scores of novice, intermediate, and experienced participantsfor number of hand and instrument motions (690 ± 190, 597 ± 169, 469 ± 110; p < 0.001), number of tumor breaches (29 ± 34, 16 ± 11, 9 ± 6; p < 0.001), time per resection (677 ± 331 seconds, 561 ± 210 seconds, 449 ± 148 seconds; p < 0.001), mass of completely excised specimens (22 ± 7g, 21 ± 11g, 17 ± 6 g; p = 0.035), and rate of positive margin (68%, 50%, 28%; p < 0.001). There was "strong" and "moderate" relationships between motion scores and Objective Structured Assessment of Technical Skill scores, and time per resection and Objective Structured Assessment of Technical Skill scores respectively (r = -0.60, p < 0.001; r = -0.54, p < 0.001).

CONCLUSION

Surgery Tutor scores demonstrate evidenceof construct validity with regards to good internal structure, consequences, and relationship to other variables in the assessment of technical proficiency duringopen soft-tissue tumor resections in a simulated setting. Utilization of Surgery Tutor can provide formative feedback and objective assessment of surgical proficiency in a simulated setting.

Objectively Assessing Intraoperative Arthroscopic Skills Performance and the Transfer of Simulation Training in Knee Arthroscopy: A Randomized Controlled Trial

PURPOSE

To objectively investigate the transfer validity of simulation training using wireless elbow-worn motion sensors intraoperatively to assess whether surgical simulation leads to improvements in intraoperative arthroscopic performance.

METHODS

In this randomized controlled trial, postgraduate year 2 to 3 trainees in nationally approved orthopaedic surgery posts were randomized to standard junior residency training (control group) or standard training plus additional weekly simulation training (intervention group). Both groups performed a supervised real-life diagnostic knee arthroscopy in the operating room at 13 weeks. Performance was measured using wireless elbow-worn motion sensors recording objective surgical performance metrics: number of hand movements, smoothness, and time taken. A participant-supervisor performance ratio was used to adjust for variation in case mix and difficulty. The study took place in a surgical simulation suite and the orthopaedic operating rooms of a university teaching hospital.

RESULTS

The intervention group objectively outperformed the control group in all outcome metrics. Procedures performed by the intervention group required fewer hand movements (544 [interquartile range (IQR), 465-593] vs 893 [IQR, 747-1,242]; P < .001), had smoother movements (25,842 ms^-3 [IQR, 20,867-27,468 ms^-3] vs 36,846 ms^-3 [IQR, 29,840-53,949 ms^-3]; P < .001), and took less time (320 seconds [IQR, 294-392 seconds] vs 573 seconds [IQR, 477-860 seconds]; P < .001) than those performed by the control group. The cases were comparable between the groups. Standardized to the supervisor's performance, the intervention group required fewer hand movements (1.9 [IQR, 1.5-2.1] vs 3.3 [IQR, 2.2-4.8]; P = .0091), required less time (1.2 [IQR, 1.1-1.7] vs 2.6 [IQR, 1.6-3.0]; P = .0037), and were smoother (2.1 [IQR, 1.8-2.8] vs 4.3 [IQR, 2.8-5.4]; P = .0037) than the control group, but they did not perform as well as their supervisors.

CONCLUSIONS

This study uses intraoperative motion-analysis technology to objectively show that surgical simulation training improves actual intraoperative technical skills performance.

CLINICAL RELEVANCE

The described wireless objective assessment method complements the subjective observational performance assessments commonly used. Further studies are required to assess how these measures of intraoperative performance correlate to patient outcomes. Intraoperative motion analysis is translatable across surgical specialties, offering potential for objective assessment of progression through competency-based training, revalidation, and talent selection for specialist training.

Systematic review of team performance in minimally invasive abdominal surgery

Background

Adverse events in the operating theatre related to non‐technical skills and teamwork are still an issue. The influence of minimally invasive techniques on team performance and subsequent impact on patient safety remains unclear. The aim of this review was to assess the methodology used to objectify and rate team performance in minimally invasive abdominal surgery.

Methods

A systematic literature search was conducted according to the PRISMA guidelines. Studies on assessment of surgical team performance or non‐technical skills of the surgical team in the setting of minimally invasive abdominal surgery were included. Study aim, methodology, results and conclusion were extracted for qualitative synthesis.

Results

Sixteen studies involving 677 surgical procedures were included. All studies consisted of observational case series that used heterogeneous methodologies to assess team performance and were of low methodological quality. The most commonly used team performance objectification tools were ‘construct’‐ and ‘incident’‐based tools. Evidence of validity for the assessed outcome was spread widely across objectification tools, ranging from low to high. Diverse and poorly defined outcomes were reported.

Conclusion

Team demands for minimally invasive approaches to abdominal procedures remain unclear. The current literature consists of studies with heterogeneous methodology and poorly defined outcomes.

Assessment of surgical performance of laparoscopic benign hiatal surgery: a systematic review

BACKGROUND

Operative skills correlate with patient outcomes, yet at the completion of training or after learning a new procedure, these skills are rarely formally evaluated. There is interest in the use of summative video assessment of laparoscopic benign foregut and hiatal surgery (LFS). If this is to be used to determine competency, it must meet the robust criteria established for high-stakes assessments. The purpose of this review is to identify tools that have been used to assess performance of LFS and evaluate the available validity evidence for each instrument.

METHODS

A systematic search was conducted up to July 2017. Eligible studies reported data on tools used to assess performance in the operating room during LFS. Two independent reviewers considered 1084 citations for eligibility. The characteristics and testing conditions of each assessment tool were recorded. Validity evidence was evaluated using five sources of validity (content, response process, internal structure, relationship to other variables, and consequences).

RESULTS

There were six separate tools identified. Two tools were generic to laparoscopy, and four were specific to LFS [two specific to Nissen fundoplication (NF), one heller myotomy (HM), and one paraesophageal hernia repair (PEH)]. Overall, only one assessment was supported by moderate evidence while the others had limited or unknown evidence. Validity evidence was based mainly on internal structure (all tools reporting reliability and item analysis) and content (two studies referencing previous papers for tool development in the context of clinical assessment, and four listing items without specifying the development procedures). There was little or no evidence supporting test response process (one study reporting rater training), relationship to other variables (two comparing scores in subjects with different clinical experience), and consequences (no studies). Two tools were identified to have evidence for video assessment, specific to NF.

CONCLUSION

There is limited evidence supporting the validity of assessment tools for laparoscopic foregut surgery. This precludes their use for summative video-based assessment to verify competency. Further research is needed to develop an assessment tool designed for this purpose.

How do visual-spatial and psychomotor abilities influence clinical performance in periodontal plastic surgery?

AIM

We want to evaluate the relationship of self-assessed experience and proficiency, manual dexterity and visual-spatial ability with surgical performance.

MATERIAL AND METHODS

A total of 26 professionals were included in the study which consisted of four parts: (a) self-assessment by a questionnaire regarding proficiency and experience, (b) evaluation of visual-spatial ability, (c) testing of manual dexterity assessed by validated psychomotor tests and (d) evaluation of surgical performance by Objective Structured Assessment of Technical Skills (OSATS).

RESULTS

Self-assessed proficiency and experience levels did not correlate with objectively evaluated surgical performances (OSATS). However, low-level visual-spatial ability tests strongly correlated with OSATS while intermediate- and high-level tests did not. No correlation was found between psychomotor ability and clinical performance.

CONCLUSIONS

Self-assessed proficiency is not a good predictor for surgical performance as experts tend to be overconfident. To evaluate and predict surgical performance, visual-spatial ability tests seem to be more appropriate than measuring manual dexterity which failed to correlate with the surgical outcome.

Measuring the teamwork performance of teams in crisis situations: a systematic review of assessment tools and their measurement properties

BACKGROUND

Educational interventions to improve teamwork in crisis situations have proliferated in recent years with substantial variation in teamwork measurement. This systematic review aimed to synthesise available tools and their measurement properties in order to identify the most robust tool for measuring the teamwork performance of teams in crisis situations.

METHODS

Searches were conducted in Embase (via OVID), PsycINFO, Cumulative Index to Nursing and Allied Health Literature, Education Resources Information Center, Medline and Medline In-Process (via OVID) (through 12 January 2017). Studies evaluating the measurement properties of teamwork assessment tools for teams in clinical or simulated crisis situations were included. Two independent reviewers screened studies based on predetermined criteria and completed data extraction. Risk of bias was assessed using the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) checklist.

RESULTS

The search yielded 1822 references. Twenty studies were included, representing 13 assessment tools. Tools were primarily assessed in simulated resuscitation scenarios for emergency department teams. The Team Emergency Assessment Measure (TEAM) had the most validation studies (n=5), which demonstrated three sources of validity (content, construct and concurrent) and three sources of reliability (internal consistency, inter-rater reliability and test-retest reliability). Most studies of TEAM's measurement properties were at no risk of bias.

CONCLUSIONS

A number of tools are available for assessing teamwork performance of teams in crisis situations. Although selection will ultimately depend on the user's context, TEAM may be the most promising tool given its measurement evidence. Currently, there is a lack of tools to assess teamwork performance during intraoperative crisis situations. Additional research is needed in this regard.

Validity evidence for procedural competency in virtual reality robotic simulation, establishing a credible pass/fail standard for the vaginal cuff closure procedure

BACKGROUND

The use of robotic surgery for minimally invasive procedures has increased considerably over the last decade. Robotic surgery has potential advantages compared to laparoscopic surgery but also requires new skills. Using virtual reality (VR) simulation to facilitate the acquisition of these new skills could potentially benefit training of robotic surgical skills and also be a crucial step in developing a robotic surgical training curriculum. The study's objective was to establish validity evidence for a simulation-based test for procedural competency for the vaginal cuff closure procedure that can be used in a future simulation-based, mastery learning training curriculum.

METHODS

Eleven novice gynaecological surgeons without prior robotic experience and 11 experienced gynaecological robotic surgeons (> 30 robotic procedures) were recruited. After familiarization with the VR simulator, participants completed the module 'Guided Vaginal Cuff Closure' six times. Validity evidence was investigated for 18 preselected simulator metrics. The internal consistency was assessed using Cronbach's alpha and a composite score was calculated based on metrics with significant discriminative ability between the two groups. Finally, a pass/fail standard was established using the contrasting groups' method.

RESULTS

The experienced surgeons significantly outperformed the novice surgeons on 6 of the 18 metrics. The internal consistency was 0.58 (Cronbach's alpha). The experienced surgeons' mean composite score for all six repetitions were significantly better than the novice surgeons' (76.1 vs. 63.0, respectively, p < 0.001). A pass/fail standard of 75/100 was established. Four novice surgeons passed this standard (false positives) and three experienced surgeons failed (false negatives).

CONCLUSION

Our study has gathered validity evidence for a simulation-based test for procedural robotic surgical competency in the vaginal cuff closure procedure and established a credible pass/fail standard for future proficiency-based training.

Economy of Hand Motion During Cleft Palate Surgery Using a High-Fidelity Cleft Palate Simulator

OBJECTIVE

The objectives of this study were to assess economy of hand motion of residents, fellows, and staff surgeons using a high-fidelity cleft palate simulator to (1) stratify performance for the purpose of simulator validation and (2) to estimate the learning curve.

DESIGN

Two residents, 2 fellows, and 2 staff surgeons performed cleft palate surgery on a high-fidelity cleft palate simulator while their hand motion was tracked using an electromagnetic hand sensor. The time, number of hand movements, and path length of their hands were determined for 10 steps of the procedure. The magnitude of these metrics was compared among the 3 groups of participants and utilized to estimate the learning curve using curve-fitting analysis.

RESULTS

The residents required the most time, number of hand movements, and path length to complete the procedure. Although the number of hand movements was closely matched between the fellows and staff, the overall total path length was shorter for the staff. Inverse curves were fit to the data to represent the learning curve and 25 and 113 simulation sessions are required to reach within 5% and 1% of the expert level, respectively.

CONCLUSION

The simulator successfully stratified performance using economy of hand motion. Path length is better matched to previous level of experience compared to time or number of hand movements.

A radiation-free mixed-reality training environment and assessment concept for C-arm-based surgery

PURPOSE

The discrepancy of continuously decreasing opportunities for clinical training and assessment and the increasing complexity of interventions in surgery has led to the development of different training and assessment options like anatomical models, computer-based simulators or cadaver trainings. However, trainees, following training, assessment and ultimately performing patient treatment, still face a steep learning curve.

METHODS

To address this problem for C-arm-based surgery, we introduce a realistic radiation-free simulation system that combines patient-based 3D printed anatomy and simulated X-ray imaging using a physical C-arm. To explore the fidelity and usefulness of the proposed mixed-reality system for training and assessment, we conducted a user study with six surgical experts performing a facet joint injection on the simulator.

RESULTS

In a technical evaluation, we show that our system simulates X-ray images accurately with an RMSE of 1.85 mm compared to real X-ray imaging. The participants expressed agreement with the overall realism of the simulation, the usefulness of the system for assessment and strong agreement with the usefulness of such a mixed-reality system for training of novices and experts. In a quantitative analysis, we furthermore evaluated the suitability of the system for the assessment of surgical skills and gather preliminary evidence for validity.

CONCLUSION

The proposed mixed-reality simulation system facilitates a transition to C-arm-based surgery and has the potential to complement or even replace large parts of cadaver training, to provide a safe assessment environment and to reduce the risk for errors when proceeding to patient treatment. We propose an assessment concept and outline the steps necessary to expand the system into a test instrument that provides reliable and justified assessments scores indicative of surgical proficiency with sufficient evidence for validity.

A scoping review of assessment tools for laparoscopic suturing

BACKGROUND

A needs assessment identified a gap in teaching and assessment of laparoscopic suturing (LS) skills. The purpose of this review is to identify assessment tools that were used to assess LS skills, to evaluate validity evidence available, and to provide guidance for selecting the right assessment tool for specific assessment conditions.

METHODS

Bibliographic databases were searched till April 2017. Full-text articles were included if they reported on assessment tools used in the operating room/simulation to (1) assess procedures that require LS or (2) specifically assess LS skills.

RESULTS

Forty-two tools were identified, of which 26 were used for assessing LS skills specifically and 26 for procedures that require LS. Tools had the most evidence in internal structure and relationship to other variables, and least in consequences.

CONCLUSION

Through identification and evaluation of assessment tools, the results of this review could be used as a guideline when implementing assessment tools into training programs.

Direct Observation vs. Video-Based Assessment in Flexible Cystoscopy

OBJECTIVE

Direct observation in assessment of clinical skills is prone to bias, demands the observer to be present at a certain location at a specific time, and is time-consuming. Video-based assessment could remove the risk of bias, increase flexibility, and reduce the time spent on assessment. This study investigated if video-based assessment was a reliable tool for cystoscopy and if direct observers were prone to bias compared with video-raters.

DESIGN

This study was a blinded observational trial. Twenty medical students and 9 urologists were recorded during 2 cystoscopies and rated by a direct observer and subsequently by 2 blinded video-raters on a global rating scale (GRS) for cystoscopy. Both intrarater and interrater reliability were explored. Furthermore, direct observer bias was explored by a paired samples t-test.

RESULTS

Intrarater reliability calculated by Pearson's r was 0.86. Interrater reliability was 0.74 for single measure and 0.85 for average measures. A hawk-dove effect was seen between the 2 raters. Direct observer bias was detected when comparing direct observer scores to the assessment by an independent video-rater (p < 0.001).

CONCLUSION

This study found that video-based assessment was a reliable tool for cystoscopy with 2 video-raters. There was a significant bias when comparing direct observation with blinded video-based assessment.

Relationship between intraoperative non-technical performance and technical events in bariatric surgery

Background

The operating theatre is a unique environment with complex team interactions, where technical and non-technical performance affect patient outcomes. The correlation between technical and non-technical performance, however, remains underinvestigated. The purpose of this study was to explore these interactions in the operating theatre.

Methods

A prospective single-centre observational study was conducted at a tertiary academic medical centre. One surgeon and three fellows participated as main operators. All patients who underwent a laparoscopic Roux-en-Y gastric bypass and had the procedures captured using the Operating Room Black Box® platform were included. Technical assessment was performed using the Objective Structured Assessment of Technical Skills and Generic Error Rating Tool instruments. For non-technical assessment, the Non-Technical Skills for Surgeons (NOTSS) and Scrub Practitioners' List of Intraoperative Non-Technical Skills (SPLINTS) tools were used. Spearman rank-order correlation and N-gram statistics were conducted.

Results

Fifty-six patients were included in the study and 90 procedural steps (gastrojejunostomy and jejunojejunostomy) were analysed. There was a moderate to strong correlation between technical adverse events (rs = 0·417–0·687), rectifications (rs = 0·380–0·768) and non-technical performance of the surgical and nursing teams (NOTSS and SPLINTS). N-gram statistics showed that after technical errors, events and prior rectifications, the staff surgeon and the scrub nurse exhibited the most positive non-technical behaviours, irrespective of operator (staff surgeon or fellow).

Conclusion

This study demonstrated that technical and non-technical performances are related, on both an individual and a team level. Valuable data can be obtained around intraoperative errors, events and rectifications.

Contrasting groups' standard setting for consequences analysis in validity studies: reporting considerations

Background

The contrasting groups’ standard setting method is commonly used for consequences analysis in validity studies for performance in medicine and surgery. The method identifies a pass/fail cut-off score, from which it is possible to determine false positives and false negatives based on observed numbers in each group. Since groups in validity studies are often small, e.g., due to a limited number of experts, these analyses are sensitive to outliers on the normal distribution curve.

Methods

We propose that these shortcomings can be addressed in a simple manner using the cumulative distribution function.

Results

We demonstrate considerable absolute differences between the observed false positives/negatives and the theoretical false positives/negatives. In addition, several important examples are given.

Conclusions

We propose that a better reporting strategy is to report theoretical false positives and false negatives together with the observed false positives and negatives, and we have developed an Excel sheet to facilitate such calculations.

Trial registration

Not relevant.

Electronic supplementary material

The online version of this article (10.1186/s41077-018-0064-7) contains supplementary material, which is available to authorized users.

Developing the Blueprint for a General Surgery Technical Skills Certification Examination: A Validation Study

INTRODUCTION

There is a recognized need to develop high-stakes technical skills assessments for decisions of certification and resident promotion. High-stakes examinations requires a rigorous approach in accruing validity evidence throughout the developmental process. One of the first steps in development is the creation of a blueprint which outlines the potential content of examination. The purpose of this validation study was to develop an examination blueprint for a Canadian General Surgery assessment of technical skill certifying examination.

METHODS

A Delphi methodology was used to gain consensus amongst Canadian General Surgery program directors as to the content (tasks or procedures) that could be included in a certifying Canadian General Surgery examination. Consensus was defined a priori as a Cronbach's α ≥ 0.70. All procedures or tasks reaching a positive consensus (defined as ≥80% of program directors rated items as ≥4 on the 5-point Likert scale) were then included in the final examination blueprint.

RESULTS

Two Delphi rounds were needed to reach consensus. Of the 17 General Surgery Program directors across the country, 14 (82.4%) and 10 (58.8%) program directors responded to the first and second round, respectively. A total of 59 items and procedures reached positive consensus and were included in the final examination blueprint.

CONCLUSIONS

The present study has outlined the development of an examination blueprint for a General Surgery certifying examination using a consensus-based methodology. This validation study will serve as the foundational work from which simulated model will be developed, pilot tested and evaluated.

Validation of the Peripheral Ultrasound-guided Vascular Access Rating Scale

Evidence-based standards in proficiency are needed for ultrasound-guided peripheral intravenous access. In this study, we explored the validity of the Peripheral Ultrasound-Guided Vascular Access (P-UGVA) Rating Scale.We recruited 3 groups of physicians (5 novices, 5 intermediates, and 5 experts) of increasing proficiency in peripheral ultrasound-guided intravenous access. All participants performed 3 peripheral ultrasound-guided intravenous accesses on three different patients. Performance was video-recorded by 3 cameras and the ultrasound image. Synchronized and anonymized split-screen film clips were rated using the P-UGVA rating scale by 2 assessors, which also assessed overall performance on a 1-5 Likert-scale. Evidence of validity was explored using the contemporary validity framework by Messick (content, response process, internal structure, relations to other variables, and consequences).Content and response process was ensured in the development of the rating scale and validity study. Internal consistency of the P-UGVA rating scale was excellent and sufficient high for certification purposes (Cronbach's alpha = 0.91). Proficiency groups were successfully discriminated by the UPGIVA rating scale (P = .029, one-way ANOVA), and the P-UGVA rating scale scores also correlated strongly with the overall performance evaluations (rho = 0.87, P < .001, Pearson correlation). We calculated a pass/fail score of 29, which lead to a theoretical false positive rate of 26.5% and false negative rate of 8.5%.We present validity evidence for the P-UGVA rating scale and an evidence-based standard in proficiency for ultrasound-guided peripheral intravenous access.

A cadaveric procedural anatomy simulation course improves video-based assessment of operative performance

BACKGROUND

Inadequate anatomic knowledge has been cited as a major contributor to declining surgical resident operative competence. We analyzed the impact of a comprehensive, procedurally oriented cadaveric procedural anatomy dissection laboratory on the operative performance of surgery residents, hypothesizing that trainees' performance of surgical procedures would improve after such a dissection course.

MATERIALS AND METHODS

Midlevel general surgery residents (n = 9) participated in an 8 wk, 16-h surgery faculty-led procedurally oriented cadaver simulation course. Both before and after completion of the course, residents participated in a practical examination, in which they were randomized to perform one of nine Surgical Council on Resident Education-designated "essential" procedures. The procedures were recorded using wearable video technology. Videos were deidentified before evaluation by six faculty raters blinded to examinee and whether performances occurred before or after an examinee had taken the course. Raters used the validated Operative Performance Rating System and Objective Structured Assessment of Technical Skill scales.

RESULTS

After the course residents had higher procedure-specific scores (median, 4.0 versus 2.4, P < 0.0001), instrument-handling (4.0 versus 3.0, P = 0.006), respect for tissue (4.0 versus 3.0, P = 0.0004), time and motion (3.0 versus 2.0, P = 0.0007), operation flow (3.0 versus 2.0, P = 0.0005), procedural knowledge (4.0 versus 2.0, P = 0.0001), and overall performance scores (4.0 versus 2.0, P < 0.0001). Operative Performance Rating System and Objective Structured Assessment of Technical Skill scales averaged by number of items in each were also higher (3.2 versus 2.0, P = 0.0002 and 3.1 versus 2.2, P = 0.002, respectively).

CONCLUSIONS

A cadaveric procedural anatomy simulation course covering a broad range of open general surgery procedures was associated with significant improvements in trainees' operative performance.

Development of a reliable simulation-based test for diagnostic abdominal ultrasound with a pass/fail standard usable for mastery learning

BACKGROUND

This study aimed to develop a test with validity evidence for abdominal diagnostic ultrasound with a pass/fail-standard to facilitate mastery learning.

METHOD

The simulator had 150 real-life patient abdominal scans of which 15 cases with 44 findings were selected, representing level 1 from The European Federation of Societies for Ultrasound in Medicine and Biology. Four groups of experience levels were constructed: Novices (medical students), trainees (first-year radiology residents), intermediates (third- to fourth-year radiology residents) and advanced (physicians with ultrasound fellowship). Participants were tested in a standardized setup and scored by two blinded reviewers prior to an item analysis.

RESULTS

The item analysis excluded 14 diagnoses. Both internal consistency (Cronbach's alpha 0.96) and inter-rater reliability (0.99) were good and there were statistically significant differences (p < 0.001) between all four groups, except the intermediate and advanced groups (p = 1.0). There was a statistically significant correlation between experience and test scores (Pearson's r = 0.82, p < 0.001). The pass/fail-standard failed all novices (no false positives) and passed all advanced (no false negatives). All intermediate participants and six out of 14 trainees passed.

CONCLUSION

We developed a test for diagnostic abdominal ultrasound with solid validity evidence and a pass/fail-standard without any false-positive or false-negative scores.

KEY POINTS

• Ultrasound training can benefit from competency-based education based on reliable tests. • This simulation-based test can differentiate between competency levels of ultrasound examiners. • This test is suitable for competency-based education, e.g. mastery learning. • We provide a pass/fail standard without false-negative or false-positive scores.

Efficacy and Feasibility of Objective Structured Clinical Examination in the Internal Assessment for Surgery Postgraduates

INTRODUCTION

Traditionally assessment in medical training programs has been through subjective faculty evaluations or multiple choice questions. Conventional examinations provide assessment of the global performance rather than individual competencies thus making the final feedback less meaningful. The objective structured clinical examination (OSCE) is a relatively new multidimensional tool for evaluating training. This study was carried out to determine the efficacy and feasibility of OSCE as a tool for the internal assessment of surgery residents.

METHODS

This study was carried out on the marks obtained by surgery residents at different levels of training in a single tertiary center in India over the 4 OSCEs conducted in the years 2015 and 2016. The marks of the OSCE were collected from the departmental records and analyzed. Reliability was calculated using internal consistency using Cronbach's α. Validity was calculated by item total correlation. Content validation was done by obtaining expert reviews from 5 experts using a proforma, to assess the content and checklist of each station of the OSCE.

RESULTS

A total of 49 surgery residents were assessed in small batches during the above mentioned period. Of the 4 OSCEs conducted by us, 3 had a high value of Cronbach's α of greater than 0.9, as opposed to the set standard of 0.7. Out of 23 stations used in the 4 examinations separately, only 3 stations were found to have a low correlation coefficient (item total correlation), and hence, a low validity. The remaining 20 stations were found to have a high validity. Expert review showed unanimous validation of the content of 17 out of the 23 stations, with few suggestions for change in the remaining 6 stations. The material and manpower used was minimal and easy to obtain, thus making the OSCE feasible to conduct.

CONCLUSION

OSCE is a reliable, valid. and feasible method for evaluating surgery residents at various levels of training.

Objective Assessment of Surgical Technical Skill and Competency in the Operating Room

Training skillful and competent surgeons is critical to ensure high quality of care and to minimize disparities in access to effective care. Traditional models to train surgeons are being challenged by rapid advances in technology, an intensified patient-safety culture, and a need for value-driven health systems. Simultaneously, technological developments are enabling capture and analysis of large amounts of complex surgical data. These developments are motivating a "surgical data science" approach to objective computer-aided technical skill evaluation (OCASE-T) for scalable, accurate assessment; individualized feedback; and automated coaching. We define the problem space for OCASE-T and summarize 45 publications representing recent research in this domain. We find that most studies on OCASE-T are simulation based; very few are in the operating room. The algorithms and validation methodologies used for OCASE-T are highly varied; there is no uniform consensus. Future research should emphasize competency assessment in the operating room, validation against patient outcomes, and effectiveness for surgical training.

Evaluation of Procedural Simulation as a Training and Assessment Tool in General Surgery-Simulating a Laparoscopic Appendectomy

BACKGROUND

Laparoscopic appendectomy is a commonly performed surgical procedure, but few training models have been described for it. We examined a virtual reality module for practising a laparoscopic appendectomy.

METHODS

A prospective cohort study with the following 3 groups of surgeons (n = 45): novices (0 procedures), intermediates (10-50 procedures), and experienced (>100 procedures). After being introduced to the simulator and 1 familiarization attempt on the procedural module, the participants practiced the module 20 times. Movements, task time, and procedure-specific parameters were compared over time.

RESULTS

The time and movement parameters were significantly different during the first attempt, and more experienced surgeons used fewer movements and less time than novices (p < 0.01), although only 2 parameters were significantly different between novices and intermediates. All 3 groups improved significantly over 20 attempts (p < 0.0001). The intraclass correlation coefficient varied between 0.55 and 0.68 and did not differ significantly between the 3 groups (p > 0.05). When comparing novices with experienced surgeons, novices had a higher risk of burn damage to cecum (odds ratio [OR] = 3.0 [95% CI: 1.3; 7.0] p = 0.03), pressure damage to appendix (OR = 3.1 [95% CI: 2.0; 4.9] p < 0.0001), and grasping of the appendix (OR = 2.9 [95% CI: 1.8; 4.7] p < 0.0001). The risk of causing a perforation was not significantly different among the different experience levels (OR = 1.9 [95% CI: 0.9; 3.8] p = 0.14). Only 3 out of 5 error parameters differed significantly when comparing novices and experienced surgeons. Similarly, when comparing intermediates and novices, it was only 2 of the parameters that differed.

DISCUSSION

The simulator module for practising laparoscopic appendectomy may be useful as a training tool, but further development is required before it can be used for assessment purposes. Procedural simulation may demonstrate more variation for movement parameters, and future research should focus on developing better procedure-specific parameters.

Twelve tips for assessing surgical performance and use of technical assessment scales

Using validated assessment scales for technical competence can help structure and standardize assessment and feedback for both the trainee and the supervisor and thereby avoid bias and drive learning. Correct assessment of operative skills can establish learning curves and allow adequate monitoring. However, the assessment of surgical performance is not an easy task, since it includes many proxy parameters, which are hard to measure. Although numerous technical assessment scales exist, both within laparoscopic and open surgery, the validity evidence is often sparse, and this can raise doubts about reliability and educational outcome. Furthermore, the implementation of technical assessment scales varies due to several obstacles and doubts about accurate use. In this 12-tips article, we aim to give the readers a critical and useful appraisal of some of the common questions and misunderstandings regarding the use of surgical assessment scales and provide tips to ease and overcome potential pitfalls.

Introduction of Basic Dermatologic Ultrasound in Undergraduate Medical Education

Purpose: Teaching ultrasound procedures to undergraduates has recently been proposed to improve the quality of medical education. We address the impact of applying standardized dermatologic ultrasound teaching to our undergraduates. Materials and Methods: Medical students were offered an additional theoretical and practical seminar involving hands-on ultrasound dermatologic ultrasound during their mandatory dermatology practical training. The students' theoretical knowledge and dermatologic ultrasoud skills were tested with a multiple choice questionnaire extracted from Level 1 Spanish Society of Ultrasound Dermatologic Ultrasound accreditation exam before and after the course. After the course, the students were asked to answer a course evaluation questionnaire Results: The multiple-choice question scores after the course showed statistically significant improvement (5.82 vs. 8.71%; P<0.001). The questionnaire revealed that students were satisfied with the course, felt that it increased both their dermatologic and ultrasound knowledge, and indicated that they wanted more sonographic hands-on training in both dermatologic ultrasound and other medical fields. Conclusion: Using both objective and subjective methods, we showed that the introduction of standardized ultrasound training programs in undergraduate medical education can improve both students' understanding of the technique and the quality of medical education in dermatology.

Ensuring Competency of Novice Laparoscopic Surgeons-Exploring Standard Setting Methods and their Consequences

OBJECTIVE

Simulation-based assessment tools have been developed to allow for proficiency-based simulator training in laparoscopy. However, few studies have examined the consequences of different standard setting methods or examined what level of proficiency is considered adequate for trainees. The objectives of the present study were to explore the consequences of different standard setting methods and to examine the proficiency level that surgical trainees are expected to reach, before performing supervised surgery on patients.

DESIGN

Study participants undertook the Training and Assessment of Basic Laparoscopic Techniques test. The tests were video-recorded and rated using a simple scoring system based on number of errors and time. Participants were then asked to assess how high a score a novice should reach before performing supervised surgery on a patient. We then compared 3 methods of standard setting: expert performance level, contrasting groups method, and a modified Angoff method.

SETTING

The study was conducted at the Copenhagen Academy for Medical Education and Simulation. The academy provides surgical simulation training in laparoscopy for trainees at the hospitals in the Capital Region and the Zealand Region of Denmark.

PARTICIPANTS

Participants were recruited among surgical trainees in their first year of specialty training from surgery, gynecology, and urology departments. A total of 40 participants were included and completed the trial.

RESULTS

The different standard setting methods resulted in different pass/fail levels. At the expert performance level, the pass/fail level was 474 points-the contrasting groups method resulted in 358 points and the modified Angoff method resulted in 311 points among experienced surgeons, and 386 points among trainees. The different proficiency levels resulted in a failure rate of 0% to 50% of experienced surgeons and a pass rate of 0% to 25% of novices. Novice laparoscopic surgeons set a higher pass/fail level than experienced surgeons did (p = 0.008).

CONCLUSION

Required proficiency levels varies depending on the standard setting method used, which highlights the importance of using an established standard setting method to set the pass/fail level.

Using virtual reality simulation to assess competence in video-assisted thoracoscopic surgery (VATS) lobectomy

BACKGROUND

The societies of thoracic surgery are working to incorporate simulation and competency-based assessment into specialty training. One challenge is the development of a simulation-based test, which can be used as an assessment tool. The study objective was to establish validity evidence for a virtual reality simulator test of a video-assisted thoracoscopic surgery (VATS) lobectomy of a right upper lobe.

METHODS

Participants with varying experience in VATS lobectomy were included. They were familiarized with a virtual reality simulator (LapSim^®) and introduced to the steps of the procedure for a VATS right upper lobe lobectomy. The participants performed two VATS lobectomies on the simulator with a 5-min break between attempts. Nineteen pre-defined simulator metrics were recorded.

RESULTS

Fifty-three participants from nine different countries were included. High internal consistency was found for the metrics with Cronbach's alpha coefficient for standardized items of 0.91. Significant test-retest reliability was found for 15 of the metrics (p-values <0.05). Significant correlations between the metrics and the participants VATS lobectomy experience were identified for seven metrics (p-values <0.001), and 10 metrics showed significant differences between novices (0 VATS lobectomies performed) and experienced surgeons (>50 VATS lobectomies performed). A pass/fail level defined as approximately one standard deviation from the mean metric scores for experienced surgeons passed none of the novices (0 % false positives) and failed four of the experienced surgeons (29 % false negatives).

CONCLUSION

This study is the first to establish validity evidence for a VATS right upper lobe lobectomy virtual reality simulator test. Several simulator metrics demonstrated significant differences between novices and experienced surgeons and pass/fail criteria for the test were set with acceptable consequences. This test can be used as a first step in assessing thoracic surgery trainees' VATS lobectomy competency.

Training and Assessment of Hysteroscopic Skills: A Systematic Review

OBJECTIVE

The aim of this systematic review was to identify studies on hysteroscopic training and assessment.

DESIGN

PubMed, Excerpta Medica, the Cochrane Library, and Web of Science were searched in January 2015. Manual screening of references and citation tracking were also performed. Studies on hysteroscopic educational interventions were selected without restrictions on study design, populations, language, or publication year. A qualitative data synthesis including the setting, study participants, training model, training characteristics, hysteroscopic skills, assessment parameters, and study outcomes was performed by 2 authors working independently. Effect sizes were calculated when possible. Overall, 2 raters independently evaluated sources of validity evidence supporting the outcomes of the hysteroscopy assessment tools.

RESULTS

A total of 25 studies on hysteroscopy training were identified, of which 23 were performed in simulated settings. Overall, 10 studies used virtual-reality simulators and reported effect sizes for technical skills ranging from 0.31 to 2.65; 12 used inanimate models and reported effect sizes for technical skills ranging from 0.35 to 3.19. One study involved live animal models; 2 studies were performed in clinical settings. The validity evidence supporting the assessment tools used was low. Consensus between the 2 raters on the reported validity evidence was high (94%).

CONCLUSIONS

This systematic review demonstrated large variations in the effect of different tools for hysteroscopy training. The validity evidence supporting the assessment of hysteroscopic skills was limited.