Assessment of EVAR Competence: Validity of a Novel Rating Scale (EVARATE) in a Simulated Setting

Certification of Basic Skills in Endovascular Aortic Repair Through a Modular Simulation Course With Real Time Performance Assessment

OBJECTIVE

Endovascular aortic repair (EVAR) is being used increasingly for the treatment of infrarenal abdominal aortic aneurysms. Improvement in educational strategies is required to teach future vascular surgeons EVAR skills, but a comprehensive, pre-defined e-learning and simulation curriculum remains to be developed and tested. EndoVascular Aortic Repair Assessment of Technical Expertise (EVARATE), an assessment tool for simulation based education (SBE) in EVAR, has previously been designed to assess EVAR skills, and a pass limit defining mastery level has been set. However, EVARATE was developed for anonymous video ratings in a research setting, and its feasibility for real time ratings in a standardised SBE programme in EVAR is unproven. This study aimed to test the effect of a newly developed simulation based modular course in EVAR. In addition, the applicability of EVARATE for real time performance assessments was investigated.

METHODS

The European Society of Vascular Surgery (ESVS) and Copenhagen Certification Programme in EVAR (ENHANCE-EVAR) was tested in a prospective cohort study. ENHANCE-EVAR is a modular SBE programme in EVAR consisting of e-learning and hands-on SBE. Participants were rated with the EVARATE tool by experienced EVAR surgeons.

RESULTS

Twenty-four physicians completed the study. The mean improvement in EVARATE score during the course was +11.8 (95% confidence interval 9.8 - 13.7) points (p < .001). Twenty-two participants (92%) passed with a mean number of 2.8 ± 0.7 test attempts to reach the pass limit. Cronbach's alpha coefficient was 0.91, corresponding to excellent reliability of the EVARATE scale. Differences between instructors' EVARATE ratings were insignificant (p = .16), with a maximum variation between instructors of ± 1.3 points.

CONCLUSION

ENHANCE-EVAR, a comprehensive certifying EVAR course, was proven to be effective. EndoVascular Aortic Repair Assessment of Technical Expertise (EVARATE) is a trustworthy tool for assessing performance within an authentic educational setting, enabling real time feedback.

Facilitators and Barriers to Implementation of Simulation Based Education in Vascular Surgery in Europe

OBJECTIVE

This study explored the status and availability of simulation based education (SBE) for learning vascular surgical procedures identified in the 2019 General Needs Assessment in vascular surgery in Europe (GNA-2019) and identified facilitators and barriers to SBE implementation in vascular surgery.

METHODS

A three round iterative survey was distributed via the European Society for Vascular Surgery and the Union Européenne des Médecins Spécialistes. Members from leading committees and organisations within the European vascular surgical community were invited to participate as key opinion leaders (KOLs). Three online survey rounds explored demographics, SBE availability, and facilitators and barriers to SBE implementation.

RESULTS

Overall, 147 KOLs (target population 338) accepted invitation to round 1, representing 30 European countries. The dropout rates for rounds 2 and 3 were 29% and 40%, respectively. Most respondents (88%) were senior, consultant level or higher. No mandatory SBE training was required in their department before training on patients, according to 84% of the KOLs. There was high consensus on the need for structured SBE (87%) and mandatory SBE (81%). SBE is available for the top three prioritised procedures in GNA-2019 (basic open skills, basic endovascular skills, and vascular imaging interpretation) in 24, 23, and 20 of the 30 represented European countries, respectively. The highest ranking facilitators were structured SBE programmes, availability of simulation equipment locally and regionally, good quality simulators, and having a dedicated person running the SBE. The highest ranked barriers were lack of structured SBE curriculum, equipment costs, lack of SBE culture, no or limited dedicated time for faculty to teach in SBE, and clinical work overload.

CONCLUSION

Based largely on the opinions of KOLs in vascular surgery in Europe, this study revealed that SBE is needed in vascular surgery training and that systematic and structured programmes are required to ensure successful implementation.

An Image Information-Based Objective Assessment Method of Technical Manipulation Skills for Intravascular Interventions

The clinical success of vascular interventional surgery relies heavily on a surgeon's catheter/guidewire manipulation skills and strategies. An objective and accurate assessment method plays a critical role in evaluating the surgeon's technical manipulation skill level. Most of the existing evaluation methods incorporate the use of information technology to find more objective assessment models based on various metrics. However, in these models, sensors are often attached to the surgeon's hands or to interventional devices for data collection, which constrains the surgeon's operational movements or exerts an influence on the motion trajectory of interventional devices. In this paper, an image information-based assessment method is proposed for the evaluation of the surgeon's manipulation skills without the requirement of attaching sensors to the surgeon or catheters/guidewires. Surgeons are allowed to use their natural bedside manipulation skills during the data collection process. Their manipulation features during different catheterization tasks are derived from the motion analysis of the catheter/guidewire in video sequences. Notably, data relating to the number of speed peaks, slope variations, and the number of collisions are included in the assessment. Furthermore, the contact forces, resulting from interactions between the catheter/guidewire and the vascular model, are sensed by a 6-DoF F/T sensor. A support vector machine (SVM) classification framework is developed to discriminate the surgeon's catheterization skill levels. The experimental results demonstrate that the proposed SVM-based assessment method can obtain an accuracy of 97.02% to distinguish between the expert and novice manipulations, which is higher than that of other existing research achievements. The proposed method has great potential to facilitate skill assessment and training of novice surgeons in vascular interventional surgery.

A Systematic Review of Simulation-Based Training in Vascular Surgery

Introduction

Recent advancements in surgical technology, reduced working hours, and training opportunities exacerbated by the COVID-19 pandemic have led to an increase in simulation-based training. Furthermore, a rise in endovascular procedures has led to a requirement for high-fidelity simulators that offer comprehensive feedback. This review aims to identify vascular surgery simulation models and assess their validity and levels of effectiveness (LoE) for each model in order to successfully implement them into current training curricula.

Methods

PubMed and EMBASE were searched on January 1, 2021, for full-text English studies on vascular surgery simulators. Eligible articles were given validity ratings based on Messick’s modern concept of validity alongside an LoE score according to McGaghie’s translational outcomes.

Results

Overall 76 eligible articles validated 34 vascular surgery simulators and training courses for open and endovascular procedures. High validity ratings were achieved across studies for: content (35), response processes (12), the internal structure (5), relations to other variables (57), and consequences (2). Only seven studies achieved an LoE greater than 3/5. Overall, ANGIO Mentor was the most highly validated and effective simulator and was the only simulator to achieve an LoE of 5/5.

Conclusions

Simulation-based training in vascular surgery is a continuously developing field with exciting future prospects, demonstrated by the vast number of models and training courses. To effectively integrate simulation models into current vascular surgery curricula and assessments, there is a need for studies to look at trainee skill retention over a longer period of time. A more detailed discussion on cost-effectiveness is also needed.

Training and assessment of musculoskeletal ultrasound and injection skills-a systematic review

OBJECTIVES

To examine how residents are trained and assessed in musculoskeletal ultrasound (MSUS), MSUS-guided and landmark-guided joint aspiration and injection. Additionally, to present the available assessment tools and examine their supporting validity evidence.

METHODS

A systematic search of PubMed, Cochrane Library, and Embase was conducted in accordance with the PRISMA guidelines and studies published from January 1, 2000 to May 31, 2021 were included. Two independent reviewers performed the search and data extraction. The studies were evaluated using the Medical Education Research Quality Instrument (MERSQI).

RESULTS

9,884 articles were screened, and 43 were included; 3 were randomized studies, 21 pre- and post-test studies, 16 descriptive studies, and 3 studies developing assessment tools. The studies used various theoretical training modalities e.g. lectures, anatomical quizzes and e-learning. The practical training models varied from mannequins and cadavers to healthy volunteers and patients. The majority of studies used subjective "comfort level" as assessment, others used practical examination and/or theoretical examination. All training programs increased trainees' self-confidence, theoretical knowledge, and/or practical performance, however few used validated assessment tools to measure the effect. Only one study met the MERSQI high methodical quality cut-off score of 14.

CONCLUSION

The included studies were heterogeneous, and most were of poor methodological quality and not based on contemporary educational theories. This review highlights the need for educational studies using validated theoretical and practical assessment tools to ensure optimal MSUS training and assessment in rheumatology.

Use of Generalizability Theory for Exploring Reliability of and Sources of Variance in Assessment of Technical Skills: A Systematic Review and Meta-Analysis

PURPOSE

Competency-based education relies on the validity and reliability of assessment scores. Generalizability (G) theory is well suited to explore the reliability of assessment tools in medical education but has only been applied to a limited extent. This study aimed to systematically review the literature using G-theory to explore the reliability of structured assessment of medical and surgical technical skills and to assess the relative contributions of different factors to variance.

METHOD

In June 2020, 11 databases, including PubMed, were searched from inception through May 31, 2020. Eligible studies included the use of G-theory to explore reliability in the context of assessment of medical and surgical technical skills. Descriptive information on study, assessment context, assessment protocol, participants being assessed, and G-analyses was extracted. Data were used to map G-theory and explore variance components analyses. A meta-analysis was conducted to synthesize the extracted data on the sources of variance and reliability.

RESULTS

Forty-four studies were included; of these, 39 had sufficient data for meta-analysis. The total pool included 35,284 unique assessments of 31,496 unique performances of 4,154 participants. Person variance had a pooled effect of 44.2% (95% confidence interval [CI], 36.8%-51.5%). Only assessment tool type (Objective Structured Assessment of Technical Skills-type vs task-based checklist-type) had a significant effect on person variance. The pooled reliability (G-coefficient) was 0.65 (95% CI, .59-.70). Most studies included decision studies (39, 88.6%) and generally seemed to have higher ratios of performances to assessors to achieve a sufficiently reliable assessment.

CONCLUSIONS

G-theory is increasingly being used to examine reliability of technical skills assessment in medical education, but more rigor in reporting is warranted. Contextual factors can potentially affect variance components and thereby reliability estimates and should be considered, especially in high-stakes assessment. Reliability analysis should be a best practice when developing assessment of technical skills.

Reliable and valid assessment of procedural skills in resuscitative endovascular balloon occlusion of the aorta

BACKGROUND

Valid and reliable assessment of skills is essential for improved and evidence-based training concepts. In a recent study, we presented a novel tool to assess procedural skills in resuscitative endovascular balloon occlusion of the aorta (REBOA), REBOA-RATE, based on international expert consensus. Although expert consensus is a strong foundation, the performance of REBOA-RATE has not been explored. The study aimed to examine the reliability and validity of REBOA-RATE.

METHODS

This was an experimental simulation-based study. We enrolled doctors with three levels of expertise to perform two REBOA procedures in a simulated scenario of out-of-hospital cardiac arrest. Procedures were video-recorded, and videos were blinded and randomized. Three clinical experts independently rated all procedures using REBOA-RATE. Data were analyzed using Messick's framework for validity evidence, including generalizability analysis of reliability and determination of a pass/fail standard.

RESULTS

Forty-two doctors were enrolled: 16 novices, 13 anesthesiologists, and 13 endovascular experts. They all performed two procedures, yielding 84 procedures and 252 ratings. The REBOA-RATE assessment tool showed high internal consistency (Cronbach's α = 0.95) and excellent interrater reliability (intraclass correlation coefficient, 0.97). Assessment using one rater and three procedures could ensure overall reliability suitable for high-stakes testing (G-coefficient >0.80). Mean scores (SD) for the three groups in the second procedure were as follows: novices, 32% (24%); anesthesiologists, 55% (29%); endovascular experts, 93% (4%) (p < 0.001). The pass/fail standard was set at 81%, which all experts but no novices passed.

CONCLUSION

Data strongly support the reliability and validity of REBOA-RATE, which successfully discriminated between all experience levels. The REBOA-RATE assessment tool requires minimal instruction, and one rater is sufficient for reliable assessment. Together, these are strong arguments for the use of REBOA-RATE to assess REBOA skills, allowing for competency-based training and certification concepts.

LEVEL OF EVIDENCE

Diagnostic test, no or poor gold standard, level V.

Standard Setting in Simulation-Based Training of Surgical Procedures: A Systematic Review

OBJECTIVE

This systematic review aims to examine the use of standard-setting methods in the context of simulation-based training of surgical procedures.

SUMMARY OF BACKGROUND

Simulation-based training is increasingly used in surgical education. However, it is important to determine which level of competency trainees must reach during simulation-based training before operating on patients. Therefore, pass/fail standards must be established using systematic, transparent, and valid methods.

METHODS

Systematic literature search was done in four databases (Ovid MEDLINE, Embase, Web of Science, and Cochrane Library). Original studies investigating simulation-based assessment of surgical procedures with application of a standard setting were included. Quality of evidence was appraised using GRADE.

RESULTS

Of 24,299 studies identified by searches, 232 studies met the inclusion criteria. Publications using already established standard settings were excluded (N = 70), resulting in 162 original studies included in the final analyses. Most studies described how the standard setting was determined (N = 147, 91%) and most used the mean or median performance score of experienced surgeons (n = 65, 40%) for standard setting. We found considerable differences across most of the studies regarding study design, set-up, and expert level classification. The studies were appraised as having low and moderate evidence.

CONCLUSION

Surgical education is shifting towards competency-based education, and simulation-based training is increasingly used for acquiring skills and assessment. Most studies consider and describe how standard settings are established using more or less structured methods but for current and future educational programs, a critical approach is needed so that the learners receive a fair, valid and reliable assessment.

Developing a tool to assess competence in resuscitative endovascular balloon occlusion of the aorta: An international Delphi consensus study

BACKGROUND

Resuscitative endovascular balloon occlusion of the aorta (REBOA) is an emergency procedure that is potentially lifesaving in major noncompressible torso hemorrhage. It may also improve outcome in nontraumatic cardiac arrest. However, the procedure can be technically challenging and requires the immediate presence of a qualified operator. Thus, evidence-based training and assessment of operator skills are essential for successful implementation and patient safety. A prerequisite for this is a valid and reliable assessment tool specific for the procedure. The aim of this study was to develop a tool for assessing procedural competence in REBOA based on best-available knowledge from international experts in the field.

METHODS

We invited international REBOA experts from multiple specialties to participate in an anonymous three-round iterative Delphi study to reach consensus on the design and content of an assessment tool. In round 1, participants suggested items to be included. In rounds 2 and 3, the relevance of each suggested item was evaluated by all participants to reach consensus. Interround data processing was done systematically by a steering group.

RESULTS

Forty panelists representing both clinical and educational expertise in REBOA from 16 countries (in Europe, Asia, and North and South America) and seven different specialties participated in the study. After 3 Delphi rounds and 532 initial item suggestions, the panelists reached consensus on a 10-item assessment tool with behaviorally anchored rating scales. It includes assessment of teamwork, procedure time, selection and preparation of equipment, puncture technique, guidewire handling, sheath handling, placement of REBOA catheter, occlusion, and evaluation.

CONCLUSION

We present the REBOA-RATE assessment tool developed systematically by international experts in the field to optimize content validity. Following further studies of its validity and reliability, this tool represents an important next step in evidence-based training programs in REBOA, for example, using mastery learning.

LEVEL OF EVIDENCE

Therapeutic, level V.

Video-Rated Performance Assessment of Simulated Laparoscopic Sleeve Gastrectomy: Validation of a Sleeve Gastrectomy Rating Scale

PURPOSE

The global rise in obesity has been accompanied by widespread uptake of the procedure of laparoscopic sleeve gastrectomy. Despite this, the key components for performance assessment have not been standardized for this procedure. The aim of this study was to develop and demonstrate the validity of a Sleeve Objective Structured Assessment of Technical Skill (SOSATS) scale for learning the procedure of laparoscopic sleeve gastrectomy (LSG).

MATERIALS AND METHODS

The SOSATS evaluation tool was based upon critical steps of the LSG procedure. Both the SOSATS and the Global Rating Scale (GRS) component of the Objective Structured Assessment of Technical Skill (OSATS) tools were utilized in a prospective single-blinded observational study design of 26 video recordings of surgeons performing sleeve gastrectomies using a novel simulation. The surgeons were allocated into "novice" or "experienced" groups dependent on case-volume criteria. Surgical performance was assessed using both the GRS and SOSATS scales by blinded assessors of the video recordings.

RESULTS

Face and content validity were demonstrated for key components of the simulated model. An overall positive correlation was established inferring concurrent validity between the accepted OSATS Global Rating Scale against the SOSATS procedural scale. Construct validity was established for a number of areas of the SOSATS scale.

CONCLUSION

The SOSATS scale is shown to exhibit construct and concurrent validity in the simulated setting for the procedure of sleeve gastrectomy. Utilizing this scale to review surgical performance is potentially feasible and reliable but would require further research prior to use in high-stakes assessment processes such as credentialing.

Evaluation of tools to assess operative competence in endovascular procedures: a systematic review

BACKGROUND

With an increase in the use of endovascular interventions as an alternative to open surgery and the unique technical skills required, current methods for assessing the competence of vascular surgery trainees may not be optimal, suggesting a need for a shift in assessment modalities. We conducted this systematic review to explore current assessment methods used in vascular surgery training to assess competence specific to endovascular procedures.

METHODS

A comprehensive literature search was performed with a structured search strategy using terms focusing on endovascular procedures and assessment. Inclusion and exclusion criteria were used in order to screen for suitable articles.

RESULTS

We identified 54 articles that satisfied the inclusion criteria. These included a single randomized controlled trial, a single systematic review, a single narrative review and a single literature review, with the vast majority having level 2 evidence. Global rating scales, proficiency assessments and written/oral examinations were described as standard current assessment tools. These modalities lack reproducibility and objectivity, neglecting the needs of assessment of endovascular procedures requiring specialized decision making and finger dexterity. Novel methods such as high fidelity simulation and virtual reality promote reproducible and objective assessment methods in the context of endovascular surgery, and have a promising future.

CONCLUSION

While current assessment methods in vascular surgery are widely supported the changing skills required of a vascular surgery trainee warrants a shift in assessment modalities to better align to these requirements. High fidelity simulations show promise, although they require more extensive research to understand their relative merits.

Radiation Safety Performance is More than Simply Measuring Doses! Development of a Radiation Safety Rating Scale

PURPOSE

Radiation safety performance is often evaluated using dose parameters measured by personal dosimeters and/or the C-arm, which provide limited information about teams' actual radiation safety behaviors. This study aimed to develop a rating scale to evaluate team radiation safety behaviors more accurately and investigate its reliability.

MATERIALS AND METHODS

A modified Delphi consensus was organized involving European vascular surgeons (VS), interventional radiologists, and interventional cardiologists. Initial items and anchors were drafted a priori and rated using five-point Likert scales. Participants could suggest additional items or adjustments. Consensus was defined as ≥ 80% agreement (rating ≥ 4) with Cronbach's alpha ≥ .80. Two VS with expertise in radiation safety evaluated 15 video-recorded endovascular repairs of infrarenal aortic aneurysms (EVAR) to assess usability, inter and intrarater reliability.

RESULTS

Thirty-one of 46 invited specialists completed three rating rounds to generate the final rating scale. Five items underwent major adjustments. In the final round, consensus was achieved for all items (alpha = .804; agreement > 87%): 'Pre-procedural planning', 'Preparation in angiosuite/operating room', 'Shielding equipment', 'Personal protective equipment', 'Position of operator/team', 'Radiation usage awareness', 'C-arm handling', 'Adjusting image quality', 'Additional dose reducing functions', 'Communication/leadership', and 'Overall radiation performance and ALARA principle'. All EVARs were rated, yielding excellent Cronbach's alpha (.877) with acceptable interrater and excellent intrarater reliability (ICC = .782; ICC = .963, respectively).

CONCLUSION

A reliable framework was developed to assess radiation safety behaviors in endovascular practice and provide teams with formative feedback. The final scale is provided in this publication.

Learn EVAR sizing from scratch: The results of a one-day intensive course in EVAR sizing and stent graft selection for vascular trainees

Background and aim

Recognition of structured training in endovascular aortic repair (EVAR) for vascular trainees is increasing. Nevertheless, how trainees can achieve sufficient skills in EVAR sizing and graft selection is sparsely described. The aim of this study was to investigate the effect of systematic training in basic EVAR sizing and graft selection on vascular surgery trainees using a validated assessment tool.

Methods

Sixteen vascular surgery trainees were included in an intensive 6-h hands-on workshop in aortic sizing and stent graft selection for EVAR with a trainer-to-trainee ratio of 1:2. After 1-h lecture, participants did 5 h of supervised training on increasingly complex cases. Finally, the participants were tested using a validated assessment tool.

Results

All participants were able to size the test-case and select a stent graft combination in 24:35 (13:30–48:20) min (median and range). The participants’ overall test scores (lower is better) were in median 17.9 (11.9–28.4). This did not differ from the scores of experienced EVAR operators 14.7 (11.7–25.2) (<200 EVAR’s) ( p = .32) but was inferior to the score of EVAR experts 11.2 (9.8 –18.7) (≥200 EVAR’s) ( p = .01). The sub-score for anatomical measurements was 10.6 (3.9–18.8) and comparable with the experienced group 9.7 (8.1–12.8) ( p = .83) but inferior to the expert operators 6.5 (5.2–10.2) ( p = .04). The sub-score for stent graft selection was 7.5 (4.9–14.1) and comparable with experienced operators scoring 4.5 (3.6–12.3) ( p = .09) but inferior to the expert operators score of 5.0 (3.6–8.4) ( p = .01).

Conclusion

This study presents the results of a standardised one-day basic EVAR sizing and graft selection workshop. Vascular surgery trainees with no prior EVAR experience learned to size and select stent grafts for a simple infra-renal AAA on par with experienced EVAR operators.

Training and assessment of competence in resuscitative endovascular balloon occlusion of the aorta (REBOA) - a systematic review

BACKGROUND

Resuscitative endovascular balloon occlusion of the aorta (REBOA) is a potentially life-saving but high-risk emergency procedure in patients with haemorrhagic shock. Lack of physicians with competence in the procedure is a barrier to implementation of REBOA. It is currently unclear how training and assessment of competence should be done.

OBJECTIVES

To report and evaluate research in training and assessment of competence in REBOA and femoral arterial access with the aim to investigate the effect of simulation-based training in the procedure and to provide suggestions for the future design of training programs and assessment tools.

METHODS

Following PRISMA guidelines, PubMed, Embase, and Cochrane Library databases were searched for studies on training or assessment of competence in REBOA and femoral arterial access. Bias assessment was done using the Medical Education Research Study Quality Instrument. Evidence level was assessed using GRADE.

RESULTS

Sixteen studies were included, six of them published as abstracts. Full-text studies included 189 trainees ranging in experience level from military medics to surgical specialists. Outcome measures were heterogenous; the most used were rater checklists, knowledge testing, and procedure time. All studies confirmed an effect of training of REBOA on procedural competence in a simulation setting but had a high degree of bias. No study developed or used an assessment tool supported by validity evidence and no study investigated mid and long-term outcomes.

CONCLUSION

Simulation-based training of REBOA improves skills, however, the evidence level is very low and data cannot answer important questions on effect size, skill transfer and retention, and optimal course design. To advance research and training programmes, an assessment tool supported by validity evidence with broad applicability is needed.