Residents' Procedural Experience Does Not Ensure Competence: A Research Synthesis

Simulation-based education in ultrasound - diagnostic and interventional abdominal focus

Simulation-based training (SBT) is increasingly acknowledged worldwide and has become a popular tool for ultrasound education. Ultrasound simulation involves the use of technology and software to create a virtual training setting. Simulation-based training allows healthcare professionals to learn, practice, and improve their ultrasound imaging skills in a safe learning-based environment. SBT can provide a realistic and focused learning experience that creates a deep and immersive understanding of the complexity of ultrasound, including enhancing knowledge and confidence in specific areas of interest. Abdominal ultrasound simulation is a tool to increase patient safety and can be a cost-efficient training method. In this paper, we provide an overview of various types of abdominal ultrasound simulators, and the benefits, and challenges of SBT. We also provide examples of how to develop SBT programs and learning strategies including mastery learning. In conclusion, the growing demand for medical imaging increases the need for healthcare professionals to start using ultrasound simulators in order to keep up with the rising standards.

Preparing interns for clinical practice through an institution-wide simulation-based mastery learning program for teaching central venous catheter placement

Central venous catheter (CVC) placement is a challenging procedure with known iatrogenic risks. However, there are no residency program requirements to demonstrate baseline CVC procedural competency. Competency-based procedural education has been shown to decrease CVC-associated morbidity, but there has been limited literature about institution-wide efforts to ensure initial trainee competency for CVC placement. This study describes the implementation of a competency-based CVC curriculum for first-year interns across an institution before supervised clinical care. An institution-wide, simulation-based mastery training curriculum was designed to assess initial competency in CVC placement in first-year residents during 2021 and 2022. A checklist was internally developed with a multidisciplinary team. Using the Mastery-Angoff technique, minimum passing standards were derived to define competency levels considered appropriate for intern participation in supervised clinical care. Interns were trained through the competency-based program with faculty assessing intern performance using the CVC checklist to verify procedural competency. Over 2 academic cycles, 229 interns from 20 specialties/subspecialties participated. Overall, 83% of interns met performance standards on their first posttest attempt, 14% on the second attempt, and 3% on the third attempt. Interns from both cycles demonstrated significant improvement from baseline to posttest scores (P < .001). Overall, 10.5% of interns performed dangerous actions during assessment (malpositioning, retained guidewire, or carotid dilation). All interns ultimately achieved the passing standard to demonstrate initial competency in the simulation assessment. All participating interns demonstrated simulation-based competency allowing them to place CVCs under supervised clinical care. Dangerous actions, however, were not uncommon. Simulation-based teaching and learning frameworks were a feasible method to promote patient safety through an institutional-wide verification of preliminary procedural competency.

Behavioral skills training for teaching safety skills to mental health service providers compared to training-as-usual: a pragmatic randomized control trial

BACKGROUND

Violence in the healthcare workplace has been a global concern for over two decades, with a high prevalence of violence towards healthcare workers reported. Workplace violence has become a healthcare quality indicator and embedded in quality improvement initiatives of many healthcare organizations. The Centre for Addiction and Mental Health (CAMH), Canada's largest mental health hospital, provides all clinical staff with mandated staff safety training for self-protection and team-control skills. These skills are to be used as a last resort when a patient is at imminent risk of harm to self or others. The purpose of this study is to compare the effectiveness of two training methods of this mandated staff safety training for workplace violence in a large psychiatric hospital setting.

METHODS

Using a pragmatic randomized control trial design, this study compares two approaches to teaching safety skills CAMH's training-as-usual (TAU) using the 3D approach (description, demonstration and doing) and behavioural skills training (BST), from the field of applied behaviour analysis, using instruction, modeling, practice and feedback loop. Staff were assessed on three outcome measures (competency, mastery and confidence), across three time points: before training (baseline), immediately after training (post-training) and one month later (follow-up). This study was registered with the ISRCTN registry on 06/09/2023 (ISRCTN18133140).

RESULTS

With a sample size of 99 new staff, results indicate that BST was significantly better than TAU in improving observed performance of self-protection and team-control skills. Both methods were associated with improved skills and confidence. However, there was a decrease in skill performance levels at the one-month follow-up for both methods, with BST remaining higher than TAU scores across all three time points. The impact of training improved staff confidence in both training methods and remained high across all three time points.

CONCLUSIONS

The study findings suggest that BST is more effective than TAU in improving safety skills among healthcare workers. However, the retention of skills over time remains a concern, and therefore a single training session without on-the-job-feedback or booster sessions based on objective assessments of skill may not be sufficient. Further research is needed to confirm and expand upon these findings in different settings.

Effect of Polypectomy Simulation-Based Mastery Learning on Skill Retention Among Practicing Endoscopists

PURPOSE

Practicing endoscopists frequently perform and teach screening colonoscopies and polypectomies, but there is no standardized method to train and assess physicians who perform polypectomy procedures. The authors created a polypectomy simulation-based mastery learning (SBML) curriculum and hypothesized that completion of the curriculum would lead to immediate improvement in polypectomy skills and skill retention at 6 and 12 months after training.

METHOD

The authors performed a pretest-posttest cohort study with endoscopists who completed SBML and were randomized to follow-up at 6 or 12 months from May 2021 to August 2022. Participants underwent SBML training, including a pretest, a video lecture, deliberate practice, and a posttest. All learners were required to meet or exceed a minimum passing standard on a 17-item skills checklist before completing training and were randomized to follow-up at 6 or 12 months. The authors compared simulated polypectomy skills performance on the checklist from pretest to posttest and posttest to 6- or 12-month follow-up test.

RESULTS

Twenty-four of 30 eligible participants (80.0%) completed the SBML intervention, and 20 of 24 (83.3%) completed follow-up testing. The minimum passing standard was set at 93% of checklist items correct. The pretest passing rate was 4 of 24 participants (16.7%) compared with 24 of 24 participants (100%) at posttest ( P < .001). There were no significant differences in passing rates from posttest to combined 6- and 12-month posttest in which 18 of 20 participants (90.0%) passed.

CONCLUSIONS

Before training and despite years of clinical experience, practicing endoscopists demonstrated poor performance of polypectomy skills. SBML was an effective method for practicing endoscopists to acquire and maintain polypectomy skills during a 6- to 12-month period.

Efficiency of Simulation-Based Learning Using an ABC POCUS Protocol on a High-Fidelity Simulator

Critically ill patients with rapidly deteriorating clinical status secondary to respiratory and cardio-vascular compromise are at risk for immediate collapse if the underlying pathology is not recognized and treated. Rapid diagnosis is of utmost importance regardless of the setting. Although there are data to support the use of point-of-care ultrasound in critical patients, there is no consensus about the best educational strategy to implement. We designed a curriculum based on the ABC (Airway, Breathing, Circulation) protocol that covers essential airway, lung, and cardiac ultrasound skills needed for fast diagnosis in critical patients and applied it in high-fidelity simulation-based medical education sessions for anesthesia and intensive care residents year one and two. After theoretical and practical assessments, our results show statistical differences in the theoretical knowledge and above-average results in practical assessment. Our proposed curriculum based on a simple ABC POCUS protocol, with an Airway, Breathing, and Circulation approach, is useful in teaching ultrasound basics regarding airway, lung, and cardiac examination using high-fidelity simulation training to anesthesia and intensive care residents, but further research is needed to establish the utility of Simulation-Based Medical Education in Point of Care Ultrasound in the critical patient.

Barriers and enablers to achieving clinical procedure competency-based outcomes in a national paediatric training/residency program-a multi-centered qualitative study

BACKGROUND

In 2018, the Royal College of Physicians of Ireland revised its paediatric training program to a competency-based medical education (CBME) training/residency curriculum. This included a requirement to achieve competence in a number of core procedural skills to progress within the program. Internationally, simulation-based medical education (SBME) is gaining interest as an effective teaching pedagogy for training procedural skill competency. The objectives of this study were to (1) identify enablers and barriers for paediatric trainees to achieve their required procedural competencies, (2) gain insight on the feasibility of achieving the required procedural skills, and (3) explore what simulation-based resources are used as well as their role in achieving the required procedural skill competencies.

METHODS

A multi-centered qualitative study using semi-structured interviews was performed. Twenty-four paediatric consultants and trainees were recruited from two academic tertiary hospitals using purposive and snowball sampling. Interviews were conducted between March and September 2021, audio recorded, transcribed, and analyzed using thematic analysis.

RESULTS

Three main themes regarding enablers for achieving procedural competencies were reported and include having protected training time, routine assessments, and a standardized curriculum. Barriers to achieving procedural competencies focused mainly on limited clinical exposure. The use of SBME was recommended by all participants (n = 24, 100%) to assist in achieving procedural competencies and most (n = 15, 62.5%) reported it is feasible to attain the required procedural skills in the paediatric CBME program.

CONCLUSION

It is feasible to achieve the required procedural competencies for most paediatric trainees, but this can be improved with protected training time, routine assessments, and a standardized curriculum. Barriers to achieving these skills mainly center on limited clinical exposure, which can be remedied by SBME. Further research is warranted to determine the costs and types of SBME tools available as well as teaching pedagogies to support paediatric trainees achieve their required procedural competencies.

An investigation into emergency medicine resident cricothyrotomy competency: Is three the magic number?

Objectives

Cricothyrotomy is a high-stakes emergency procedure. Because the procedure is rare, simulation is often used to train residents. The Accreditation Council for Graduate Medical Education (ACGME) requires performance of three cricothyrotomies during residency, but the optimal number of training repetitions is unknown. Additional repetitions beyond three could increase proficiency, though it is unknown whether there is a threshold beyond which there is no benefit to additional repetition. The objective of this study was to establish a minimum number of simulated cricothyrotomy attempts beyond which additional attempts did not increase proficiency.

Methods

This was a prospective, observational study conducted over 3 years at the simulation center of an academic emergency medicine residency program. Participants were residents participating in a cricothyrotomy training as part of a longitudinal airway curriculum course. The primary outcome was time to successful completion of the procedure as first-year residents. Secondary outcomes included time to completion as second- and third-year residents. Procedure times were plotted as a function of attempt number. Data were analyzed using descriptive statistics, repeated-measures analysis of variance, and correlation analysis. Preprocedure surveys collected further data regarding procedure experience, confidence, and comfort.

Results

Sixty-nine first-year residents participated in the study. Steady improvement in time to completion was seen through the first six attempts (from a mean of 75 to 41 sec), after which no further significant improvement was found. Second- and third-year residents initially demonstrated slower performance than first-year residents but rapidly improved to surpass their first-year performance. Resident mean times at five attempts were faster with each year of residency (first-year 48 sec, second-year 30 sec, third-year 24 sec). There was no statistically significant correlation between confidence and time to complete the procedure.

Conclusions

Additional repetition beyond the ACGME-endorsed three cricothyrotomy attempts may help increase proficiency. Periodic retraining may be important to maintain skills.

Validation of an Educational Tool for Skin Abscess Incision and Drainage by Delphi and Angoff Methods

BACKGROUND

Bedside incision and drainage (I&D) of skin abscesses is a common medical procedure performed in a variety of medical settings. Yet, there is a paucity of published validated educational tools to teach and assess competency for this procedure.

OBJECTIVE

To validate an educational tool to teach and assess competency for bedside I&D of skin abscesses via the Delphi consensus and Angoff standard setting methods.

DESIGN

Expert consensus on the importance of each procedural step in the educational tool was obtained using the Delphi method, consisting of four rounds of iterative revisions based on input from a panel of experts. The passing cut-off score for a proficient provider was determined using the modified dichotomous Angoff method.

PARTICIPANTS

All participants met the minimum criteria of active involvement in resident education and performance of at least 20 skin abscess I&D's within the past 5 years. Participant specialties included general surgery, emergency medicine, and internal medicine.

MAIN MEASURES

The primary outcome was consensus on procedural steps and errors, defined as an interquartile range ≤ 2 on a 9-point Likert scale. A cut-off score was determined by the average across all respondents for the anticipated number of errors that would be committed by a provider with the level of proficiency defined in the survey. Qualitative input was incorporated into the educational tool.

KEY RESULTS

At the end of four rounds of review via the Delphi process, participants achieved consensus on 93% of items on the clinical checklist and 85% of errors on the assessment checklist. Via the modified dichotomous Angoff method, the determined passing cut-off for competency was 6 out of 22 errors.

CONCLUSION

An educational and evaluation tool for bedside I&D of skin abscesses was validated via the Delphi and Angoff methods.

Anticipation or avoidance: internal medicine resident experiences performing invasive bedside procedures

Background

Internal Medicine (IM) residents are required to perform bedside procedures for diagnostic and therapeutic purposes. Residents' experiences with procedures vary widely, for unclear reasons.

Objective

To explore IM residents' experiences with performing bedside procedures and to identify barriers and facilitators to obtaining sufficient experience.

Methods

Using an inductive, thematic approach, we conducted five individual semi-structured interviews and one focus group with seven IM residents (12 residents in total) during the 2017-2018 academic year at a Canadian tertiary care centre. We used iterative, open-ended questions to elicit residents' experiences, and barriers and facilitators, to performing bedside procedures. Transcripts were analyzed for themes using Braun and Clarke's method.

Results

We identified four themes 1) Patient-specific factors such as body habitus and procedure urgency; 2) Systems factors such as time constraints and accessibility of materials; 3) Faculty factors including availability to supervise, comfort level, and referral preferences, and 4) Resident-specific factors including preparation, prior experiences, and confidence. Some residents expressed procedure-related anxiety and avoidance.

Conclusion

Educational interventions aimed to improve procedural efficiency and ensure availability of supervisors may help facilitate residents to perform procedures, yet may not address procedure-related anxiety. Further study is required to understand better how procedure-averse residents can gain confidence to seek out procedures.

Pre-clerkship procedural training in venipuncture: a prospective cohort study on skills acquisition and durability

BACKGROUND

The effectiveness of simulation-based training for skill acquisition is widely recognized. However, the impact of simulation-based procedural training (SBPT) on pre-clerkship medical students and the retention of procedural skills learned through this modality are rarely investigated.

METHODS

A prospective cohort study was conducted among pre-clerkship medical students. Learners underwent SBPT in venipuncture in the skills laboratory. Assessments were conducted at two main points: 1) immediate assessment following the training and 2) delayed assessment one year after training. Learner self-assessments, independent assessor assessments for procedural competency, and communication skills assessments were conducted in both instances. The students were assessed for their competency in performing venipuncture by an independent assessor immediately following the training in the simulated setting and one-year post-training in the clinical setting, using the Integrated Procedural Protocol Instrument (IPPI). The student's communication skills were assessed by standardized patients (SP) and actual patients in the simulated and clinical settings, respectively, using the Communication Assessment Tool (CAT).

RESULTS

Fifty-five pre-clerkship medical students were recruited for the study. A significant increase was observed in self-confidence [mean: 2.89 SD (Standard Deviation) (0.69)] and self-perceived competency [mean: 2.42 SD (0.57)] in performing venipuncture, which further improved at the delayed assessment conducted in the clinical setting (p < 0.001). Similarly, the IPPI ratings showed an improvement [immediate assessment: mean: 2.25 SD (1.62); delayed assessment: mean: 2.78 SD (0.53); p < 0.01] in venipuncture skills when assessed by an independent assessor blinded to the study design. A significant difference (p < 0.01) was also observed in doctor-patient communication when evaluated by SPs [mean: 2.49 SD (0.57)] and patients [mean: 3.76 SD (0.74)].

CONCLUSION

Simulation-based venipuncture training enabled students to perform the procedure with confidence and technical accuracy. Improved rating scores received at a one-year interval denote the impact of clinical training on skills acquisition. The durability of skills learned via SBPT needs to be further investigated.

Characterizing individual trainee learning curves in surgical training: Challenges and opportunities

The surgical learning curve is an observable and measurable phenomenon. In the era of competency-based approaches to surgical training, monitoring the trajectory of individual trainee competence attainment could represent a meaningful method of formative and summative assessment. While technology can assist this approach, a number of significant barriers to the implementation of such assessment methods remain, including: accurate data collection, standard setting, and reliable assessment. Translating individual learning curve data into quantifiable case minimum targets in training poses further difficulties, and may not be possible for all procedures, particularly those that are less frequently performed and assessed. In spite of these challenges, significant benefits could be realized through an individualized approach to competency assessment using trainee learning curve data. Tracking competence acquisition against criterion-referenced standards could allow for targeted training and remediation, conforming with modern theories of adult education and empowering trainees to take control of their own learning. Learning curve data could also be used to assess the effects of educational interventions such as simulation-based training on subsequent competence acquisition rates. Ultimately, the individual learning curves of trainees could be used to inform personalised decisions regarding entrustment, credentialing, and certification, allowing training programmes to move beyond minimum operative experience targets as a crude proxy measure of competence.

Putting the "learning" in "pre-learning": effects of a self-directed study hall on skill acquisition in a simulation-based central line insertion course

BACKGROUND

Opportunities to practice procedural skills in the clinical learning environment are decreasing, and faculty time to coach skills is limited, even in simulation-based training. Self-directed learning with hands-on practice early in a procedural skill course might help maximize the benefit of later faculty coaching and clinical experience. However, it may also lead to well-learned errors if learners lack critical guidance. The present study sought to investigate the effects of a hands-on, self-directed "study hall" for central line insertion among first-year residents.

METHODS

Learner cohorts before vs. after introduction of the study hall (n = 49) were compared on their pre- and post-test performance of key procedural behaviors that were comparable across cohorts, with all learners receiving traditional instructor-led training between tests.

RESULTS

Study hall participants spent a median of 116 min in hands-on practice (range 57-175). They scored higher at pre-test (44% vs. 27%, p = .00; Cohen's d = 0.95) and at post-test (80% vs. 72%, p = .02; Cohen's d = 0.69). A dose-response relationship was found, such that 2 h of study hall were roughly equivalent to the performance improvement seen with four clinical observations or supervised insertions of central lines.

CONCLUSIONS

Self-directed, hands-on "study hall" supported improved procedural skill learning in the context of limited faculty availability. Potential additional benefits make the approach worth further experimentation and evaluation.

Necessity of Pretests in Central Venous Catheter Insertion Simulation-Based Mastery Learning: A Randomized Controlled Trial

PURPOSE

Simulation-based mastery learning (SBML) is a rigorous form of competency-based learning. Components of SBML include a pretest, deliberate practice, and a posttest; all learners must meet or exceed a minimum passing standard (MPS) on the posttest before completing training. The authors aimed to explore whether a modified SBML curriculum (without a pretest assessment) was as effective as the standard SBML curriculum (with a pretest assessment).

METHOD

The authors performed a randomized controlled trial of internal medicine residents who participated in an internal jugular central venous catheter insertion SBML curriculum at a tertiary care academic medical center in Chicago, Illinois, from December 2018 through December 2021. Residents were randomly assigned to complete the usual SBML intervention (pretest group) or to complete a modified SBML intervention without a pretest (no pretest group). The authors compared initial posttest performance and training time between groups.

RESULTS

Eighty-nine of 120 eligible residents (74.1%) completed the study: 43 in the pretest group and 46 in the no pretest group. Median (IQR) initial posttest scores were not statistically different between the pretest group (96.6 [93.1-100]) and the no pretest group (96.6 [92.4-100]). However, all 43 residents (100%) in the pretest group reached the MPS at the initial posttest compared with 41 of the 46 (89%) in the no pretest group ( P = .06). Residents in the pretest group required 16.5 hours more faculty and learning time than the no pretest group.

CONCLUSIONS

More residents who completed a pretest reached the MPS at initial posttest. However, incorporating a pretest during the internal jugular central venous catheter SBML curriculum required substantially more learner and faculty time without clear performance benefits.

A novel simulator-based checklist for evaluating residents' competence in cerebral angiography in China

Background

Nowadays, with the fast-increasing demand for neuro-endovascular therapy, surgeons in this field are in urgent need. Unfortunately, there is still no formal skill assessment in neuro-endovascular therapy in China.

Methods

We used a Delphi method to design a newly objective checklist for standards of cerebrovascular angiography in China and evaluated its validity and reliability. A total of 19 neuro-residents with no interventional experience and 19 neuro-endovascular surgeons from two centers (Guangzhou and Tianjin) were recruited; they were divided into two groups: residents and surgeons. Residents completed a simulation-based cerebrovascular angiography operation training before assessment. Assessments were under live and video record forms with two tools: the existing global rating scale (GRS) of endovascular performance and the new checklist.

Results

The average scores of residents were significantly increased after training in two centers (p < 0.05). There is good consistency between GRS and the checklist (p = 0.856). Intra-rater reliability (Spearman's rho) of the checklist was >0.9, and the same result was also observed in raters between different centers and different assessment forms (p < 0.001, rho > 0.9). The reliability of the checklist was higher than that of the GRS (Kendall's harmonious coefficient is 0.849, while GRS is 0.684).

Conclusion

The newly developed checklist appears reliable and valid for evaluating the technical performance of cerebral angiography and differentiating between trained and untrained trainees' performance well. For its efficiency, our method has been proven to be a feasible tool for resident angiography examination in certification nationwide.

Impact of number of critical care procedural skill repetitions on supervision level and teaching style

BACKGROUND

During critical care procedural skills training (e.g., in intubation and pericardiocentesis) the appropriate supervision level is important to ensure correct use of techniques and guarantee patient safety. The appropriate teaching style should be selected to address residents' learning behavior and foster their competence. The aim of this study was to explore the number of repetitions for given skills needed to achieve a specified supervision level and a specific teaching style.

METHODS

This cross-sectional multicenter survey obtained data from residents and faculty of three multidisciplinary intensive care units (ICU) in Switzerland. Using a 4-point Likert scale, participants were asked to indicate the number of repetitions required to achieve the specified supervision level and teaching style.

RESULTS

Among 91 physicians, the response rate was 64% (n = 59). Their median estimations of the numbers of skill repetitions needed to achieve the final fourth level of supervision and final fourth stage of teaching style were as follows: arterial catheter insertion: supervision level 32, teaching style 17.5; peritoneal paracentesis: supervision level 27, teaching style 17; central venous catheter insertion: supervision level 38, teaching style 28; lumbar puncture: supervision level 38, teaching style 21; endotracheal intubation: supervision level 100, teaching style 45; chest drain insertion: supervision level 27, teaching style 21.5; temporary pacemaker placement: supervision level 50, teaching style 19.5; percutaneous tracheostomy: supervision level 50, teaching style 29; pericardiocentesis: supervision level 50, teaching style 35. Comparison of repetitions between supervision level and teaching style revealed no difference at the first and second levels, except for endotracheal intubation at level 2 (p = 0.03). Differences were observed at the third and fourth levels of supervision level and teaching style (p≤0.04).

CONCLUSIONS

It appears that the supervision level and teaching style applied by faculty should change according to both the number of repetitions and the difficulty of critical care procedural skills.

Simulation-based Mastery Learning Improves Emergency Medicine Residents' Ability to Perform Temporary Transvenous Cardiac Pacing

INTRODUCTION

Temporary transvenous cardiac pacing (TVP) is a critical intervention that emergency physicians perform infrequently in clinical practice. Prior simulation studies revealed that emergency medicine (EM) residents and board-certified emergency physicians perform TVP poorly during checklist-based assessments. Our objective in this report was to describe the design and implementation of a simulation-based mastery learning (SBML) curriculum and evaluate its impact on EM residents' ability to perform TVP.

METHODS

An expert panel of emergency physicians and cardiologists set a minimum passing standard (MPS) for a previously developed 30-item TVP checklist using the Mastery Angoff approach. Emergency medicine residents were assessed using this checklist and a high-fidelity TVP task trainer. Residents who did not meet the MPS during baseline testing viewed a procedure video and completed a 30-minute individual deliberate practice session before retesting. Residents who did not meet the MPS during initial post-testing completed additional deliberate practice and assessment until meeting or exceeding the MPS.

RESULTS

The expert panel set an MPS of correctly performing 28 (93.3%) checklist items. Fifty-seven EM residents participated. Mean checklist scores improved from 13.4 (95% CI 11.8-15.0) during baseline testing to 27.5 (95% CI 26.9-28.1) during initial post-testing (P < 0.01). No residents met the MPS at baseline testing. The 21 (36.8%) residents who did not meet the MPS during initial post-testing all met or exceeded the MPS after completing one additional 30-minute deliberate practice session.

CONCLUSION

Emergency medicine residents demonstrated significantly improved TVP performance with reduced variability in checklist scores after completing a simulation-based mastery learning curriculum.

Learning pediatric emergency medicine over time: A realist evaluation of a longitudinal pediatric emergency medicine clinical experience

INTRODUCTION

Emergency medicine (EM) practitioners must be proficient at caring for patients of all ages, including pediatric patients. Traditionally, EM trainees learn pediatric emergency medicine (PEM) through block rotations. This is problematic due to the seasonal nature of pediatric diseases and infrequent critical events. Spaced repetition learning theory suggests PEM would be better learned through longitudinal rotations. The transition to competency-based medical education (CBME) in Canada is accelerating the need to find novel ways to attain competencies in postgraduate training. At McMaster University, senior EM trainees can choose either traditional PEM blocks or longitudinal rotations. Our objective was to understand how learners experience these different rotations given the transition to CBME in Canada.

METHODS

Using a realist framework of program evaluation, we conducted semistructured interviews with key stakeholders (trainees, program directors, attending physicians) in EM. The realist framework was used to understand how context interacts with theoretical mechanisms to produce outcomes of interest. Data were analyzed using inductive, conventional content analysis. All investigators coded a subset of transcripts independently and in duplicate to achieve intercoder agreement.

RESULTS

A total of 13 interviews were completed with trainees (n = 11) and staff physicians (n = 2). The learning experience exists within an educational and clinical context, which are logistically distinct but inseparable. The longitudinal learning experience appears to improve learning through spaced repetition, which prevents atrophy of skills and knowledge while also benefitting from the offsetting of seasonal variability associated with many pediatric diseases. Improved feedback and entrustment are facilitated through the building of coaching relationships over time. Barriers to the learning experience are related mainly to logistical difficulties associated with resolving longitudinal and blocked learning experiences. Improved relationships with the interprofessional team may provide distinct learning opportunities and improved team functioning. Block rotations were identified as more valuable to junior trainees learning fundamental concepts.

CONCLUSIONS

Longitudinal learning provides numerous advantages to learning PEM, including increased case variety, spaced repetition of core concepts, and a perception of greater entrustment of the learner through formation of coaching relationships over time. Future projects looking to quantify the differences between longitudinal and block learning to objectively show a difference in skills and knowledge are needed.

Practice Doesn't Make Perfect: Clinical Experience With Procedures Does Not Correlate Well With Competence in Third-Year Medical Students

OBJECTIVE

There is limited data available about factors which promote competence with procedures in medical students. Specifically, the relationship between procedural clinical experience and performance on an assessment is unclear. We sought to determine whether a correlation exists between the amount and type of clinical experience with a procedure and student performance on a standardized assessment of that procedure.

DESIGN

Faculty performed standardized assessments of third-year medical students on ten procedures using simulation. We prospectively surveyed students about 3 types of experience (performed, observed, and simulated) with these procedures during their clerkships. We then analyzed whether a correlation exists between student experience and their competency assessment scores using Pearson's correlation.

SETTING/PARTICIPANTS

Third-year medical students at the University of Kentucky College of Medicine.

RESULTS

In 2018 to 2019, 131 students were assessed on procedural competency with 10 failures. One hundred and twenty students (91.6%) completed the clinical experience survey. Correlations between types of experience and competency scores were small to moderate, with only 5 of 40 being significant. We found no correlation between experience having performed a procedure and competency score.

CONCLUSIONS

Overall, we did not find convincing evidence of a correlation between experience with procedures during clerkships and performance on a competency assessment. This suggests other factors may be contributing to procedural competence, which has implications for how educators should develop procedural competence in students.

Effectiveness of training primary care internal medicine residents in etonogestrel implants and impact on their future practice: A cross-sectional study

OBJECTIVE

To evaluate the impact of an etonogestrel implant training program within a primary care Internal Medicine residency training program.

STUDY DESIGN

We surveyed graduates of our primary care Internal Medicine residency program in the Bronx, New York who performed implant procedures though the first 32 months after implementation of a monthly faculty-supervised resident implant clinic. We assessed the number of implants placed and removed per graduate, and surveyed graduates' satisfaction with the implant training program, perceived competence with implant procedures, and intent and ability to perform implant procedures and barriers to performing implant procedures postgraduation.

RESULTS

Between July 2017 and February 2020, 14 residents placed a total of 34 devices and removed four. All 14 program graduates completed the survey in August 2020. All but one respondent felt this training was valuable and 11 felt competent placing implants without supervision. Although 10 planned to provide implants following graduation, none have been able to, largely because of credentialing and clinic-practice level barriers.

CONCLUSIONS

The primary care Internal Medicine program graduates we surveyed (n = 14) valued our etonogestrel implant training program and perceived competence, particularly with implant placement. However, even those who intended to provide etonogestrel implants postgraduation were unable to do so.

IMPLICATIONS

Internal Medicine residents trained to place and remove etonogestrel implants are most comfortable with implant placement. However, these physicians may face barriers related to credentialing and ambulatory practice scope when attempting to provide this care in clinical practice.

Accuracy of vertical cup-to-disc ratio discrimination among clinical optometry trainees with different years of clinical experience

Purpose

Accurate evaluation of the cup-to-disc ratio is crucial for optometrists and may be influenced by their clinical experience. The study’s objective was to compare the thresholds for discriminating vertical cup-to-disc ratio (VCDR) between years 2, 3, and 4 clinical optometry trainees.

Methods

One hundred fundus photos with various VCDR sizes were selected from a clinic database. The median VCDR (0.43) photo was assigned as the standard, while the other 99 were assigned as the test photos. The participant’s task was to discriminate using a 2-alternate-forced-choice paradigm whether the test photos’ VCDR were larger or smaller than the standard VCDR. Data were fit with a Weibull function, and three discrimination thresholds were determined: the point of subjective equality (PSE), the range of VCDR uncertainty, and the ability to judge VCDR that was 0.1 unit larger than the standard photo.

Results

Year 4 trainees had better VCDR discrimination thresholds. However, the difference between the three participant groups was not statistically different for all measurements (PSE: F(2,27) = 0.43, p = 0.657; VCDR uncertainty range: F(2,27) = 0.12, p = 0.887), and thresholds for correctly discriminating VCDR 0.1 larger than the standard photo’s VCDR: F(2,27) = 0.69, p = 0.512).

Conclusion

Although Year 4 optometry trainees performed slightly better at estimating VCDR than their Year 3 and Year 2 peers, the number of years of clinical experiences did not significantly affect their VCDR discrimination thresholds when 2-dimensional fundus photos were used as stimuli.

Development and evaluation of a simulation-based mastery learning maintenance of certification course

BACKGROUND/OBJECTIVES

To develop and evaluate a post-acute care simulation-based mastery learning (SBML) continuing medical education (CME)/maintenance of certification (MOC) procedure course.

DESIGN

Pretest-posttest study of the SBML intervention.

SETTING

A 2-day post-acute care procedures course.

PARTICIPANTS

Sixteen practicing clinicians (5 physicians,11 advanced practice providers). Participants engaged in a skills pretest on knee aspiration/injection, gastrostomy tube removal/replacement, tracheostomy tube exchange, and basic suturing using a checklist created for each procedure. Participants received a didactic on each procedure followed by deliberate practice with feedback. Using the same checklists, participants completed a skills posttest and were required to meet a minimum passing standard (MPS) to obtain CME/MOC credit.

MEASUREMENTS

The MPS for each skills checklist was determined by a multidisciplinary panel of 11 experts. Participants completed surveys on procedure self-confidence and a course evaluation.

RESULTS

There was statistically significant improvement between pre- and posttests for all four procedures (p < .001). All participants were able to meet or exceed the MPS for each skill during the 2-day course. Participants' self-confidence regarding each procedure improved significantly (p < .001).

CONCLUSION

An SBML training course granting CME/MOC credit for post-acute care providers significantly improves performance of knee aspiration/injection, gastrostomy tube removal/replacement, tracheostomy tube exchange, and basic suturing.

Assessment of esophagogastroduodenoscopy skills on simulators before real-life performance

Background and study aims  Operator competency is essential for esophagogastroduodenoscopy (EGD) quality, which makes appropriate training with a final test important. The aims of this study were to develop a test for assessing skills in performing EGD, gather validity evidence for the test, and establish a credible pass/fail score. Methods  An expert panel developed a practical test using the Simbionix GI Mentor II simulator (3 D Systems) and an EGD phantom (OGI 4, CLA Medical) with a diagnostic (DP) and a technical skills part (TSP) for a prospective validation study. During the test a supervisor measured: 1) total time; 2) degree of mucosal visualization; and 3) landmarks and pathology identification. The contrasting groups standard setting method was used to establish a pass/fail score. Results  We included 15 novices (N), 10 intermediates (I), and 10 experienced endoscopists (E). The internal structure was high with a Cronbach's alpha of 0.76 for TSP time consumption and 0.74 for the identification of landmarks. Mean total times, in minutes, for the DP were N 15.7, I 11.3, and E 7.0, and for TSP., they were N 7.9, I 8.9, and E 2.9. The total numbers of identified landmarks were N 26, I 41, and E 48. Mean visualization percentages were N 80, I 71, and E 71. A pass/fail standard was established requiring identification of all landmarks and performance of the TSP in < 5 minutes. All experienced endoscopists passed, while none of the endoscopists in the other categories did. Conclusions  We established a test that can distinguish between participants with different competencies. This enables an objective and evidence-based approach to assessment of competencies in EGD.

Simulation-based Education Improves Military Trainees' Skill Performance and Self- Confidence in Tourniquet Placement: A Randomized Controlled Trial

BACKGROUND

Tactical Combat Casualty Care (TCCC) is the standard of care for stabilization and treatment of military trauma patients. The DoD has mandated that all Service members receive role-based TCCC training and certification. Simulation education can increase procedural skills by providing opportunities for deliberate practice in safe, controlled environments. We developed and evaluated the effectiveness of a simulation-based TCCC training intervention to improve participants' skill performance and self-confidence in tourniquet placement.

METHODS

This study was a single-blinded, randomized trial with waitlist controls. Army ROTC cadets from a single training battalion comprised the study population. After randomization and baseline assessment of all participants, Group A alone received focused, simulation-based TCCC tourniquet application training. Three months later, all participants underwent repeat testing, and after crossover, the waitlist Group B received the same intervention. Two months later, all cadets underwent a third/final assessment. The primary outcome was tourniquet placement proficiency assessed by total score achieved on a standardized 8-item skill checklist. A secondary outcome was self-confidence in tourniquet application skill as judged by participants' Likert scale ratings.

RESULTS

Forty-three Army ROTC cadets completed the study protocol. Participants in both Group A (n = 25) and Group B (n = 18) demonstrated significantly higher performance from baseline to final assessment at five months and two months, respectively, following the intervention. Mean total checklist score of the entire study cohort increased significantly from 5.53 (SD = 2.00) at baseline to 7.56 (SD = 1.08) at Time 3, a gain of 36.7% (p < 0.001). Both groups rated their self-confidence in tourniquet placement significantly higher following the training.

CONCLUSIONS

A simulation-based TCCC curriculum resulted in significant, consistent, and sustained improvement in participants' skill proficiency and self-confidence in tourniquet placement. Participants maintained these gains two to five months after initial training.

LEVEL OF EVIDENCE

Level II - RCT with significant difference and only one negative criterion (<80% follow-up).

Assessment of Chief Resident Practice Readiness in a Porcine Lab: A 4-Year Experience

OBJECTIVE

General surgery training prepares residents for the autonomous practice of surgery; however, assessment for readiness for independent practice presents several challenges. The simulation lab offers a safe and standardized environment for assessing the technical skills of a resident in the absence of numerous confounders of the real operating room. We describe our experience with evaluation and remediation of chief resident assessments in a porcine simulation lab.

DESIGN

Operative skill assessment of surgical residents was conducted using anesthetized porcine models. Procedure's representative of basic and complex operative skill was chosen for the assessment. Faculty assessed the residents using a checklist for the completion of all critical operative steps. A "failing" score or "critical fail" on a given procedure determined mandatory remediation. For remediation, faculty provided immediate post-procedure feedback on all errors, and residents were offered supervised practice. Residents were then retested to demonstrate competency.

SETTING

Large animal research center at Indiana University School of Medicine, Indianapolis, IN PARTICIPANTS: From 2017 to 2020, thirty-seven PGY5 residents participated in the porcine lab over a 4-year period. These general surgery residents were assessed at the beginning of their chief year.

RESULTS

There were a total of 6 residents that failed 1 or more procedures. There were no failures in the cholecystectomy, 3 failures for Nissen, 4 failures for Hand sewn anastomosis, and 1 failure for stapled anastomosis. Two residents failed 2 procedures. All residents received remediation with a faculty member and were subsequently able to perform the procedure competently.

CONCLUSIONS

A formal simulation-based assessment of procedural competence can identify technical performance deficiencies even at the chief resident level. Combined with a formal remediation program, such deficiencies can be addressed well in advance of residency graduation. Determining the relationship of such simulation-based assessments with operative performance is currently underway.

Training and assessment of musculoskeletal ultrasound and injection skills-a systematic review

OBJECTIVES

To examine how residents are trained and assessed in musculoskeletal ultrasound (MSUS), MSUS-guided and landmark-guided joint aspiration and injection. Additionally, to present the available assessment tools and examine their supporting validity evidence.

METHODS

A systematic search of PubMed, Cochrane Library, and Embase was conducted in accordance with the PRISMA guidelines and studies published from January 1, 2000 to May 31, 2021 were included. Two independent reviewers performed the search and data extraction. The studies were evaluated using the Medical Education Research Quality Instrument (MERSQI).

RESULTS

9,884 articles were screened, and 43 were included; 3 were randomized studies, 21 pre- and post-test studies, 16 descriptive studies, and 3 studies developing assessment tools. The studies used various theoretical training modalities e.g. lectures, anatomical quizzes and e-learning. The practical training models varied from mannequins and cadavers to healthy volunteers and patients. The majority of studies used subjective "comfort level" as assessment, others used practical examination and/or theoretical examination. All training programs increased trainees' self-confidence, theoretical knowledge, and/or practical performance, however few used validated assessment tools to measure the effect. Only one study met the MERSQI high methodical quality cut-off score of 14.

CONCLUSION

The included studies were heterogeneous, and most were of poor methodological quality and not based on contemporary educational theories. This review highlights the need for educational studies using validated theoretical and practical assessment tools to ensure optimal MSUS training and assessment in rheumatology.

Simulation-Based Mastery Learning Improves Ultrasound-Guided Peripheral Intravenous Catheter Insertion Skills of Practicing Nurses

INTRODUCTION

Difficult intravenous (IV) access (DIVA) is frequently encountered in the hospital setting. Ultrasound-guided peripheral IV catheter (USGPIV) insertion has emerged as an effective procedure to establish access in patients with DIVA. Despite the increased use of USGPIV, little is known about the optimal training paradigms for bedside nurses. Therefore, we developed and evaluated a novel, sustainable, USGPIV simulation-based mastery learning (SBML) curriculum for nurses.

METHODS

This is a prospective cohort study of an USGPIV SBML training program for bedside nurses over a 12-month period. We evaluated skills and self-confidence before and after training and measured the proportion of the nurses achieving independent, proctor, and instructor status. Procedure logs and surveys were used to explore the nurse experience and utilization of USGPIV on real patients with DIVA 3 months after the intervention.

RESULTS

Two hundred thirty-eight nurses enrolled in the study. The USGPIV skill checklist scores increased from median of 6.0 [interquartile range = 4.0-9.0 (pretest) to 29.0, interquartile range = 28-30 (posttest), P < 0.001]. The USGPIV confidence improved from before (mean = 2.32, SD = 1.17) to after (mean = 3.85, SD = 0.73, P < 0.001) training (5-point Likert scale). Sixty-two percent of the nurses enrolled achieved independent status, 47.5% became proctors, and 11.3% course trainers. At 3-month posttraining, the nurses had attempted a mean of 35.6 USGPIV insertions with an 89.5% success rate.

CONCLUSIONS

This novel USGPIV SBML curriculum improves nurses' insertion skills, self-confidence, and progresses patient care through USGPIV insertions on hospitalized patients with DIVA.

Current practice, education, and recommendations for training of central line insertion for trainees and fellows in adult ICUs across Australia and New Zealand

BACKGROUND

The objective of this study was to explore variability in existing training and accreditation processes for Central venous access device (CVAD) insertion among different ICUs, current practices of CVAD insertion among fellows of the College of Intensive Care Medicine (CICM) working in Australia and New Zealand (ANZ) and their recommendations for improvement.

METHODS

A prospective cross-sectional web-based survey was sent via email and CICM e-Newsletter to intensivists and directors of intensive care units (ICUs) across ANZ. All responses were tabulated, post-hoc exploratory analysis using multivariable ordinal logistic regression was used and free texts were analysed thematically and summarized.

RESULTS

115 responses were received from various public and private ICUs from all states of ANZ. 32% of the participants did not have any accreditation process for CVAD insertion skill in their ICUs, whereas 91% of respondents revealed there were no processes to assess deskilling. Most intensivists recommended supervision, simulation, various education tools, and ultrasound training to improve training and assessment. 35% of the participants inserted 0 to 5 CVADs and more than half of the intensivists had inserted <10 CVADs in one-year period. Two-thirds of the respondents recommended inserting between 6 to 20 CVADs each year to maintain competence.

CONCLUSION

The study identified wide variability in current practice, training methods and accreditation process for CVAD insertion amongst intensivists and ICU trainees in ANZ. Policy makers should consider revising the current clinical practice and training policies to new policies for accreditation and ongoing assessment for CVAD insertions across ANZ ICUs. This article is protected by copyright. All rights reserved.

Hands-On Time in Simulation-Based Ultrasound Training - A Dose-Related Response Study

Purpose Point of care ultrasound (POCUS) is widely used, but the sensitivity and specificity of the findings are highly user-dependent. There are many different approaches to ultrasound training. The aim of this study was to explore the effects of hands-on practice when learning POCUS. Methods Junior doctors with no or limited ultrasound experience were included in the study and divided into three groups. They all completed a Focused Assessment with Sonography for Trauma (FAST) course with different amounts of hands-on practice: 40 minutes (n=67), 60 minutes (n=12), and 90 minutes of hands-on time (n=27). By the end of the course, they all completed a previously validated test. Results More hands-on time improved the mean test scores and decreased the test time. The scores of the 40-, 60-, and 90-minute groups were 11.6 (SD 2.1), 12.8 (SD 2.5), and 13.7 (SD 2.5), respectively (p<0.001). The 90-minute group completed the test significantly faster than the other two groups (20 versus 26 minutes, p=0.003). A large inter-individual variation was seen. Conclusion The necessary amount of hands-on training is unknown. This study demonstrates that performance increases with prolonged hands-on time but the inter-individual variation among trainees is very large, thereby making it impossible to define the "optimal" time. This supports the use of the concept of mastery learning where each individual trainee can continue training until proficiency is reached.

Training Pulmonary Critical Care Medicine Fellows in Thoracentesis Using a Head-Mounted Video Camera

BACKGROUND

Determination of competence to perform procedures during pulmonary critical care medicine fellowship training has traditionally been based on subjective faculty opinion and numerical requirements.

OBJECTIVE

To describe an objective means of assessing competence of fellows to perform thoracentesis using a head-mounted video camera with offline scoring of the thoracentesis performed on an actual patient.

METHODS

To test competence in performance of thoracentesis after a multimodality training program, a total of eight first-year fellows performed a thoracentesis on an actual patient while recording the procedure with a lightweight head-mounted video camera in 2017 and 2018. The recordings were scored offline by two faculty members using a 30-point checklist. The percentage agreement between scorers was measured, as was the opinion of the fellows and the scorers on the testing process. If a fellow failed completion of all checklist items, they were provided with further training and retested to assure competence. As part of their training, fellows reviewed the video record of their procedures.

RESULTS

Eight first-year fellows were tested, of whom seven successfully completed key checklist items as determined by the video scorers. One failing fellow passed after further training and testing. The percentage agreement between the scorers was high, and fellows indicated that the video device was useful for training.

CONCLUSION

This study supports the use of video-based testing for assessment of competence and for training in performance of thoracentesis by fellows.

Cohort study of hospitalists' procedural skills: baseline competence and durability after simulation-based training

OBJECTIVES

Hospitalists are expected to be competent in performing bedside procedures, which are associated with significant morbidity and mortality. A national decline in procedures performed by hospitalists has prompted questions about their procedural competency. Additionally, though simulation-based mastery learning (SBML) has been shown to be effective among trainees whether this approach has enduring benefits for independent practitioners who already have experience is unknown. We aimed to assess the baseline procedural skill of hospitalists already credentialed to perform procedures. We hypothesised that simulation-based training of hospitalists would result in durable skill gains after several months.

DESIGN

Prospective cohort study with pretraining and post-training measurements.

SETTING

Single, large, urban academic medical centre in the USA.

PARTICIPANTS

Twenty-two out of 38 eligible participants defined as hospitalists working on teaching services where they would supervise trainees performing procedures.

INTERVENTIONS

One-on-one, 60 min SBML of lumbar puncture (LP) and abdominal paracentesis (AP).

PRIMARY AND SECONDARY OUTCOME MEASURES

Our primary outcome was the percentage of hospitalists obtaining minimum passing scores (MPS) on LP and AP checklists; our secondary outcomes were average checklist scores and self-reported confidence.

RESULTS

At baseline, only 16% hospitalists met or exceeded the MPS for LP and 32% for AP. Immediately after SBML, 100% of hospitalists reached this threshold. Reassessment an average of 7 months later revealed that only 40% of hospitalists achieved the MPS. Confidence increased initially after training but declined over time.

CONCLUSIONS

Hospitalists may be performing invasive bedside procedures without demonstration of adequate skill. A single evidence-based training intervention was insufficient to sustain skills for the majority of hospitalists over a short period of time. More stringent practices for certifying hospitalists who perform risky procedures are warranted, as well as mechanisms to support skill maintenance, such as periodic simulation-based training and assessment.

Virtual reality simulator metrics cannot be used to assess competence in ureteronephroscopy and stone removal - a validation study

OBJECTIVE

The growing use of simulation-based training makes it necessary to develop efficient training programs in order to ensure optimal use of time and resources. Our aim was to develop and gather validity evidence for a simulation-based test in ureteronephroscopy and set a pass/fail standard for the test that will allow future mastery learning.

DESIGN

This study is a validation study. A test in ureteronephroscopy and stone removal on the URO Mentor™ virtual reality simulator (3D Systems, USA) was developed by two experienced urologists in order to ensure content. Participants with different experience completed three standardized tasks on the simulator and simulator-generated metrics were used as outcome parameters to minimize bias and ensure a fair response process.

RESULTS

Twenty novices, 15 intermediates, and 8 experienced urologists were included in the study. Validity evidence for internal structure and relationship to other variables was questionable with weak and mostly insignificant correlations across all four metrics (Cronbach's alpha = 0.14, p = 0.15) and across the three modules (Cronbach's alpha = 0.41 (p = 0.02), 0.35 (p = 0.06), 0.10 (p = 0.35), and 0.30 (p = 0.09) for each metric, respectively). It was not possible to establish a pass/fail score for the simulation test with meaningful consequences.

CONCLUSION

Our study showed that automatically generated simulator metrics cannot be used as a valid way of assessing competence in ureteronephroscopy. Virtual-reality simulator training could still be a valuable and patient-safe way to practice these skills, but an experienced supervisor is needed to determine when the trainee is ready to continue to supervised practice on patients.

Mastery learning improves simulated central venous catheter insertion by emergency medicine teaching faculty

OBJECTIVES

Routine competency assessments of procedure skills, such as central venous catheter (CVC) insertion, do not occur beyond residency training. Evidence suggests variable, suboptimal attending physician procedure skills. Our study aimed to assess CVC insertion skill by academic emergency physicians, determine whether a simulation-based mastery learning (SBML) intervention improves performance and investigate for variables that predict competence.

METHODS

This is a pretest-posttest study that evaluated simulated CVC insertion by emergency medicine (EM) faculty physicians. We assessed 44 volunteer participants at a large academic medical center over a 1-month period using a published 29-item checklist. Our primary outcome was the difference in assessment score before and after a SBML intervention. A secondary analysis evaluated predictors of pretest performance.

RESULTS

A total of 44 subjects participated. Only four of 44 (9.1%) of subjects met a predefined minimum passing score on pretest. Mean assessment scores increased by 21.5% following the SBML intervention (95% confidence interval [CI] of the difference = 18.1% to 24.8%, p < 0.001). In a regression model, pretest scores increased by 10.8% (95% CI = 2.9 to 18.7%, p = 0.009) if subjects completed postgraduate training within 5 years. Frequency of CVC insertion did not predict performance, but 25 of 44 (56.8%) faculty members had no documented performance or supervision of a CVC insertion within 1 year of assessment.

CONCLUSIONS

SBML is a promising method to assess and improve CVC insertion performance by EM faculty physicians. Recent completion of postgraduate training was a significant predictor of CVC insertion performance. Our results require validation in larger cohorts of EM physicians across other academic institutions.

Development of a Rigorously Designed Procedural Checklist for Assessment of Emergency Medicine Resident Performance of Temporary Transvenous Cardiac Pacing

OBJECTIVES

Temporary transvenous cardiac pacing (TVP) is a potentially lifesaving intervention included in the list of essential core procedures for emergency medicine (EM) training; however, opportunities to perform TVP during residency cannot be guaranteed. EM graduates report feeling subjectively underprepared for this procedure, but objective performance data are lacking. Checklist-based simulated assessment is an increasing focus of competency-based medical education, particularly for invasive procedures like TVP. The objectives of this paper were as follows: first, to enlist a multidisciplinary team of experts to create an assessment tool for TVP using best practices in checklist development; second, to determine the reliability of checklist scoring; and third, to assess EM residents' baseline ability to perform TVP using a dedicated task trainer.

METHODS

This study was conducted at a single 4-year EM residency. A panel of emergency physicians and cardiologists designed a TVP checklist using a modified Delphi approach. After consensus was achieved on a final checklist, EM residents were assessed using a dedicated TVP task trainer. Inter-rater reliability was determined using Cohen's kappa coefficient. Resident performance was determined by number of correctly performed checklist items.

RESULTS

The expert panel achieved consensus on a 30-item checklist after three rounds of revisions. The Cohen's kappa coefficient for the overall checklist score was 0.87, with individual checklist items ranging from 0.63 to 1.00. In total, 58 residents were assessed with a mean score of 13.5 of 30 checklist items. Scores increased with each year of training.

CONCLUSIONS

This study details the rigorous development of a TVP checklist designed by a multidisciplinary team of experts. Checklist scores demonstrated strong inter-rater reliability. The overall poor performance of this cohort suggests the current approach to TVP training does not provide sufficient preparation for EM residents. Competency-based techniques, such as simulation-based mastery learning, should be explored.

See One, Do One, Forget One: Early Skill Decay After Paracentesis Training

INTRODUCTION

Internal medicine residents perform paracentesis, but programs lack standard methods for assessing competence or maintenance of competence and instead rely on number of procedures completed. This study describes differences in resident competence in paracentesis over time.

METHODS

From 2016 to 2017, internal medicine residents (n = 118) underwent paracentesis simulation training. Competence was assessed using the Paracentesis Competency Assessment Tool (PCAT), which combines a checklist, global scale, and entrustment score. The PCAT also delineates two categorical cut-point scores: the Minimum Passing Standard (MPS) and the Unsupervised Practice Standard (UPS). Residents were randomized to return to the simulation lab at 3 and 6 months (group A, n = 60) or only 6 months (group B, n = 58). At each session, faculty raters assessed resident performance. Data were analyzed to compare resident performance at each session compared with initial training scores, and performance between groups at 6 months.

RESULTS

After initial training, all residents met the MPS. The number achieving UPS did not differ between groups: group A = 24 (40%), group B = 20 (34.5%), p = 0.67. When group A was retested at 3 months, performance on each PCAT component significantly declined, as did the proportion of residents meeting the MPS and UPS. At the 6-month test, residents in group A performed significantly better than residents in group B, with 52 (89.7%) and 20 (34.5%) achieving the MPS and UPS, respectively, in group A compared with 25 (46.3%) and 2 (3.70%) in group B (p < .001 for both comparison).

DISCUSSION

Skill in paracentesis declines as early as 3 months after training. However, retraining may help interrupt skill decay. Only a small proportion of residents met the UPS 6 months after training. This suggests using the PCAT to objectively measure competence would reclassify residents from being permitted to perform paracentesis independently to needing further supervision.

Procedural Competency Among Hospitalists: A Literature Review and Future Considerations

BACKGROUND

As general internists practicing in the inpatient setting, hospitalists at many institutions are expected to perform invasive bedside procedures, as defined by professional standards. In reality, hospitalists are doing fewer procedures and increasingly are referring to specialists, which threatens their ability to maintain procedural skills. The discrepancy between expectations and reality, especially when hospitalists may be fully credentialed to perform procedures, poses significant risks to patients because of morbidity and mortality associated with complications, some of which derive from practitioner inexperience.

METHODS

We performed a structured search of the peer-reviewed literature to identify articles focused on hospitalists performing procedures.

RESULTS

Our synthesis of the literature characterizes contributors to hospitalists' procedural competency and discusses: (1) temporal trends for procedures performed by hospitalists and their associated referral patterns, (2) data comparing use and clinical outcomes of procedures performed by hospitalists compared with specialists, (3) the lack of nationwide standardization of hospitalist procedural training and credentialing, and (4) the role of medical procedure services, although limited in supportive evidence, in concentrating procedural skill and mitigating risk in the hands of a few well-trained hospitalists.

CONCLUSION

We conclude with recommendations for hospital medicine groups to ensure the safety of hospitalized patients undergoing bedside procedures.

Simulation-Based Assessments and Graduating Neurology Residents' Milestones: Status Epilepticus Milestones

BACKGROUND

The American Board of Psychiatry and Neurology and the Accreditation Council for Graduate Medical Education (ACGME) developed Milestones that provide a framework for residents' assessment. However, Milestones do not provide a description for how programs should perform assessments.

OBJECTIVES

We evaluated graduating residents' status epilepticus (SE) identification and management skills and how they correlate with ACGME Milestones reported for epilepsy and management/treatment by their program's clinical competency committee (CCC).

METHODS

We performed a cohort study of graduating neurology residents from 3 academic medical centers in Chicago in 2018. We evaluated residents' skills identifying and managing SE using a simulation-based assessment (26-item checklist). Simulation-based assessment scores were compared to experience (number of SE cases each resident reported identifying and managing during residency), self-confidence in identifying and managing these cases, and their end of residency Milestones assigned by a CCC based on end-of-rotation evaluations.

RESULTS

Sixteen of 21 (76%) eligible residents participated in the study. Average SE checklist score was 15.6 of 26 checklist items correct (60%, SD 12.2%). There were no significant correlations between resident checklist performance and experience or self-confidence. The average participant's level of Milestone for epilepsy and management/treatment was high at 4.3 of 5 (SD 0.4) and 4.4 of 5 (SD 0.4), respectively. There were no significant associations between checklist skills performance and level of Milestone assigned.

CONCLUSIONS

Simulated SE skills performance of graduating neurology residents was poor. Our study suggests that end-of-rotation evaluations alone are inadequate for assigning Milestones for high-stakes clinical skills such as identification and management of SE.

Short-term Retention of Patient and Caregiver Ventricular Assist Device Self-care Skills after Simulation-based Mastery Learning

Background

We developed a simulation-based mastery learning (SBML) curriculum that boosted self-care skills for patients with a ventricular assist device (VAD). In this study, we evaluated short-term skills retention.

Methods

We assessed skill retention among patients and caregivers who participated in VAD self-care SBML at a tertiary care center. We compared discharge skills tests (immediately after completing SBML) to 1- and 3-month follow-up tests to assess skill retention.

Results

Fifteen patients and 15 caregivers completed discharge and follow-up testing. Skills were largely retained at 1- and 3-months.

Conclusions

SBML results in short-term retention of VAD self-care skills.

Resident competencies before and after short intensive care unit rotations: a multicentre pilot observational study

PURPOSE

Residency programs need to understand the competencies developed by residents during an intensive care unit (ICU) rotation, so that curricula and assessments maximize residents' learning. The primary study objective was to evaluate the feasibility for training programs and acceptability by residents of conducting a multi-competency assessment during a four-week ICU rotation.

METHODS

We conducted a prospective, multicentre observational pilot study in three ICUs. During weeks 1 and 4 of an ICU rotation, we conducted repeated standardized assessments of non-critical care specialty residents' competencies in cognitive reasoning (script concordance test [SCT]), procedural skills (objective structured assessment of technical skills [OSATS]-global rating scale], and communication skills through a written test, two procedural simulations, and a simulated encounter with a "family member". The feasibility outcomes included program costs, the proportion of enrolled residents able to complete at least one three-station assessment during their four-week ICU rotation, and acceptability of the assessment for the trainees.

RESULTS

We enrolled 63 (69%) of 91 eligible residents, with 58 (92%) completing at least one assessment. The total cost to conduct 90 assessments was CAD 33,800. The majority of participants agreed that the assessment was fair and that it measured important clinical abilities. For the 32 residents who completed two assessments, the mean (standard deviation) cognitive reasoning and procedural skill scores increased between weeks 1 and 4 [SCT difference, 3.1 (6.5), P = 0.01; OSATS difference for bag-mask ventilation and central line insertion, 0.4 (0.5) and 0.6 (0.8), respectively; both P ≤ 0.001]. Nevertheless, the communication scores did not change significantly.

CONCLUSIONS

A monthly multi-competency assessment for specialty residents rotating in the ICU is likely feasible for most programs with appropriate resources, and generally acceptable for residents. Specialty residents' cognitive reasoning and procedural skills may improve during a four-week ICU rotation, whereas communication skills may not.

Impact of Simulation-based Mastery Learning on Resident Skill Managing Mechanical Ventilators

Background: Caring for patients requiring mechanical ventilation is complex, and residents may lack adequate skill for managing these patients. Simulation-based mastery learning (SBML) is an educational model that trains clinicians to a high standard and can reduce complications. The mastery learning model has not been applied to ventilator management. Objective: The purpose of this study was to determine whether SBML, as compared with traditional training, is an effective strategy for teaching residents the skills necessary to manage patients requiring mechanical ventilation. Methods: We developed an SBML curriculum and a 47-item skills checklist to test ventilator management for patients with normal, restricted, and obstructed lung physiology. A minimum passing standard (MPS) on the checklist was set using the Mastery Angoff method. Residents rotating through the medical intensive care unit in Academic Year 2017-2018 were assigned to SBML or traditional training based on their medical intensive care unit team. The SBML group was pretested on a ventilator simulator using the skills checklist. They then received a 1.5-hour session (45 min didactic and 45 min deliberate practice on the simulator with feedback). At rotation completion, they were posttested on the simulator using the checklist until the MPS was met. Both SBML-trained and traditionally trained groups received teaching during daily bedside rounds and twice weekly didactic lectures. At rotation completion, traditionally trained residents were tested using the same skills checklist on the simulator. We compared pretest and posttest performance among SBML-trained residents and end of the rotation test performances between the SBML-trained and traditionally trained residents. Results: The MPS was set at 87% on the checklist. Fifty-seven residents were assigned to the SBML-trained group and 49 were assigned to the traditionally trained group. Mean checklist scores for SBML-trained residents improved from 51.4% (standard deviation [SD] = 17.5%) at pretest to 86.1% (SD = 7.6%) at initial posttest and 92.5% (SD = 3.7%) at final (mastery) posttest (both P < 0.001). Forty-two percent of residents required more than one attempt at the posttest to meet or exceed the MPS. At rotation completion, the traditionally trained residents had a mean test score of 60.9% (SD = 13.3%). Conclusion: SBML is an effective strategy to train residents on mechanical ventilator management. An SBML curriculum may augment traditional training methods to further equip residents to safely manage ventilated patients.

Simulation-Based Medical Education Improves Procedural Confidence in Core Invasive Procedures for Military Internal Medicine Residents

Introduction

The American Board of Internal Medicine (ABIM) requires that trainees receive procedural training for certification; however, Internal Medicine (IM) residents perform a variable number of procedures throughout residency training. This results in differences in confidence levels as well as procedural competence. For active-duty military trainees, this is especially problematic, as these procedural skills are often required during deployment soon after residency graduation. This deficit can be improved through standardized simulation-based training.

Methods

All internal medicine residents at our institution were invited to participate in a standardized simulation-based training program for core internal medicine procedures (lumbar puncture, arterial line, central line, thoracentesis, paracentesis, and arthrocentesis). Residents were asked to qualitatively rate their perceived procedural confidence using a Likert scale ranging from 1 (not at all confident) to 5 (extremely confident) in their ability to independently perform core internal medicine procedures prior to the simulation exercise. Experienced senior residents and internal medicine faculty instructed and supervised each resident as they performed the procedures. Following the simulation exercise, the residents repeated the survey and were asked to report whether or not they found the exercise useful. 

Results

Of the 96 residents invited to participate, 49 completed the pre-simulation questionnaire and 36 completed the post-simulation questionnaire. The cumulative mean Likert scale confidence rating for all procedures showed a statistically significant improvement post-simulation as compared to pre-simulation, including lumbar puncture (2.45±1.1 vs. 3.42±0.87, p<0.05), arterial line (2.48±1.06 vs. 3.39±1.04, p < 0.05), central line (2.86±1.08 vs. 3.5±1.02, p < 0.05), thoracentesis (2.67±1.10 vs. 3.64±0.83, p < 0.05), paracentesis (3.1±1.08 vs. 3.82±0.74, p < 0.05), and arthrocentesis (2.56±1.07 vs. 3.67±0.80, p < 0.05). All (36/36) trainees reported that they perceived the simulation exercise as valuable.

Conclusion

Internal medicine residents across all post-graduate year (PGY) levels at our institution lacked confidence to independently perform core internal medicine procedures. Utilizing simulation-based medical education as an adjunct to clinical training is well accepted by internal medicine trainees, and resulted in significantly improved procedural confidence. This intervention was well received by trainees and could feasibly be replicated at other active-duty military internal medicine residency programs to assist with readiness. Research is currently in progress to correlate in-situ competency and evaluate clinical outcomes of this improved confidence.

Expert Consensus Recommendations for Robotic Surgery Credentialing

OBJECTIVE

To define criteria for robotic credentialing using expert consensus.

BACKGROUND

A recent review of institutional robotic credentialing policies identified significant variability and determined current policies are largely inadequate to ensure surgeon proficiency and may threaten patient safety.

METHODS

28 national robotic surgery experts were invited to participate in a consensus conference. After review of available institutional policies and discussion, the group developed a 91 proposed criteria. Using a modified Delphi process the experts were asked to indicate their agreement with the proposed criteria in three electronic survey rounds after the conference. Criteria that achieved 80% or more in agreement (consensus) in all rounds were included in the final list.

RESULTS

All experts agreed that there is a need for standardized robotic surgery credentialing criteria across institutions that promote surgeon proficiency. 49 items reached consensus in the first round, 19 in the second, and 8 in the third for a total of 76 final items. Experts agreed that privileges should be granted based on video review of surgical performance and attainment of clearly defined objective proficiency benchmarks. Parameters for ongoing outcome monitoring were determined and recommendations for technical skills training, proctoring, and performance assessment were defined.

CONCLUSIONS

Using a systematic approach, detailed credentialing criteria for robotic surgery were defined. Implementation of these criteria uniformly across institutions will promote proficiency of robotic surgeons and has the potential to positively impact patient outcomes.

Use of a simulation-based mastery learning curriculum to improve ultrasound-guided vascular access skills of pediatric anesthesiologists

BACKGROUND

Pediatric vascular access is inherently challenging due to the small caliber of children's vessels. Ultrasound-guided intravenous catheter insertion has been shown to increase success rates and decrease time to cannulation in patients with difficult intravenous access. Although proficiency in ultrasound-guided intravenous catheter insertion is a critical skill in pediatric anesthesia, there are no published competency-based training curricula.

AIMS

The objective of this study was to evaluate the performance of pediatric anesthesiologists who participated in a novel ultrasound-guided intravenous catheter insertion simulation-based mastery learning curriculum.

METHODS

Pediatric anesthesia attendings, fellows, and rotating residents participated in the ultrasound-guided intravenous catheter insertion simulation-based mastery learning curriculum from August 2019 to February 2020. The 2-hour curriculum consisted of participants first undergoing a simulated skills pretest followed by watching a video on ultrasound-guided intravenous catheter insertion and deliberate practice on a simulator. Subsequently, all participants took a post-test and were required to meet or exceed a minimum passing standard. Those who were unable to meet the minimum passing standard participated in further practice until they could be retested and met this standard. We compared pre to post-test ultrasound-guided intravenous catheter insertion skills and self-confidence before and after participation in the curriculum.

RESULTS

Twenty-six pediatric anesthesia attendings, 12 fellows, and 38 residents participated in the curriculum. At pretest, 16/76 (21%) participants were able to meet or exceed the minimum passing standard. The median score on the pretest was 21/25 skills checklist items correct and improved to 24/25 at post-test (95% CI 3.0-4.0, P < .01). Self-confidence significantly improved after the course from an average of 3.2 before the course to a postcourse score of 3.9 (95% CI 0.5-0.9, P < .01; 1 = Not all confident, 5 = Very confident).

CONCLUSIONS

Simulation-based mastery learning significantly improved anesthesiologists' ultrasound-guided intravenous catheter insertion performance in a simulated setting.

Using structured progress to measure competence in flexible bronchoscopy

Background

Flexible bronchoscopy is a core invasive procedure in pulmonary medicine and training in the procedure is mandatory. Diagnostic completeness and procedure time have been identified as useful measures of competence. No outcome measures have been developed regarding navigational path in bronchoscopy to assess whether the bronchial segments have been identified in an arbitrary or structured order. We investigated whether a new outcome measure for structured progression could be used to assess competency in flexible bronchoscopy.

Methods

The study was designed as a prospective comparative study. Twelve novices, eleven intermediates, and ten expert bronchoscopy operators completed three full bronchoscopies in a simulated setting on a phantom. The following outcome measures were collected through a checklist evaluation by a trained rater: Diagnostic Completeness as amount of visualized bronchial segments, Structured Progress between the bronchial segments in ascending order, and average intersegmental time (AIT).

Results

The ability to follow a structured ascending path through the bronchial tree correlated with a higher amount of identified bronchial segments (Pearson’s correlation, r=0.62, P<0.001) and a lower AIT (Pearson’s correlation, r=−0.52, P<0.001).

Conclusions

Operators should advance through the bronchial tree in a structured ascending order to ensure systematic progress with the highest level of diagnostic yield and the lowest procedure time. Structured progression is a useful measure to evaluate competency in flexible bronchoscopy.

The Evolution of Assessment: Thinking Longitudinally and Developmentally

Becoming a physician or other health care professional is a complex and intensely developmental process occurring over a prolonged period of time. The learning path for each medical student, resident, and fellow varies due to different individual learner abilities and curricular designs, clinical contexts, and assessments used by the training program. The slow and uneven evolution to outcomes-based medical education is partly the result of inadequate approaches to programmatic assessment that do not fully address all essential core competencies needed for practice or account for the developmental nature of training. Too many assessments in medical education still focus on single point-in-time performance or function as indirect proxies for actual performance in clinical care for patients and families.Milestones are a modest first step of providing predictive, longitudinal data on a national scale. Longitudinal Milestones data can facilitate the continuous improvement efforts of programs in assessment. However, Milestone judgments are only as good as the assessment data and group processes that inform them. Programmatic assessment should be longitudinally focused and provide all learners with comprehensive and actionable data to guide their professional development and support creation of meaningful individualized action plans. Efforts are urgently needed to rebalance programmatic assessment away from an overreliance on assessment proxies toward more effectively using developmentally focused work-based assessments, routinely incorporate clinical performance and patient experience data, and partner with learners through iterative coproduced assessment activities.

A call for collaboration and consensus on training for endotracheal intubation in the medical intensive care unit

Endotracheal intubation (EI) is a potentially lifesaving but high-risk procedure in critically ill patients. While the ACGME mandates that trainees in pulmonary and critical care medicine (PCCM) achieve competence in this procedure, there is wide variation in EI training across the USA. One study suggests that 40% of the US PCCM trainees feel they would not be proficient in EI upon graduation. This article presents a review of the EI training literature; the recommendations of a national group of PCCM, anesthesiology, emergency medicine, and pediatric experts; and a call for further research, collaboration, and consensus guidelines.