Procedural training and assessment of competency utilizing simulation

The Power of Faculty Development: The Impact on Teaching a Procedural Skill Framework

BACKGROUND

Teaching outpatient procedures is a skill often overlooked in faculty development. This oversight may lead to faculty employing a haphazard approach. Competency in procedural skills is inherent, and acquiring proficiency in procedural skills is necessary across all medical specialties, with some centers moving toward a blended simulation-based approach rather than the traditional Halstedian "see one, do one, teach one" mantra. While both formats have their pros and cons, they share the unifying concept of performance-based assessments and a standardized method for teaching procedures, which has typically been lacking a formal framework.

OBJECTIVE

This study aimed to implement and evaluate the impact of teaching an educational technique in a multidisciplinary faculty education workshop about the Sawyer framework for psychomotor skill acquisition.

METHODS

 An interactive 90-minute workshop through the Uniformed Services University Faculty Development Program was developed and presented from February 2021 to October 2023 at multiple military treatment facilities. Participants enrolled in the workshop either by online registration or by walking in on the day of the workshop. A postworkshop survey was collected voluntarily. Through the survey, participants self-evaluated their current teaching strategy and made changes to their future strategy based on the framework they learned during the workshop. This was a mixed methods approach with quantitative survey data that were analyzed using Microsoft Excel (Microsoft Corporation, Redmond, WA) and qualitative data through thematic analysis using a constructivist inductive approach.

RESULTS

 There were 52 sessions with a total of 570 participants across 22 unique specialties. The response rate was 50%. Before the workshop, 22% of responding participants had no teaching strategy, and 49% had a partial but not explicit strategy for teaching. After the workshop, 89% of respondents answered that they would either implement a new or modify an existing strategy. Ninety-three percent of respondents reported that the Sawyer method was applicable to their future teaching. The overall themes from participants were that this procedural framework allowed for personal improvement in clear communication, individualized learner-centered teaching, and improved intentionality of teaching procedures.

CONCLUSION

 Almost two-thirds of the faculty did not have a formal teaching method before this course, which is consistent with current data. Implementing a standardized framework for teaching procedures through faculty development workshops for multidisciplinary medical faculty educators can improve the educational quality of procedural skills.

Trends in Procedures in the Neonatal Intensive Care Unit

OBJECTIVE

The aim of the study is to identify the rates and trends of various procedures performed on newborns.

STUDY DESIGN

The Healthcare Cost and Utilization Project (HCUP) database for the years 2002 to 2015 was queried for the number of livebirths, and various procedures using International Classification of Diseases, Ninth Revision (ICD-9) codes. These were adjusted to the rate of livebirths in each particular year. A hypothetical high-volume hospital based on data from the last 5 years was used to estimate the frequency of each procedure.

RESULTS

Over the study period, there was a decline in the rates of exchange transfusions and placement of arterial catheters. There was an increase in the rates of thoracentesis, abdominal paracentesis, placement of umbilical venous catheter (UVC) lines, and central lines with ultrasound or fluoroscopic guidance. No change was observed in the rates of unguided central lines, pericardiocentesis, bladder aspiration, intubations, and LP. Intubations were the most performed procedures. Placement of UVC, central venous lines (including PICCs), arterial catheters, and LP were relatively common, whereas others were rare such as pericardiocentesis and paracentesis.

CONCLUSION

Some potentially lifesaving procedures are extremely rare or decreasing in incidence. There has also been an increase in utilization of fluoroscopic/ultrasound guidance for the placement of central venous catheters.

KEY POINTS

· Advances in neonatal care have impacted the number of procedures performed in the NICU.. · It is unclear whether invasive procedures occur at rates sufficient for adequate training and maintenance of skills.. · Understanding the NICU procedural trends is important in designing simulation and competency-based medical education programs..

In Situ Simulation and Clinical Outcomes in Infants Born Preterm

OBJECTIVE

To evaluate impact of a multihospital collaborative quality improvement project implementing in situ simulation training for neonatal resuscitation on clinical outcomes for infants born preterm.

STUDY DESIGN

Twelve neonatal intensive care units were divided into 4 cohorts; each completed a 15-month long program in a stepped wedge manner. Data from California Perinatal Quality Care Collaborative were used to evaluate clinical outcomes. Infants with very low birth weight between 22 through 31 weeks gestation were included. Primary outcome was survival without chronic lung disease (CLD); secondary outcomes included intubation in the delivery room, delivery room continuous positive airway pressure, hypothermia (<36°C) upon neonatal intensive care unit admission, severe intraventricular hemorrhage, and mortality before hospital discharge. A mixed effects multivariable regression model was used to assess the intervention effect.

RESULTS

Between March 2017 and December 2020, a total of 2626 eligible very low birth weight births occurred at 12 collaborative participating sites. Rate of survival without CLD at participating sites was 74.1% in March to August 2017 and 76.0% in July to December 2020 (risk ratio 1.03; [0.94-1.12]); no significant improvement occurred during the study period for both participating and nonparticipating sites. The effect of in situ simulation on all secondary outcomes was stable.

CONCLUSIONS

Implementation of a multihospital collaborative providing in situ training for neonatal resuscitation did not result in significant improvement in survival without CLD. Ongoing in situ simulations may have an impact on unit practice and unmeasured outcomes.

Competency-based assessment in neonatal simulation-based training

Simulation is a cornerstone of training in neonatal clinical care, allowing learners to practice skills in a safe and controlled environment. Competency-based assessment provides a systematic approach to evaluating technical and behavioral skills observed in the simulation environment to ensure the learner is prepared to safely perform the skill in a clinical setting. Accurate assessment of competency requires the creation of tools with evidence of validity and reliability. There has been considerable work on the use of competency-based assessment in the field of neonatology. In this chapter, we review neonatal simulation-based training, examine competency-based assessment tools, explore methods to gather evidence of the validity and reliability, and review an evidence-based approach to competency-based assessment using simulation.

Simulation-Based Education for Enhancing Obstetric Emergency Response: A Needs Impact Evaluation

Simulation is an ideal method for procedural training in obstetrics. To maximise training opportunities through simulation, the evaluation of these educational activities should be based on a standardised evidence-based approach. As such, the tools used in the evaluative process should be validated for content and context, as this ensures consistency of approach. It also makes the findings and recommendations acceptable, applicable and credible. More so, the information can be used for planning further learning, assessment of the competency of the trainers and educational governance purposes. In our view, simulation should be used in conjunction with other forms of procedural assessment such as mini-clinical examinations and case-based discussions to translate skills to actual life events. The learners will be able to further consolidate their learning, improve professional skills and feel involved throughout the programme.

How Good is Good Enough?: Current-Day Pediatric Residency Program Directors' Challenges in Assessing and Achieving Resident Procedural Competency

OBJECTIVE

Significant gaps exist in the pediatric resident (PR) procedural experience. Graduating PRs are not achieving competency in the 13 ACGME recommended procedures. It is unclear why PR are not able to achieve competency, or how existing gaps may be addressed.

METHODS

We performed in-depth one-on-one semistructured interviews with 12 pediatric residency program directors (PPDs). The interviews were audio-recorded, and transcribed verbatim. Coding of the data using conventional content analysis led to generation of categories, which were validated through consensus development.

RESULTS

We identified 4 main categories, including (1) programs struggle to ensure adequate training in procedural skills for PRs, with various barriers reported; (2) programs develop individualized strategies to address challenges in procedural skills training, and multiple options are necessary; (3) PPDs face challenges defining procedural competency and standardizing expectations; and (4) expectations for PR procedural training may require modification based upon current practice environments. Solutions include simulation, procedural boot camps, and procedural/subspecialty electives.

CONCLUSIONS

Numerous methods to combat challenges in PR procedural training have been identified by participating PPDs, including simulation, tailoring electives, and developing institutional guidelines. However, accreditation bodies may need to update procedural expectations based on individual resident career goals and realities of current day practice.

Advanced neonatal procedural skills: a simulation-based workshop: impact and skill decay

BACKGROUND

Trainees aiming to specialize in Neonatal Perinatal Medicine (NPM), must be competent in a wide range of procedural skills as per the Royal College of Canada. While common neonatal procedures are frequent in daily clinical practice with opportunity to acquire competence, there are substantial gaps in the acquisition of advanced neonatal procedural skills. With the advent of competency by design into NPM training, simulation offers a unique opportunity to acquire, practice and teach potentially life-saving procedural skills. Little is known on the effect of simulation training on different areas of competence, and on skill decay.

METHODS

We designed a unique simulation-based 4-h workshop covering 6 advanced procedures chosen because of their rarity yet life-saving effect: chest tube insertion, defibrillation, exchange transfusion, intra-osseus (IO) access, ultrasound-guided paracentesis and pericardiocentesis. Direct observation of procedural skills (DOPS), self-perceived competence, comfort level and cognitive knowledge were measured before (1), directly after (2), for the same participants after 9-12 months (skill decay, 3), and directly after a second workshop (4) in a group of NPM and senior general pediatric volunteers.

RESULTS

The DOPS for all six procedures combined for 23 participants increased from 3.83 to 4.59. Steepest DOPS increase pre versus post first workshop were seen for Defibrillation and chest tube insertion. Skill decay was evident for all procedures with largest decrease for Exchange Transfusion, followed by Pericardiocentesis, Defibrillation and Chest Tube. Self-perceived competence, comfort and cognitive knowledge increased for all six procedures over the four time points. Exchange Transfusion stood out without DOPS increase, largest skill decay and minimal impact on self-assessed competence and comfort. All skills were judged as better by the preceptor, compared to self-assessments.

CONCLUSIONS

The simulation-based intervention advanced procedural skills day increased preceptor-assessed directly observed procedural skills for all skills examined, except exchange transfusion. Skill decay affected these skills after 9-12 months. Chest tube insertions and Defibrillations may benefit from reminder sessions, Pericardiocentesis may suffice by teaching once. Trainees' observed skills were better than their own assessment. The effect of a booster session was less than the first intervention, but the final scores were higher than pre-intervention.

TRIAL REGISTRATION

Not applicable, not a health care intervention.

Moral Distress and Burnout in NICU Nurses

The objectives of this study were to factor analyze the Moral Distress Scale-Revised (MDS-R) in NICU nurses and to evaluate the relationships among dimensions of the MDS-R and the demoralization, exhaustion, and loss of motive dimensions of the Burnout Measure (BM). A total of 142 NICU nurses completed modified pen-and-paper versions of the MDS-R and BM. Exploratory and confirmatory factor analyses showed that the MDS-R-14 was a relatively good fit for the data. The compromised care dimension predicted BM demoralization (β = 0.24) and exhaustion (β = 0.22), the futile care dimension predicted BM exhaustion (β = 0.18), and the untruthful care dimension predicted BM demoralization (β = .25). Strategies to mitigate moral distress and resulting burnout in NICU nurses should address futile care, compromised care, and untruthful care.

Simulation in neonatal care: towards a change in traditional training?

Simulation has traditionally been used in neonatal medicine for educational purposes which include training of novice learners, maintaining competency of health care providers, and training of multidisciplinary teams to handle crisis situations such as neonatal resuscitation. Current guidelines recommend the use of simulation as an education tool in neonatal practice. The place of simulation-based education has gradually expanded, including in limited resource settings, and is starting to show its impact on improving patient outcomes on a global basis. Over the past years, simulation has become a cornerstone in clinical settings with the goal of establishing high quality, safe, reliable systems. The aim of this review is to describe neonatal simulation training as an effective tool to improve quality of care and patient outcomes, and to encourage the use of simulation-based training in the neonatal intensive care unit (NICU) for not only education, but equally for team building, risk management and quality improvement.

CONCLUSION

Simulation is a promising tool to improve patient safety, team performance, and ultimately patient outcomes, but scarcity of data on clinically relevant outcomes makes it difficult to estimate its real impact. The integration of simulation into the clinical reality with a goal of establishing high quality, safe, reliable, and robust systems to improve patient safety and patient outcomes in neonatology must be a priority.

WHAT IS KNOWN

• Simulation-based education has traditionally focused on procedural and technical skills. • Simulation-based training is effective in teaching non-technical skills such as communication, leadership, and teamwork, and is recommended in neonatal resuscitation.

WHAT IS NEW

• There is emerging evidence for the impact of simulation-based training on patient outcomes in neonatal care, but data on clinically relevant outcomes are scarce. • Simulation is a promising tool for establishing high quality, safe, reliable, and robust systems to improve patient safety and patient outcomes.

Intubation Competence During Neonatal Fellowship Training

OBJECTIVES

To characterize neonatal-perinatal medicine fellows' progression toward neonatal intubation procedural competence during fellowship training.

METHODS

Multi-center cohort study of neonatal intubation encounters performed by neonatal-perinatal medicine fellows between 2014 through 2018 at North American academic centers in the National Emergency Airway Registry for Neonates. Cumulative sum analysis was used to characterize progression of individual fellows' intubation competence, defined by an 80% overall success rate within 2 intubation attempts. We employed multivariable analysis to assess the independent impact of advancing quarter of fellowship training on intubation success.

RESULTS

There were 2297 intubation encounters performed by 92 fellows in 8 hospitals. Of these, 1766 (77%) were successful within 2 attempts. Of the 40 fellows assessed from the start of training, 18 (45%) achieved procedural competence, and 12 (30%) exceeded the deficiency threshold. Among fellows who achieved competence, the number of intubations to meet this threshold was variable, with an absolute range of 8 to 46 procedures. After adjusting for patient and practice characteristics, advancing quarter of training was independently associated with an increased odds of successful intubation (adjusted odds ratio: 1.10; 95% confidence interval 1.07-1.14).

CONCLUSIONS

The number of neonatal intubations required to achieve procedural competence is variable, and overall intubation competence rates are modest. Although repetition leads to skill acquisition for many trainees, some learners may require adjunctive educational strategies. An individualized approach to assess trainees' progression toward intubation competence is warranted.

An Intelligent Augmented Reality Training Framework for Neonatal Endotracheal Intubation

Neonatal Endotracheal Intubation (ETI) is a critical resuscitation skill that requires tremendous practice of trainees before clinical exposure. However, current manikin-based training regimen is ineffective in providing satisfactory real-time procedural guidance for accurate assessment due to the lack of see-through visualization within the manikin. The training efficiency is further reduced by the limited availability of expert instructors, which inevitably results in a long learning curve for trainees. To this end, we propose an intelligent Augmented Reality (AR) training framework that provides trainees with a complete visualization of the ETI procedure for real-time guidance and assessment. Specifically, the proposed framework is capable of capturing the motions of the laryngoscope and the manikin and offer 3D see-through visualization rendered to the head-mounted display (HMD). Furthermore, an attention-based Convolutional Neural Network (CNN) model is developed to automatically assess the ETI performance from the captured motions as well as identify regions of motions that significantly contribute to the performance evaluation. Lastly, augmented user-friendly feedback is delivered with interpretable results with the ETI scoring rubric through the color-coded motion trajectory that classifies highlighted regions that need more practice. The classification accuracy of our machine learning model is 84.6%.

Assessing competency using simulation: the SimZones approach

Introduction

Competency-based medical education (CBME) is a system of medical training that focuses on a structured approach to developing the clinical abilities of medical education graduates and practicing physicians. CBME requires a robust and multifaceted system of assessment in order to both measure and guide the progress of learners toward pre-established goals. Simulation has been proposed as one method for assessing competency in healthcare workers. However, a longitudinal framework for assessing competency using simulation has not been developed.

Methods

Conjecture mapping methodology was used to map Miller's framework for competency assessment-'knows', 'knows how', 'shows how', and 'does'-to the five SimZones described by Roussin and Weinstock. The SimZones describe a system of organising the development and delivery of simulation-based education and offer a foundation for both guiding and organising assessment in a simulation context.

Results

A conceptualised alignment of the SimZones with Miller's pyramid of assessment was developed, as well as a detailed conjecture map. SimZone 0 (auto-feedback) and SimZone 1 (foundational instruction) mapped to 'knows' and 'knows how'. SimZone 2 (acute care instruction) mapped to 'shows how'. SimZone 3 (team and system development) mapped to 'shows how'. SimZone 4 (real-life debriefing and development) mapped to 'does'.

Conclusion

The SimZones system of competency assessment offers a robust, flexible, and multifaceted system to guide both formative and summative assessment in CBME. The SimZones approach adds to the many methods of competency assessment available to educators. Adding SimZones to the vocabulary of CBME may be helpful for the full deployment of CBME.

Assessing Competence in Central Venous Catheter Placement by Pediatric Critical Care Fellows: A National Survey Study

OBJECTIVES

To describe the current approach to initial training, ongoing skill maintenance, and assessment of competence in central venous catheter placement by pediatric critical care medicine fellows, a subset of trainees in whom this skill is required.

DESIGN

Cross-sectional internet-based survey with deliberate sampling.

SETTING

United States pediatric critical care medicine fellowship programs.

SUBJECTS

Pediatric critical care medicine program directors of Accreditation Council for Graduate Medical Education-accredited fellowship programs.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

A working group of the Education in Pediatric Intensive Care Investigators research collaborative conducted a national study to assess the degree of standardization of training and competence assessment of central venous catheter placement across pediatric critical care medicine fellowship programs. After piloting, the survey was sent to all program directors (n = 67) of Accreditation Council for Graduate Medical Education-accredited pediatric critical care medicine programs between July 2017 and September 2017. The response rate was 85% (57/67). Although 98% of programs provide formalized central venous catheter placement training for first-year fellows, only 42% of programs provide ongoing maintenance training as part of fellowship. Over half (55%) of programs use a global assessment tool and 33% use a checklist-based tool when evaluating fellow central venous catheter placement competence under direct supervision. Only two programs (4%) currently use an assessment tool previously published and validated by the Education in Pediatric Intensive Care group. A majority (82%) of responding program directors believe that a standardized approach to assessment of central venous catheter competency across programs is important.

CONCLUSIONS

Despite national mandates for skill competence by many accrediting bodies, no standardized system currently exists across programs for assessing central venous catheter placement. Most pediatric critical care medicine programs use a global assessment and decisions around the ability of a fellow to place a central venous catheter under indirect supervision are largely based upon subjective assessment of performance. Further investigation is needed to determine if this finding is consistent in other specialties/subspecialties, if utilization of standardized assessment methods can improve program directors' abilities to ensure trainee competence in central venous catheter insertion in the setting of variable training approaches, and if these findings are consistent with other procedures across critical care medicine training programs, adult and pediatric.

A targeted systematic review of cost analyses for implementation of simulation-based education in healthcare

Over the past two decades, there has been an increase in the use of simulation-based education for training healthcare providers in technical and non-technical skills. Simulation education and research programs have mostly focused on the impact on clinical knowledge and improvement of technical skills rather than on cost. To study and characterize existing evidence to inform multi-stakeholder investment decisions, we performed a systematic review of the literature on costs in simulation-based education in medicine in general and in neonatal resuscitation as a particular focus. We conducted a systematic literature search of the PubMed database using two targeted queries. The first searched for cost analyses of healthcare simulation-based education more broadly, and the second was more narrowly focused on cost analyses of neonatal resuscitation training. The more general query identified 47 qualified articles. The most common specialties for education interventions were surgery (51%); obstetrics, gynecology, or pediatrics (11%); medicine, nursing, or medical school (11%); and urology (9%), accounting for over 80% of articles. The neonatal resuscitation query identified five qualified articles. The two queries identified seven large-scale training implementation studies, one in the United States and six in low-income countries. There were two articles each from Tanzania and India and one article each from Zambia and Ghana. Methods, definitions, and reported estimates varied across articles, implying interpretation, comparison, and generalization of program effects are challenging. More work is needed to understand the costs, processes, and outcomes likely to make simulation-based education programs cost-effective and scalable. To optimize return on investments in training, assessing resource requirements, associated costs, and subsequent outcomes can inform stakeholders about the potential sustainability of SBE programs. Healthcare stakeholders and decision makers will benefit from more transparent, consistent, rigorous, and explicit assessments of simulation-based education program development and implementation costs in low- and high-income countries.

Perspectives on simulation-based training from paediatric healthcare providers in Nigeria: a national survey

OBJECTIVES

The objective of this study was to explore the access to, and perceived utility of, various simulation modalities by in-service healthcare providers in a resource-scarce setting.

SETTING

Paediatric training workshops at a national paediatric conference in Nigeria.

PARTICIPANTS

All 200 healthcare workers who attended the workshop sessions were eligible to participate. A total of 161 surveys were completed (response rate 81%).

PRIMARY AND SECONDARY OUTCOME MEASURES

A paper-based 25-item cross-sectional survey on simulation-based training (SBT) was administered to a convenience sample of healthcare workers from secondary and tertiary healthcare facilities.

RESULTS

Respondents were mostly 31-40 years of age (79, 49%) and women (127, 79%). Consultant physicians (26, 16%) and nurses (56, 35%) were in both general (98, 61%) and subspecialty (56, 35%) practice. Most had 5-10 years of experience (62, 37%) in a tertiary care setting (72, 43%). Exposure to SBT varied by profession with physicians more likely to be exposed to manikin-based (29, 30% physicians vs 12, 19% nurses, p<0.001) or online training (7, 7% physician vs 3, 5% nurses, p<0.05). Despite perceived barriers to SBT, respondents thought that SBT should be expanded for continuing education (84, 88% physician vs 39, 63% nurses, p<0.001), teaching (73, 76% physicians vs 16, 26% nurses, p<0.001) and research (65, 68% physicians vs 14, 23% nurses, p<0.001). If facilities were available, nearly all respondents (92, 98% physicians; 52, 96% nurses) would recommend the use of online simulation for their centre.

CONCLUSIONS

The access of healthcare workers to SBT is limited in resource-scarce settings. While acknowledging the challenges, respondents identified many areas in which SBT may be useful, including skills acquisition, skills practice and communication training. Healthcare workers were open to the use of online SBT and expressed the need to expand SBT beyond the current scope for health professional training in Nigeria.

Neonatal Provider Workforce

This technical report reviews education, training, competency requirements, and scopes of practice of the different neonatal care providers who work to meet the special needs of neonatal patients and their families in the NICU. Additionally, this report examines the current workforce issues of NICU providers, offers suggestions for establishing and monitoring quality and safety of care, and suggests potential solutions to the NICU provider workforce shortages now and in the future.

Competency-Based Education Assessment Tools for Laser Capsulotomy, Peripheral Iridotomy, and Selective Laser Trabeculoplasty

PURPOSE

To produce Competency-Based Education Assessment Tools (C-BEAT) for performance of neodymium-doped yttrium aluminum garnet (Nd:YAG) laser capsulotomy, laser peripheral iridotomy (LPI), and selective laser trabeculoplasty (SLT) on an artificial eye model that can be used in competency-based medical education (CBME) of ophthalmology residents.

DESIGN

Survey.

PARTICIPANTS

Eight content experts (residency program director and faculty members involved in ophthalmology resident training).

METHODS

Task-specific checklists were created to assess the performance of Nd:YAG capsulotomy, LPI, and SLT on artificial eye models designed for each of these procedures, as well as a global rating scale (GRS). A modified Delphi process was used to establish face and content validity of the C-BEAT for lasers. Eight content experts participated in the Delphi process, and the consensus was achieved after 4 rounds.

MAIN OUTCOME MEASURE

Survey responses.

RESULTS

Final versions of checklists for Nd:YAG capsulotomy, LPI, and SLT, and a GRS for these laser procedures were produced with face and content validity. Modifications to the energy settings and the number of applications in the capsulotomy and peripheral iridotomy were implemented into the checklist and instructions, because the artificial models were not identical in their responses to the laser applications compared with real tissue.

CONCLUSIONS

The C-BEAT for lasers can be used in an examination setting within ophthalmology residency training programs to establish competency in these procedures before performing them on real patients. These tools help to meet the demand for assessment modalities within a CBME model that is being implemented into residency training worldwide. To establish interrater and construct validities, further studies are required.