Simulation for High-Stakes Assessment in Pediatric Emergency Medicine

Simulation vs workplace-based assessment in resuscitation: a cross-specialty descriptive analysis and comparison

Background

Simulation-based assessment can complement workplace-based assessment of rare or difficult to assess Entrustable Professional Activities (EPAs). We aimed to compare the use of simulation-based assessment for resuscitation-focused EPAs in three postgraduate medical training programs and describe faculty perceptions of simulation-based assessment.

Methods

EPA assessment scores and setting (simulation or workplace) were extracted from 2017-2020 for internal medicine, emergency medicine, and surgical foundations residents at the transition to discipline and foundations of discipline stages. A questionnaire was distributed to clinical competency committee members.

Results

Eleven percent of EPA assessments were simulation-based. The proportion of simulation-based assessment did not differ between programs but differed between transition (38%) and foundations (4%) stages within surgical foundations only. Entrustment scores differed between settings in emergency medicine at the transition level only (simulation: 4.82 ± 0.60 workplace: 3.74 ± 0.93). 70% of committee members (n=20) completed the questionnaire. Of those that use simulation-based assessment, 45% interpret them differently than workplace-based assessments. 73% and 100% trust simulation for high-stakes and low-stakes assessment, respectively.

Conclusions

The proportion of simulation-based assessment for resuscitation focused EPAs did not differ between three postgraduate medical training programs. Interpretation of simulation-based assessment data between committee members was inconsistent. All respondents trust simulation-based assessment for low-stakes, and the majority for high-stakes assessment. These findings have practical implications for the integration simulation into programs of assessment.

Preparing and Presenting Validation Studies: A Guide for the Perplexed

SUMMARY STATEMENT

Simulated environments are frequently used for learner assessment, and a wide array of assessment instruments have been created to assist with this process. It is important, therefore, that clear, compelling evidence for the validity of these assessments be established. Contemporary theory recognizes instrument validity as a unified construct that links a construct to be assessed with a population, an environment of assessment, and a decision to be made using the scores. In this article, we present a primer on 2 current frameworks (Messick and Kane), define the elements of each, present a rubric that can be used by potential authors to structure their work, and offer examples of published studies showing how each framework has been successfully used to make a validity argument. We offer this with the goal of improving the quality of validity-related publications, thereby advancing the quality of assessment in healthcare simulation.

Psychometric Testing of the Debriefing Assessment for Simulation in Healthcare (DASH) for Trainee-led, In Situ Simulations in the Pediatric Emergency Department Context

OBJECTIVES

Effective trainee-led debriefing after critical events in the pediatric emergency department has potential to improve patient care, but debriefing assessments for this context have not been developed. This study gathers preliminary validity and reliability evidence for the Debriefing Assessment for Simulation in Healthcare (DASH) as an assessment of trainee-led post-critical event debriefing.

METHODS

Eight fellows led teams in three simulated critical events, each followed by a video-recorded discussion of performance mimicking impromptu debriefings occurring after real clinical events. Three raters assessed the recorded debriefings using the DASH, and their feedback was collated. Data were analyzed using generalizability theory, Gwet's AC2, intraclass correlation coefficient (ICC), and coefficient alpha. Validity was examined using Messick's framework.

RESULTS

The DASH instrument had relatively low traditional inter-rater reliability (Gwet's AC2 = 0.24, single-rater ICC range = 0.16-0.35), with 30% fellow, 19% rater, and 23% rater by fellow variance. DASH generalizability (G) coefficient was 0.72, confirming inadequate reliability for research purposes. Decision (D) study results suggest the DASH can attain a G coefficient of 0.8 with five or more raters. Coefficient alpha was 0.95 for the DASH. A total of 90 and 40% of items from Elements 1 and 4, respectively, were deemed "not applicable" or left blank.

CONCLUSIONS

Our results suggest that the DASH does not have sufficient validity and reliability to rigorously assess debriefing in the post-critical event environment but may be amenable to modification. Further development of the tool will be needed for optimal use in this context.

Attitudes Towards Introduction of Multiple Modalities of Simulation in Objective Structured Clinical Examination (OSCE) of Emergency Medicine (EM) Final Board Examination: A Cross-Sectional Study

Purpose

Objective Structured Clinical Examination (OSCE) is the current modality of choice for evaluating practical skills for graduating emergency medicine residents of final Saudi board examination. This study aims to evaluate the attitudes of both residents and faculty towards the idea of utilizing multiple modalities of simulation in a high-stakes emergency medicine (EM) examination. The goal is to propose a method to improve the process of this examination.

Participants and Methods

The data were obtained using a cross-sectional survey questionnaire that was distributed to 141 participants, including both EM residents and instructors in the Saudi Board of Emergency Medicine. An online survey tool was used. The data were collected and subsequently analyzed to gauge the general and specific attitudes of both residents and instructors.

Results

Of the 141 participants, 136 provided complete responses; almost half were residents from all years, and the other half were primarily instructors (registrars, senior registrars, or consultants). Most of the participants from both groups (70% of the residents and 86% of the instructors) would like to see simulation incorporated into the final EM board OSCEs. Most of the participants (78%), however, had no experience with using multiple modalities of simulation in OSCEs. Overall, the majority (74.82%) expressed the belief that simulation-based OSCEs would improve the assessment of EM residents’ competencies. The modalities that received the most support were part-task trainers and hybrid simulation (70.71% and 70%, respectively).

Conclusion

From this study, we can conclude that both parties (residents and instructors) are largely willing to see multimodality simulation being incorporated into the final board examinations. Stakeholders should interpret this consensus as an impetus to proceed with such an implementation of multimodality simulation. Input from both groups should be considered when planning for such a change in this high-stakes exam.

Educational strategy for the development of skills in exchange transfusion: a randomized clinical trial protocol

BACKGROUND

Exchange transfusion is a highly complex procedure that requires high levels of expertise. Trainee paediatricians do not have adequate training in exchange transfusion because opportunities to perform this procedure in practice are scarce. This protocol seeks to compare two educational interventions for exchange transfusion that allow students to develop competencies to perform the technique in an appropriate and safe way.

METHODS/DESIGN

This is a randomized parallel single-blind clinical trial with allocation by simple randomization to the educational intervention (simulation or a digital didactic environment). Students from the paediatric specialization who volunteer to participate will be included. A practical evaluation of the procedure will be performed through a simulated scenario using a standardized clinical case. The main outcome is defined as the result of evaluation using the Objective Structured Clinical Examination; superior performance will be defined when the percentage is greater than or equal to 85%, and non-superior performance will be defined when the result is less than 84%. The chi-square independence test or the Fisher exact test will be used to evaluate the effect of the interventions. Multivariate analysis will be performed using a non-conditional logistic regression model. Stata 15® software will be used.

DISCUSSION

Exchange transfusion is a procedure that requires expertise to achieve adequate outcomes. The inclusion of new educational strategies, such as simulation and digital didactic environments, is seen as a training option that can improve performance in clinical skills, reduce adverse events and increase the level of trust.

TRIAL REGISTRATION

ClinicalTrials.gov: NCT04070066. Registered on 28 August 2019. https://clinicaltrials.gov.

Resuscitation Education Science: Educational Strategies to Improve Outcomes From Cardiac Arrest: A Scientific Statement From the American Heart Association

The formula for survival in resuscitation describes educational efficiency and local implementation as key determinants in survival after cardiac arrest. Current educational offerings in the form of standardized online and face-to-face courses are falling short, with providers demonstrating a decay of skills over time. This translates to suboptimal clinical care and poor survival outcomes from cardiac arrest. In many institutions, guidelines taught in courses are not thoughtfully implemented in the clinical environment. A current synthesis of the evidence supporting best educational and knowledge translation strategies in resuscitation is lacking. In this American Heart Association scientific statement, we provide a review of the literature describing key elements of educational efficiency and local implementation, including mastery learning and deliberate practice, spaced practice, contextual learning, feedback and debriefing, assessment, innovative educational strategies, faculty development, and knowledge translation and implementation. For each topic, we provide suggestions for improving provider performance that may ultimately optimize patient outcomes from cardiac arrest.

Education and Training in Pediatric Cardiac Critical Care

Pediatric cardiac critical care is a new and emerging field. There is no standardization to the current education provided, and high-quality patient outcomes require such standardization. For physicians, this includes fellowship training, specific competencies, and a certification process. For advanced practice providers, a standardized curriculum as well as a certification process is needed. There is evidence that supports a finding that critical care nursing experience may have a positive impact on outcomes from pediatric cardiac surgery. A rigorous orientation and meaningful continuing education may augment that. For all disciplines and levels of expertise, simulation is a useful modality in the education in pediatric cardiac critical care.