Queen’s Simulation Assessment Tool: Development and Validation of an Assessment Tool for Resuscitation Objective Structured Clinical Examination Stations in Emergency Medicine

Can peer feedback substitute for faculty feedback in predoctoral dental education?

PURPOSE/OBJECTIVES

The ability to give and receive feedback is a key skill to develop during predoctoral dental education, and the use of peer feedback specifically offers distinct benefits including a different understanding of material due to peers' proximity of knowledge development and assisting with overburdened instructors. However, it is unclear if peer feedback offers similar quality to instructor feedback.

METHODS

Dental students in two different graduation years provided quantitative and qualitative peer feedback on a case-based oral and maxillofacial pathology simulation. The data from these exercises were aggregated and analyzed to compare the quality of qualitative feedback to course examination scores. Student perceptions of peer feedback were also recorded.

RESULTS

The mean quality of feedback was not correlated with course examination scores, though the number of times students gave high-quality feedback and received high-quality feedback was correlated with course examination scores. Student feedback overall had a lower quality than instructor feedback, though there was no significant difference between instructor feedback quality and the maximum student feedback quality received. Student perceptions of the utility of feedback were positive.

CONCLUSION

While instructor feedback is more reliable and consistent, our findings suggest that in most instances, at least one peer in moderate-sized groups is able to approximate the quality of instructor feedback on case-based assignments.

Automating medical simulations

OBJECTIVE

This study aims to explore speech as an alternative modality for human activity recognition (HAR) in medical settings. While current HAR technologies rely on video and sensory modalities, they are often unsuitable for the medical environment due to interference from medical personnel, privacy concerns, and environmental limitations. Therefore, we propose an end-to-end, fully automatic objective checklist validation framework that utilizes medical personnel's uttered speech to recognize and document the executed actions in a checklist format.

METHODS

Our framework records, processes, and analyzes medical personnel's speech to extract valuable information about performed actions. This information is then used to fill the corresponding rubrics in the checklist automatically.

RESULTS

Our approach to activity recognition outperformed the online expert examiner, achieving an F1 score of 0.869 on verbal tasks and an ICC score of 0.822 with an offline examiner. Furthermore, the framework successfully identified communication failures and medical errors made by physicians and nurses.

CONCLUSION

Implementing a speech-based framework in medical settings, such as the emergency room and operation room, holds promise for improving care delivery and enabling the development of automated assistive technologies in various medical domains. By leveraging speech as a modality for HAR, we can overcome the limitations of existing technologies and enhance workflow efficiency and patient safety.

Catalytic effect of multisource feedback for trauma team captains: a mixed-methods prospective study

OBJECTIVES

To evaluate the impact and feasibility of multisource feedback compared with traditional feedback for trauma team captains (TTCs).

DESIGN

A mixed-methods, non-randomised prospective study.

SETTING

A level one trauma centre in Ontario, Canada.

PARTICIPANTS

Postgraduate medical residents in emergency medicine and general surgery participating as TTCs. Selection was based on a convenience sampling method.

INTERVENTION

Postgraduate medical residents participating as TTCs received either multisource feedback or standard feedback following trauma cases.

MAIN OUTCOME MEASURES

TTCs completed questionnaires designed to measure the self-reported intention to change practice (catalytic effect), immediately following a trauma case and 3 weeks later. Secondary outcomes included measures of perceived benefit, acceptability, and feasibility from TTCs and other trauma team members.

RESULTS

Data were collected following 24 trauma team activations: TTCs from 12 activations received multisource feedback and 12 received standard feedback. The self-reported intention for practice change was not significantly different between groups initially (4.0 vs 4.0, p=0.57) and at 3 weeks (4.0 vs 3.0, p=0.25). Multisource feedback was perceived to be helpful and superior to the existing feedback process. Feasibility was identified as a challenge.

CONCLUSIONS

The self-reported intention for practice change was no different for TTCs who received multisource feedback and those who received standard feedback. Multisource feedback was favourably received by trauma team members, and TTCs perceived multisource feedback as useful for their development.

Multi-source feedback following simulated resuscitation scenarios: a qualitative study

BACKGROUND

The direct observation and assessment of learners' resuscitation skills by an attending physician is challenging due to the unpredictable and time-sensitive nature of these events. Multisource feedback (MSF) may address this challenge and improve the quality of assessments provided to learners. We aimed to describe the similarities and differences in the assessment rationale of attending physicians, registered nurses, and resident peers in the context of a simulation-based resuscitation curriculum.

METHODS

We conducted a qualitative content analysis of narrative MSF of medical residents in their first postgraduate year of training who were participating in a simulation-based resuscitation course at two Canadian institutions. Assessments included an entrustment score and narrative comments from attending physicians, registered nurses, and resident peers in addition to self-assessment. Narrative comments were transcribed and analyzed thematically using a constant comparative method.

RESULTS

All 87 residents (100%) participating in the 2017-2018 course provided consent. A total of 223 assessments were included in our analysis. Four themes emerged from the narrative data: 1) Communication, 2) Leadership, 3) Demeanor, and 4) Medical Expert. Relative to other assessor groups, feedback from nurses focused on patient-centred care and communication while attending physicians focused on the medical expert theme. Peer feedback was the most positive. Self-assessments included comments within each of the four themes.

CONCLUSIONS

In the context of a simulation-based resuscitation curriculum, MSF provided learners with different perspectives in their narrative assessment rationale and may offer a more holistic assessment of resuscitation skills within a competency-based medical education (CBME) program of assessment.

Simulation for assessment of Entrustable Professional Activities in an emergency medicine residency program

In 2018, Canadian post-graduate Emergency Medicine (EM) programs transitioned to Competence-by-Design. Residents are now assessed using Entrustable Professional Activities (EPAs). We developed and implemented simulation for assessment to mitigate anticipated challenges with residents completing the required number of observations of resuscitation-based EPAs. Our survey of trainees who participated in these sessions suggests that it may be a feasible and acceptable method for EPA assessment.

The use of a formative OSCE to prepare emergency medicine residents for summative OSCEs: a mixed-methods cohort study

Background

The objective structured clinical examination (OSCE) is a part of emergency medicine (EM) examinations such as the Masters of Medicine in Emergency Medicine (MMed) examination and the equivalent Member of the Royal College of Emergency Medicine (MRCEM) examination. The use of formative OSCEs to prepare EM residents for summative OSCEs has not been investigated. This study aimed to evaluate the role of formative OSCEs in preparing EM residents for the MMed and MRCEM OSCE.

Methods

This was an observational, retrospective, mixed-methods cohort study. We analysed data from formative OSCEs conducted by the National Healthcare Group EM residency programme from 2013 to 2019, and from a questionnaire distributed to all residents during the study period. Residents’ formative OSCE participation and scores were compared with first-attempt summative OSCE success. Qualitative analysis of residents’ opinions on the usefulness of the formative OSCE was performed.

Results

Forty-three of the 50 (86.0%) residents attended at least one formative OSCE. Of the 46 who responded to the questionnaire, 40 (87.0%) had attempted and succeeded in the MMed or MRCEM OSCE, of whom 35 (87.5%) had succeeded on the first attempt. Residents who succeeded in the summative OSCE on the first attempt tended to have higher proximate (mean = 70.6, SD = 8.9 vs mean = 64.3, SD = 10.8) and mean (mean = 67.4, SD = 7.1 vs mean = 62.8, SD = 7.3) formative OSCE scores. All 8/40 (20.0%) residents who attended more than three formative OSCEs succeeded in the summative OSCE on their first attempt. Residents’ formative OSCE scores tended to improve with successive formative OSCEs, demonstrating a positive training effect. All residents felt that the formative OSCE was useful in preparing them for the summative OSCE.

Conclusions

Participation in multiple formative OSCEs was beneficial in preparing residents for the summative OSCE. The formative OSCE was useful in familiarising residents with the examination, giving them an opportunity to perform in near-examination conditions, and providing feedback to residents and faculty about their progress. Our findings may support the implementation of formative OSCEs in other training programmes to prepare learners for high-stake summative OSCEs.

Assessment of medical students' clinical performance using high-fidelity simulation: comparison of peer and instructor assessment

BACKGROUND

High-fidelity simulators are highly useful in assessing clinical competency; they enable reliable and valid evaluation. Recently, the importance of peer assessment has been highlighted in healthcare education, and studies using peer assessment in healthcare, such as medicine, nursing, dentistry, and pharmacy, have examined the value of peer assessment. This study aimed to analyze inter-rater reliability between peers and instructors and examine differences in scores between peers and instructors in the assessment of high-fidelity-simulation-based clinical performance by medical students.

METHODS

This study analyzed the results of two clinical performance assessments of 34 groups of fifth-year students at Ajou University School of Medicine in 2020. This study utilized a modified Queen's Simulation Assessment Tool to measure four categories: primary assessment, diagnostic actions, therapeutic actions, and communication. In order to estimate inter-rater reliability, this study calculated the intraclass correlation coefficient and used the Bland and Altman method to analyze agreement between raters. A t-test was conducted to analyze the differences in evaluation scores between colleagues and faculty members. Group differences in assessment scores between peers and instructors were analyzed using the independent t-test.

RESULTS

Overall inter-rater reliability of clinical performance assessments was high. In addition, there were no significant differences in overall assessment scores between peers and instructors in the areas of primary assessment, diagnostic actions, therapeutic actions, and communication.

CONCLUSIONS

The results indicated that peer assessment can be used as a reliable assessment method compared to instructor assessment when evaluating clinical competency using high-fidelity simulators. Efforts should be made to enable medical students to actively participate in the evaluation process as fellow assessors in high-fidelity-simulation-based assessment of clinical performance in situations similar to real clinical settings.

Assessment of Emergency Medicine Resident Performance in a Pediatric In Situ Simulation Using Multi-Source Feedback

Introduction

Multi-source feedback (MSF) is an evaluation method mandated by the Accreditation Council for Graduate Medical Education (ACGME). The Queen’s Simulation Assessment Tool (QSAT) has been validated as being able to distinguish between resident performances in a simulation setting. The QSAT has also been demonstrated to have excellent MSF agreement when used in an adult simulation performed in a simulation lab. Using the QSAT, this study sought to determine the degree of agreement of MSF in a single pediatric (Peds) simulation case conducted in situ in a Peds emergency department (ED).

Methods

This Institutional Review Board-approved study was conducted in a four-year emergency medicine residency. A Peds resuscitation case was developed with specific behavioral anchors on the QSAT, which uses a 1-5 scale in each of five categories: Primary Assessment, Diagnostic Actions, Therapeutic Actions, Communication, and Overall Assessment. Data was gathered from six participants for each simulation. The lead resident self-evaluated and received MSF from a junior peer resident, a fixed Peds ED nurse, a random ED nurse, and two faculty (one fixed, the other from a dyad). The agreement was calculated with intraclass correlation coefficients (ICC).

Results

The simulation was performed on 35 separate days over two academic years. A total of 106 MSF participants were enrolled. Enrollees included three faculty members, 35 team leaders, 34 peers, 33 ED registered nurses (RN), and one Peds RN; 50% of the enrollees were female (n=53). Mean QSAT scores ranged from 20.7 to 23.4. A fair agreement was demonstrated via ICC; there was no statistically significant difference between sources of MSF. Removing self-evaluation led to the highest ICC. ICC for any single or grouped non-faculty source of MSF was poor.

Conclusion

Using the QSAT, the findings from this single-site cohort suggest that faculty must be included in MSF. Self-evaluation appears to be of limited value in MSF with the QSAT. The degree of MSF agreement as gathered by the QSAT was lower in this cohort than previously reported for adult simulation cases performed in the simulation lab. This may be due to either the pediatric nature of the case, the location of the simulation, or both.

Direct Observation Tools in Emergency Medicine: A Systematic Review of the Literature

OBJECTIVES

Direct observation is important for assessing the competency of medical learners. Multiple tools have been described in other fields, although the degree of emergency medicine-specific literature is unclear. This review sought to summarize the current literature on direct observation tools in the emergency department (ED) setting.

METHODS

We searched PubMed, Scopus, CINAHL, the Cochrane Central Register of Clinical Trials, the Cochrane Database of Systematic Reviews, ERIC, PsycINFO, and Google Scholar from 2012 to 2020 for publications on direct observation tools in the ED setting. Data were dual extracted into a predefined worksheet, and quality analysis was performed using the Medical Education Research Study Quality Instrument.

RESULTS

We identified 38 publications, comprising 2,977 learners. Fifteen different tools were described. The most commonly assessed tools included the Milestones (nine studies), Observed Structured Clinical Exercises (seven studies), the McMaster Modular Assessment Program (six studies), Queen's Simulation Assessment Test (five studies), and the mini-Clinical Evaluation Exercise (four studies). Most of the studies were performed in a single institution, and there were limited validity or reliability assessments reported.

CONCLUSIONS

The number of publications on direct observation tools for the ED setting has markedly increased. However, there remains a need for stronger internal and external validity data.

Development and implementation of a novel, mandatory competency-based medical education simulation program for pediatric emergency medicine faculty

Background

Maintaining acute care physician competence is critically important. Current maintenance of certification (MOC) programs has started to incorporate simulation-based education (SBE). However, competency expectations have not been defined. This article describes the development of a mandatory annual SBE, competency-based simulation program for technical and resuscitation skills for pediatric emergency medicine (PEM) physicians.

Methods

The competency-based medical education (CBME) program was introduced in 2016. Procedural skill requirements were based on a needs assessment derived from Royal College PEM training guidelines. Resuscitation scenarios were modified versions of pre-existing in-situ mock codes or critical incident cases. All full-time faculty were required to participate annually in both sessions. Delivery of educational content included a flipped classroom website, deliberate practice, and stop-pause debriefing. All stations required competency checklists and global rating scales.

Results

Between 2016 and 2018, 40 physicians and 48 registered nurses attended these courses. Overall course evaluations in 2018 were 4.92/5 and 4.93/5. Barriers to implementation include the need for many simulation education experts, time commitment, and clinical scheduling during course events.

Conclusion

We have developed a mandatory simulation-based, technical, and resuscitation CBME program for PEM faculty. This simulation-based CBME program could be adapted to other acute care disciplines. Further research is required to determine if these skills are enhanced both in a simulated and real environment and if there is an impact on patient outcomes.

Supplementary Information

The online version contains supplementary material available at 10.1186/s41077-021-00170-4.

An Evaluation of the Surgical Foundations Curriculum: A National Study

BACKGROUND

Canadian Surgical Foundations (SF) residency programs transitioned to competency-based medical education in 2018. It is unknown how well the SF curriculum prepares surgery residents to recognize and manage common perioperative patient presentations. We sought to evaluate the national SF curriculum using the Kirkpatrick model of curriculum evaluation.

METHODS

We administered online surveys to 300 first-year English-speaking surgery residents across Canada to assess self-reported confidence in recognizing and managing 7 common perioperative patient presentations at 3 timepoints: pre-SF (July 2019), mid-SF (December 2019), and post-SF (May 2020). We conducted multistation simulation-based objective structured clinical examinations for surgery residents at our institution pre-SF (August 2019) and mid-SF (December 2019), and collected workplace-based assessment (WBA) data, including entrustment scores and narrative feedback, for 6 preselected entrustable professional activities (EPAs) (July 2019 to May 2020).

RESULTS

Fifty-five residents (18%) completed pre-SF, 31 (10%) completed mid-SF, and 52 (17%) completed post-SF surveys. Residents' confidence in recognizing 6 out of 7 patient presentations was high pre-SF and did not improve significantly during the SF curriculum except for recognizing poor glycemic control (p < 0.01). Residents' confidence in managing 7 out of 7 patient presentations improved significantly (p < 0.05). Objective structured clinical examinations performance did not change significantly between pre-SF and mid-SF (4 [3.5-4.5] vs 4 [3-4]; p = 0.28). Analysis of WBA data showed that residents received high entrustment scores from the start of the SF curriculum. Entrustment scores improved significantly during the SF curriculum for 2 out of 6 EPAs. Only 56% of WBA assessments had narrative feedback, 16% of which had somewhat constructive feedback.

CONCLUSION

Participation in the SF curriculum was associated with improved confidence of surgery residents in managing common perioperative patient presentations, and greater level of entrustment for some EPAs. Consideration should be given to further faculty development to increase the quantity and quality of narrative feedback in the SF curriculum.

Development of a novel global rating scale for objective structured assessment of technical skills in an emergency medical simulation training

BACKGROUND

Medical simulation trainings lead to an improvement in patient care by increasing technical and non-technical skills, procedural confidence and medical knowledge. For structured simulation-based trainings, objective assessment tools are needed to evaluate the performance during simulation and the learning progress. In surgical education, objective structured assessment of technical skills (OSATS) are widely used and validated. However, in emergency medicine and anesthesia there is a lack of validated assessment tools for technical skills. Thus, the aim of the present study was to develop and validate a novel Global Rating Scale (GRS) for emergency medical simulation trainings.

METHODS

Following the development of the GRS, 12 teams of different experience in emergency medicine (4th year medical students, paramedics, emergency physicians) were involved in a pre-hospital emergency medicine simulation scenario and assessed by four independent raters. Subsequently, interrater reliability and construct validity of the GRS were analyzed. Moreover, the results of the GRS were cross-checked with a task specific check list. Data are presented as median (minimum; maximum).

RESULTS

The GRS consists of ten items each scored on a 5-point Likert scale yielding a maximum of 50 points. The median score achieved by novice teams was 22.75 points (17;30), while experts scored 39.00 points (32;47). The GRS overall scores significantly discriminated between student-guided teams and expert teams of emergency physicians (p = 0.005). Interrater reliability for the GRS was high with a Kendall's coefficient of concordance W ranging from 0.64 to 0.90 in 9 of 10 items and 0.88 in the overall score.

CONCLUSION

The GRS represents a promising novel tool to objectively assess technical skills in simulation training with high construct validity and interrater reliability in this pilot study.

Assessing the impact of resuscitation residents on the treatment of cardiopulmonary resuscitation patients

BACKGROUND

The management of cardiac arrest patients receiving cardiopulmonary resuscitation (CPR) is an essential aspect of emergency medicine (EM) training. At our institution, we have a 1-month Resuscitation Rotation designed to augment resident training in managing critical patients. The objective of this study is to compare 30-day mortality between cardiac arrest patients with resuscitation resident (RR) involvement versus patients without. Our secondary outcome is to determine if RR involvement altered rates of initiating targeted temperature management (TTM).

METHODS

This study was conducted at a single site tertiary care Level-1 trauma center with an Emergency Department (ED) census of nearly 130,000 visits per year. Data was collected from 01/01/2015 to 01/01/2018 using electronic medical records via query. Patients admitted with cardiac arrest were separated into two groups, one with RR involvement and one without. Initial rhythm of ventricular fibrillation/tachycardia (VFIB/VTACH), 30-day mortality, history of coronary artery disease (CAD), and initiation of TTM were compared. Statistical analysis was performed.

RESULTS

Out of 885 patient encounters, 91 (10.28%) had RR participation. There was no statistical difference in 30-day mortality between patients with RR involvement compared to those without (71.42% vs 66.36%; P = 0.3613). However, TTM was initiated more in the RR group (20.70% vs 8.86%; P = 0.0025). Patients who received TTM also had a lower 30-day mortality compared to those without TTM (52.94% vs 70.87%; P = 0.0020). Patients who were older and had no history of CAD were also noted to have a statistically significant higher 30-day mortality. All other variables were not statistically significant.

CONCLUSION

Resuscitation resident involvement with the care of cardiac arrest patients had no impact in 30-day mortality. However, the involvement of RR was associated with a statistically significant increase in the initiation of TTM. One limitation is that RR participated in 10.28% of the cases analyzed herein, thus the two arms are unbalanced in size. Future work may investigate if the increase in TTM in the RR involved cases may portend improved rates of neurologically intact survival or more rapid achievement of goal temperatures.

A case for feedback and monitoring assessment in competency-based medical education

PURPOSE

Within competency-based medical education, self-regulated learning (SRL) requires residents to leverage self-assessment and faculty feedback. We sought to investigate the potential for competency-based assessments to foster SRL by quantifying the relationship between faculty feedback and entrustment ratings as well as the congruence between faculty assessment and resident self-assessment.

MATERIALS AND METHODS

We collected comments in (a) an emergency medicine objective structured clinical examination group (objective structured clinical examinations [OSCE] and emergency medicine OSCE group [EMOG]) and (b) a first-year resident multidisciplinary resuscitation "Nightmares" course assessment group (NCAG) and OSCE group (NOG). We assessed comments across five domains including Initial Assessment (IA), Diagnostic Action (DA), Therapeutic Action (TA), Communication (COM), and entrustment. Analyses included structured qualitative coding and (non)parametric and descriptive analyses.

RESULTS

In the EMOG, faculty's positive comments in the entrustment domain corresponded to lower entrustment score Mean Ranks (MRs) for IA (<11.1), DA (<11.2), and entrustment (<11.6). In NOG, faculty's negative comments resulted in lower entrustment score MRs for TA (<11.8 and <10) and DA (<12.4), and positive comments resulted in higher entrustment score MRs for IA (>15.4) and COM (>17.6). In the NCAG, faculty's positive IA comments were negatively correlated with entrustment scores (ρ = -.27, P = .04). Across programs, faculty and residents made similar domain-specific comments 13% of the time.

CONCLUSIONS

Minimal and inconsistent associations were found between narrative and numerical feedback. Performance monitoring accuracy and feedback should be included in assessment validation.

Resuscitation Resident Impact in the Treatment of Sepsis

Background The resuscitation of septic patients is a fundamental skill of emergency medicine (EM) training. We developed a required rotation designed to augment resident training in resuscitating critically ill patients in the emergency department (ED). The purpose of this study was to evaluate the successful completion of sepsis core measures alongside clinical outcomes between patients with a resuscitation resident (RR) involved in care versus patients without. Methods This retrospective study was conducted at a single site tertiary care Level 1 trauma center with an ED census of 130,000 visits annually. Data were collected from January 1, 2015, to December 31, 2016, using the electronic medical record (EMR) via an Epic query (Epic Systems Corp., Verona, WI). Patients admitted with severe sepsis or septic shock (Surviving Sepsis Campaign guidelines) were included and separated into two groups, one with RR involvement and one without. Emergency department length of stay, time to initial lactic acid draw, lactic acid value, time to bolus fluid initiation, time to antibiotic initiation, need for medical intensive care unit (ICU) admission, and 30-day mortality were compared between the two groups. Chi-square tests and Fisher's exact tests were used to analyze the categorical variables. Two-sided t-tests and Wilcoxon rank-sum tests were used to examine continuous variables. Results Out of 4,746 patients admitted, 101 patients had an RR participate in their care. The median time to initial lactic acid draw was shorter (0.53 vs 1.05 hours; p < 0. 0001) and the lactic acid level was higher (2.5 vs 1.8 mmol/L; p < 0. 0001) with the presence of an RR. Resuscitation resident was correlated with a decrease in time to antibiotics and appropriate 30 cc/kg bolus, however, these were not statistically significant (p = 0.10 and p = 0.09 respectively). Resuscitation resident involvement was also associated with more medical ICU (45.5% vs 18.8%; p<0.0001) admissions and a higher 30-day mortality (14.9% vs 29.7%; p < 0. 0001). All other variables were not statistically significant. Conclusion Resuscitation residents demonstrate a statistically significant impact on lactic acid-related bundle compliance and help facilitate the care of higher acuity severe sepsis and septic shock patients.

Development and Validation of Content Domains for Paramedic Prehospital Performance Assessment: A Focus Group and Delphi Method Approach

BACKGROUND

The chaotic and complex nature of delivering patient care in the prehospital setting complicates the provision of real-time formative feedback to paramedic students. Although the use of simulations is widespread in emergency medical services (EMS) education, a high degree of variability precludes consistent performance assessment in EMS. Objectives: The objective of this study was to define and validate key domains required to evaluate paramedic prehospital performance. Methods: We conducted a two-phase study that combined focus group and Delphi methodology. Participants were purposefully selected to attain diverse panels regarding sex, race, ethnicity, professional roles, levels of education, geographical area, and experience as a paramedic and educator. In Phase I, a panel of 11 subject matter experts (SMEs) were tasked with identifying the essential domains to be evaluated in a paramedic performance assessment. In Phase II, another panel of 11 SMEs and a four-round modified Delphi method with 28 paramedic program directors were used to validate the domains identified in Phase I. Results: The first focus group identified and achieved consensus on five domains: (1) effective communication, (2) scene management, (3) patient assessment, (4) patient management, and (5) professional behavior. These domains were validated by the second focus group. The first round of the Delphi process generated 64 content domains, which were reduced to nine unique content domains via thematic analysis. These nine content domains fit well within the broader domains identified by the focus groups with one specific area, critical thinking and reasoning, being listed in two key areas based on the definitions of the focus group domains of patient assessment and patient management. Conclusion: The content domains identified in this study provide EMS educators a theoretical framework for designing the performance assessment of newly trained professionals in the prehospital setting.

Resuscitation Education Science: Educational Strategies to Improve Outcomes From Cardiac Arrest: A Scientific Statement From the American Heart Association

The formula for survival in resuscitation describes educational efficiency and local implementation as key determinants in survival after cardiac arrest. Current educational offerings in the form of standardized online and face-to-face courses are falling short, with providers demonstrating a decay of skills over time. This translates to suboptimal clinical care and poor survival outcomes from cardiac arrest. In many institutions, guidelines taught in courses are not thoughtfully implemented in the clinical environment. A current synthesis of the evidence supporting best educational and knowledge translation strategies in resuscitation is lacking. In this American Heart Association scientific statement, we provide a review of the literature describing key elements of educational efficiency and local implementation, including mastery learning and deliberate practice, spaced practice, contextual learning, feedback and debriefing, assessment, innovative educational strategies, faculty development, and knowledge translation and implementation. For each topic, we provide suggestions for improving provider performance that may ultimately optimize patient outcomes from cardiac arrest.

Starting to Think Like an Expert: An Analysis of Resident Cognitive Processes During Simulation-Based Resuscitation Examinations

STUDY OBJECTIVE

Simulation is commonly used to teach crisis resource management skills and assess them in emergency medicine residents. However, our understanding of the cognitive processes underlying crisis resource management skills is limited because these processes are difficult to assess and describe. The objective of this study is to uncover and characterize the cognitive processes underlying crisis resource management skills and to describe how these processes vary between residents according to performance in a simulation-based examination.

METHODS

Twenty-two of 24 eligible emergency medicine trainees from 1 tertiary academic center completed 1 or 2 resuscitation-based examinations in the simulation laboratory. Resident performance was assessed by a blinded expert using an entrustment-based scoring tool. Participants wore eye-tracking glasses that generated first-person video that was used to augment subsequent interviews led by an emergency medicine faculty member. Interviews were audio recorded and then transcribed. An emergent thematic analysis was completed with a codebook that was developed by 4 research assistants, with subsequent analyses conducted by the lead research assistant with input from emergency medicine faculty. Themes from high- and low-performing residents were subsequently qualitatively compared.

RESULTS

Higher-performing residents were better able to anticipate, selectively attend to relevant information, and manage cognitive demands, and took a concurrent (as opposed to linear) approach to managing the simulated patient.

CONCLUSION

The results provide new insights into residents' cognitive processes while managing simulated patients in an examination environment and how these processes vary with performance. More work is needed to determine how best to apply these findings to improve crisis resource management education.

Trios-OSCE-based simulation course enhances the subcompetency of emergency-stabilization for postgraduate year-1 residents

BACKGROUND

For patient safety, this study aims to evaluate the effectiveness of additional objective structured clinical examination (OSCE)-based medical simulation courses to establish the "emergency-stabilization" subcompetency of postgraduate first year (PGY-1) residents.

METHODS

In the simulation course, trainees were randomly divided into three groups: intervention, regular, and control group as Trios-OSCE trainees, Single-OSCE trainees, or OSCE observers (feedback-givers) after attending the pre-OSCE common simulation workshop. Three PGY-1 residents rotated through the Trios OSCE long-station together, while single PGY-1 residents rotated through regular OSCE alone and the control group gave feedback after observation of their peers' OSCE performance. Using Queen's simulation assessment tool, either in Trios-OSCE or Single-OSCE, performance levels were rated as either inferior, novice, competent, advanced or superior in the "therapeutic actions" and "communication" domains. The "overall performances" of all trainees were graded by qualified assessors, experienced facilitators, and standardized senior nurse.

RESULTS

The proportion of "overall performance" of trainee's, rated by an experienced facilitator as "above competent level," was significantly higher in intervention group A than in regular group B. After training, the degree of increase in self-efficacy scores was higher among the intervention group than the regular and control groups. In the follow-up stage, a trend of increasing self-efficacy scores was noted in both the interventional and regular groups. For all trainees among the three groups, high postcourse value scores confirm that the new Trios-OSCE model meets the needs of trainees and also motivates the self-directed learning and self-reflection of trainees.

CONCLUSION

Our results provide initial evidence that the new emergency-stabilization-enhanced Trios-OSCE-based medical simulation course including the additional training capacity offered by adding an observer group had positive effects on PGY-1 residents' self-efficacy and clinical transfer.

Simulation versus real-world performance: a direct comparison of emergency medicine resident resuscitation entrustment scoring

Background

Simulation is increasingly being used in postgraduate medical education as an opportunity for competency assessment. However, there is limited direct evidence that supports performance in the simulation lab as a surrogate of workplace-based clinical performance for non-procedural tasks such as resuscitation in the emergency department (ED). We sought to directly compare entrustment scoring of resident performance in the simulation environment to clinical performance in the ED.

Methods

The resuscitation assessment tool (RAT) was derived from the previously implemented and studied Queen's simulation assessment tool (QSAT) via a modified expert review process. The RAT uses an anchored global assessment scale to generate an entrustment score and narrative comments. Emergency medicine (EM) residents were assessed using the RAT on cases in simulation-based examinations and in the ED during resuscitation cases from July 2016 to June 2017. Resident mean entrustment scores were compared using Pearson's correlation coefficient to determine the relationship between entrustment in simulation cases and in the ED. Inductive thematic analysis of written commentary was conducted to compare workplace-based with simulation-based feedback.

Results

There was a moderate, positive correlation found between mean entrustment scores in the simulated and workplace-based settings, which was statistically significant (r = 0.630, n = 17, p < 0.01). Further, qualitative analysis demonstrated overall management and leadership themes were more common narratives in the workplace, while more specific task-based feedback predominated in the simulation-based assessment. Both workplace-based and simulation-based narratives frequently commented on communication skills.

Conclusions

In this single-center study with a limited sample size, assessment of residents using entrustment scoring in simulation settings was demonstrated to have a moderate positive correlation with assessment of resuscitation competence in the workplace. This study suggests that resuscitation performance in simulation settings may be an indicator of competence in the clinical setting. However, multiple factors contribute to this complicated and imperfect relationship. It is imperative to consider narrative comments in supporting the rationale for numerical entrustment scores in both settings and to include both simulation and workplace-based assessment in high-stakes decisions of progression.

Assessment of Emergency Medicine Resident Performance in an Adult Simulation Using a Multisource Feedback Approach

Introduction

The Accreditation Council for Graduate Medical Education (ACGME) specifically notes multisource feedback (MSF) as a recommended means of resident assessment in the emergency medicine (EM) Milestones. High-fidelity simulation is an environment wherein residents can receive MSF from various types of healthcare professionals. Previously, the Queen’s Simulation Assessment Tool (QSAT) has been validated for faculty to assess residents in five categories: assessment; diagnostic actions; therapeutic actions; interpersonal communication, and overall assessment. We sought to determine whether the QSAT could be used to provide MSF using a standardized simulation case.

Methods

Prospectively after institutional review board approval, residents from a dual ACGME/osteopathic-approved postgraduate years (PGY) 1–4 EM residency were consented for participation. We developed a standardized resuscitation after overdose case with specific 1–5 Likert anchors used by the QSAT. A PGY 2–4 resident participated in the role of team leader, who completed a QSAT as self-assessment. The team consisted of a PGY-1 peer, an emergency medical services (EMS) provider, and a nurse. Two core faculty were present to administer the simulation case and assess. Demographics were gathered from all participants completing QSATs. We analyzed QSATs by each category and on cumulative score. Hypothesis testing was performed using intraclass correlation coefficients (ICC), with 95% confidence intervals. Interpretation of ICC results was based on previously published definitions.

Results

We enrolled 34 team leader residents along with 34 nurses. A single PGY-1, a single EMS provider and two faculty were also enrolled. Faculty provided higher cumulative QSAT scores than the other sources of MSF. QSAT scores did not increase with team leader PGY level. ICC for inter-rater reliability for all sources of MSF was 0.754 (0.572–0.867). Removing the self-evaluation scores increased inter-rater reliability to 0.838 (0.733–0.910). There was lesser agreement between faculty and nurse evaluations than from the EMS or peer evaluation.

Conclusion

In this single-site cohort using an internally developed simulation case, the QSAT provided MSF with excellent reliability. Self-assessment decreases the reliability of the MSF, and our data suggest self-assessment should not be a component of MSF. Use of the QSAT for MSF may be considered as a source of data for clinical competency committees.

A new way to look at simulation-based assessment: the relationship between gaze-tracking and exam performance

Objective

A key task of the team leader in a medical emergency is effective information gathering. Studying information gathering patterns is readily accomplished with the use of gaze-tracking glasses. This technology was used to generate hypotheses about the relationship between performance scores and expert-hypothesized visual areas of interest in residents across scenarios in simulated medical resuscitation examinations.

Methods

Emergency medicine residents wore gaze-tracking glasses during two simulation-based examinations (n=29 and 13 respectively). Blinded experts assessed video-recorded performances using a simulation performance assessment tool that has validity evidence in this context. The relationships between gaze patterns and performance scores were analyzed and potential hypotheses generated. Four scenarios were assessed in this study: diabetic ketoacidosis, bradycardia secondary to beta-blocker overdose, ruptured abdominal aortic aneurysm and metabolic acidosis caused by antifreeze ingestion.

Results

Specific gaze patterns were correlated with objective performance. High performers were more likely to fixate on task-relevant stimuli and appropriately ignore task-irrelevant stimuli compared with lower performers. For example, shorter latency to fixation on the vital signs in a case of diabetic ketoacidosis was positively correlated with performance (r=0.70, p<0.05). Conversely, total time spent fixating on lab values in a case of ruptured abdominal aortic aneurysm was negatively correlated with performance (r= −0.50, p<0.05).

Conclusions

There are differences between the visual patterns of high and low-performing residents. These findings may allow for better characterization of expertise development in resuscitation medicine and provide a framework for future study of visual behaviours in resuscitation cases.

Impact of night shifts on emergency medicine resident resuscitation performance

AIM

Emergency medicine (EM) trainees often work nightshifts. We sought to measure how this circadian disruption affects EM resident performance during simulated resuscitations.

METHODS

This retrospective cohort study enrolled EM residents at a single Canadian academic centre over a six-year period. Residents completed twice-annual simulation-based resuscitation-focused objective structured clinical examinations (OSCEs) with assessment in four domains (primary assessment, diagnostic actions, therapeutic actions and communication), and a global assessment score (GAS). Primary and secondary exposures of interest were the presence of a nightshift (late-evening shifts ending between midnight and 03h00 or overnight shifts ending after 06h00) the day before or within three days before an OSCE. A random effects linear regression model was used to quantify the association between nightshifts and OSCE scores.

RESULTS

From 57 residents, 136 OSCE scores were collected. Working a nightshift the day before an OSCE did not affect male trainee scores but was associated with a significant absolute decrease in mean total scores (-6% [95% CI -12% to 0%]), GAS (-7% [-13% to 0%]), and communication (-9% [-16% to -2%]) scores among women. Working any nightshift within three days before an OSCE lowered absolute mean total scores by 4% [-7% to 0%] and communication scores by 5% [-5% to 0%] irrespective of gender.

CONCLUSION

Our results suggest that shift work may impact EM resident resuscitation performance, particularly in the communication domain. This impact may be more significant in women than men, suggesting a need for further investigation.

Comparison of Simulation-based Resuscitation Performance Assessments With In-training Evaluation Reports in Emergency Medicine Residents: A Canadian Multicenter Study

OBJECTIVE

Simulation stands to serve an important role in modern competency-based programs of assessment in postgraduate medical education. Our objective was to compare the performance of individual emergency medicine (EM) residents in a simulation-based resuscitation objective structured clinical examination (OSCE) using the Queen's Simulation Assessment Tool (QSAT), with portfolio assessment of clinical encounters using a modified in-training evaluation report (ITER) to understand in greater detail the inferences that may be drawn from a simulation-based OSCE assessment.

METHODS

A prospective observational study was employed to explore the use of a multicenter simulation-based OSCE for evaluation of resuscitation competence. EM residents from five Canadian academic sites participated in the OSCE. Video-recorded performances were scored by blinded raters using the scenario-specific QSATs with domain-specific anchored scores (primary assessment, diagnostic actions, therapeutic actions, communication) and a global assessment score (GAS). Residents' portfolios were evaluated using a modified ITER subdivided by CanMEDS roles (medical expert, communicator, collaborator, leader, health advocate, scholar, and professional) and a GAS. Correlational and regression analyses were performed comparing components of each of the assessment methods.

RESULTS

Portfolio review and ITER scoring was performed for 79 residents participating in the simulation-based OSCE. There was a significant positive correlation between total OSCE and ITER scores (r = 0.341). The strongest correlations were found between ITER medical expert score and each of the OSCE GAS (r = 0.420), communication (r = 0.443), and therapeutic action (r = 0.484) domains. ITER medical expert was a significant predictor of OSCE total (p = 0.002). OSCE therapeutic action was a significant predictor of ITER total (p = 0.02).

CONCLUSIONS

Simulation-based resuscitation OSCEs and portfolio assessment captured by ITERs appear to measure differing aspects of competence, with weak to moderate correlation between those measures of conceptually similar constructs. In a program of competency-based assessment of EM residents, a simulation-based OSCE using the QSAT shows promise as a tool for assessing medical expert and communicator roles.

The Nightmares Course: A Longitudinal, Multidisciplinary, Simulation-Based Curriculum to Train and Assess Resident Competence in Resuscitation

BACKGROUND

Postgraduate medical education programs would benefit from a robust process for training and assessment of competence in resuscitation early in residency.

OBJECTIVE

To describe and evaluate the Nightmares Course, a novel, competency-based, transitional curriculum and assessment program in resuscitation medicine at Queen's University in Kingston, Ontario, Canada.

METHODS

First-year residents participated in the longitudinal Nightmares Course at Queen's University during the 2015-2016 academic year. An expert working group developed the entrustable professional activity and curricular design for the course. Formative feedback was provided following each simulation-based session, and we employed a summative objective structured clinical examination (OSCE) utilizing a modified Queen's Simulation Assessment Tool. A generalizability study and resident surveys were performed to evaluate the course and assessment process.

RESULTS

A total of 40 residents participated in the course, and 23 (58%) participated in the OSCE. Eight of 23 (35%) did not meet the predetermined competency threshold and required remediation. The OSCE demonstrated an acceptable phi coefficient of 0.73. The approximate costs were $240 per Nightmares session, $10,560 for the entire 44-session curriculum, and $3,900 for the summative OSCE.

CONCLUSIONS

The Nightmares Course demonstrated feasibility and acceptability, and is applicable to a broad array of postgraduate medical education programs. The entrustment-based assessment detected several residents not meeting a minimum competency threshold, and directed them to additional training.

Development of an Assessment for Entrustable Professional Activity (EPA) 10: Emergent Patient Management

INTRODUCTION

Medical schools in the United States are encouraged to prepare and certify the entrustment of medical students to perform 13 core entrustable professional activities (EPAs) prior to graduation. Entrustment is defined as the informed belief that the learner is qualified to autonomously perform specific patient-care activities. Core EPA-10 is the entrustment of a graduate to care for the emergent patient. The purpose of this project was to design a realistic performance assessment method for evaluating fourth-year medical students on EPA-10.

METHODS

First, we wrote five emergent patient case-scenarios that a medical trainee would likely confront in an acute care setting. Furthermore, we developed high-fidelity simulations to realistically portray these patient case scenarios. Finally, we designed a performance assessment instrument to evaluate the medical student's performance on executing critical actions related to EPA-10 competencies. Critical actions included the following: triage skills, mustering the medical team, identifying causes of patient decompensation, and initiating care. Up to four students were involved with each case scenario; however, only the team leader was evaluated using the assessment instruments developed for each case.

RESULTS

A total of 114 students participated in the EPA-10 assessment during their final year of medical school. Most students demonstrated competence in recognizing unstable vital signs (97%), engaging the team (93%), and making appropriate dispositions (92%). Almost 87% of the students were rated as having reached entrustment to manage the care of an emergent patient (99 of 114). Inter-rater reliability varied by case scenario, ranging from moderate to near-perfect agreement. Three of five case-scenario assessment instruments contained items that were internally consistent at measuring student performance. Additionally, the individual item scores for these case scenarios were highly correlated with the global entrustment decision.

CONCLUSION

High-fidelity simulation showed good potential for effective assessment of medical student entrustment of caring for the emergent patient. Preliminary evidence from this pilot project suggests content validity of most cases and associated checklist items. The assessments also demonstrated moderately strong faculty inter-rater reliability.