Twelve tips for developing simulation-based mastery learning clinical skills checklists

Simulation-based mastery learning is a powerful educational paradigm that leads to high levels of performance through a combination of strict standards, deliberate practice, formative feedback, and rigorous assessment. Successful mastery learning curricula often require well-designed checklists that produce reliable data that contribute to valid decisions. The following twelve tips are intended to help educators create defensible and effective clinical skills checklists for use in mastery learning curricula. These tips focus on defining the scope of a checklist using established principles of curriculum development, crafting the checklist based on a literature review and expert input, revising and testing the checklist, and recruiting judges to set a minimum passing standard. While this article has a particular focus on mastery learning, with the exception of the tips related to standard setting, the general principles discussed apply to the development of any clinical skills checklist.

Communicating Diagnostic Uncertainty at Emergency Department Discharge: A Simulation-Based Mastery Learning Randomized Trial

PURPOSE

There are no standardized approaches for communicating with patients discharged from the emergency department with diagnostic uncertainty. This trial tested efficacy of the Uncertainty Communication Education Module, a simulation-based mastery learning curriculum designed to establish competency in communicating diagnostic uncertainty.

METHOD

Resident physicians at 2 sites participated in a 2-arm waitlist randomized controlled trial from September 2019 to June 2020. After baseline (T1) assessment of all participants via a standardized patient encounter using the Uncertainty Communication Checklist (UCC), immediate access physicians received training in the Uncertainty Communication Education Module, which included immediate feedback, online educational modules, a smartphone-based application, and telehealth deliberate practice with standardized patients. All physicians were retested 16-19 weeks later (T2) via in-person standardized patient encounters; delayed access physicians then received the intervention. A final test of all physicians occurred 11-15 weeks after T2 (T3). The primary outcome measured the percentage of physicians in the immediate versus delayed access groups meeting or exceeding the UCC minimum passing standard at T2.

RESULTS

Overall, 109 physicians were randomized, with mean age 29 years (range 25-46). The majority were male (n = 69, 63%), non-Hispanic/Latino (n = 99, 91%), and White (n = 78, 72%). At T2, when only immediate access participants had received the curriculum, immediate access physicians demonstrated increased mastery (n = 29, 52.7%) compared with delayed access physicians (n = 2, 3.7%, P < .001; estimated adjusted odds ratio of mastery for the immediate access participants, 31.1 [95% CI, 6.8-143.1]). There were no significant differences when adjusting for training site or stage of training.

CONCLUSIONS

The Uncertainty Communication Education Module significantly increased mastery in communicating diagnostic uncertainty at the first postintervention test among emergency physicians in standardized patient encounters. Further work should assess the impact of clinical implementation of these communication skills.

Simulation-based Mastery Learning Improves Emergency Medicine Residents' Ability to Perform Temporary Transvenous Cardiac Pacing

INTRODUCTION

Temporary transvenous cardiac pacing (TVP) is a critical intervention that emergency physicians perform infrequently in clinical practice. Prior simulation studies revealed that emergency medicine (EM) residents and board-certified emergency physicians perform TVP poorly during checklist-based assessments. Our objective in this report was to describe the design and implementation of a simulation-based mastery learning (SBML) curriculum and evaluate its impact on EM residents' ability to perform TVP.

METHODS

An expert panel of emergency physicians and cardiologists set a minimum passing standard (MPS) for a previously developed 30-item TVP checklist using the Mastery Angoff approach. Emergency medicine residents were assessed using this checklist and a high-fidelity TVP task trainer. Residents who did not meet the MPS during baseline testing viewed a procedure video and completed a 30-minute individual deliberate practice session before retesting. Residents who did not meet the MPS during initial post-testing completed additional deliberate practice and assessment until meeting or exceeding the MPS.

RESULTS

The expert panel set an MPS of correctly performing 28 (93.3%) checklist items. Fifty-seven EM residents participated. Mean checklist scores improved from 13.4 (95% CI 11.8-15.0) during baseline testing to 27.5 (95% CI 26.9-28.1) during initial post-testing (P < 0.01). No residents met the MPS at baseline testing. The 21 (36.8%) residents who did not meet the MPS during initial post-testing all met or exceeded the MPS after completing one additional 30-minute deliberate practice session.

CONCLUSION

Emergency medicine residents demonstrated significantly improved TVP performance with reduced variability in checklist scores after completing a simulation-based mastery learning curriculum.

"Unfortunately, I don't have an answer for you": How resident physicians communicate diagnostic uncertainty to patients during emergency department discharge

OBJECTIVE

To describe how emergency medicine resident physicians discuss diagnostic uncertainty during a simulated ED discharge discussion.

METHODS

A secondary content analysis of simulated clinical encounter audiotapes completed by emergency medicine residents across two sites.

RESULTS

When discussing lack of diagnosis, residents explained the evaluation revealed no cause for symptoms, noted concerning diagnoses that were excluded, and acknowledged both symptoms and patients' feelings. Residents used explicit and implicit language to discuss diagnostic uncertainty with similar frequency. Almost half of the residents discussed the ED role as focused on emergent illness to give patients context for their uncertain diagnoses. However, 28% of residents in this study did not discuss diagnostic uncertainty in any form. All residents provided reassurance.

CONCLUSION

Residents use a range of approaches to discuss diagnostic uncertainty with patients at the time of a simulated ED discharge, with some residents omitting discussion of uncertainty entirely.

PRACTICE IMPLICATIONS

These findings represent the current state of communication, which needs improvement. These findings do not immediately transfer to clinical practice recommendations, but rather support a need for both further study and development of formal communication training on this topic.

There's an app for that: Teaching residents to communicate diagnostic uncertainty through a mobile gaming application

BACKGROUND

Clear communication is integral to good clinical care; however, communication training is cost and time intensive. Mobile applications (apps) may provide a useful adjunct to traditional simulation skills training.

OBJECTIVE

To evaluate (1) use of an app for teaching communication skills about diagnostic uncertainty, (2) feedback on app use, and (3) the association between use and skill mastery.

PATIENT INVOLVEMENT

The app under study is designed to improve doctor-patient communication.

METHODS

The study was a planned sub-analysis of a randomized controlled waitlist trial with emergency medicine resident physicians randomized to receive immediate or delayed access to an educational curriculum focused on diagnostic uncertainty. The curriculum included a web-based interactive module and the app. Metrics describing participants' use of the app, feedback on use, and association of use and achieving mastery in communicating diagnostic uncertainty are reported. Differences between groups utilizing the app were analyzed using Chi-squared test; logistic regression assessed the association between app use and achieving mastery of the communication skill.

RESULTS

Among 109 participants completing the trial, only 34 (31.2%) used the app. Most participants engaged with the app on one occasion for a median of 50 min (IQR 31, 87). Senior residents were more likely to use the app than junior residents (41.3% vs 23.8%, p=0.05). Overall reviews were positive; 76% reported the app helped them learn. There was no significant association between app use and achieving mastery of the communication skill in the trial [OR 2.1, 95% CI (0.91-4.84)].

DISCUSSION

Despite positive reviews of app use, overall use was low and there was no association with achieving mastery.

PRACTICAL VALUE

Offering an app as an auxiliary training opportunity may be beneficial to some residents, but shouldn't be planned for use as a primary didactic modality unless there is evidence for effectiveness and use is mandated.

AVAILABILITY OF DATA AND MATERIALS

The datasets generated and/or analyzed during the current study are not publicly available since some data may be identifiable but are available from the corresponding author on reasonable request.

Developing standardized patient-based cases for communication training: lessons learned from training residents to communicate diagnostic uncertainty

Health professions education has benefitted from standardized patient (SP) programs to develop and refine communication and interpersonal skills in trainees. Effective case design is essential to ensure an SP encounter successfully meets learning objectives that are focused on communication skills. Creative, well-designed case scenarios offer learners the opportunity to engage in complex patient encounters, while challenging them to address the personal and emotional contexts in which their patients are situated. Therefore, prior to considering the practical execution of the patient encounter, educators will first need a clear and structured strategy for writing, organizing, and developing cases. The authors reflect on lessons learned in developing standardized patient-based cases to train learners to communicate to patients during times of diagnostic uncertainty, and provide suggestions to develop a set of simulation cases that are both standardized and diverse. Key steps and workflow processes that can assist educators with case design are introduced. The authors review the need to increase awareness of and mitigate existing norms and implicit biases, while maximizing variation in patient diversity. Opportunities to leverage the breadth of emotional dispositions of the SP and the affective domain of a clinical encounter are also discussed as a means to guide future case development and maximize the value of a case for its respective learning outcomes.

Development of a Rigorously Designed Procedural Checklist for Assessment of Emergency Medicine Resident Performance of Temporary Transvenous Cardiac Pacing

OBJECTIVES

Temporary transvenous cardiac pacing (TVP) is a potentially lifesaving intervention included in the list of essential core procedures for emergency medicine (EM) training; however, opportunities to perform TVP during residency cannot be guaranteed. EM graduates report feeling subjectively underprepared for this procedure, but objective performance data are lacking. Checklist-based simulated assessment is an increasing focus of competency-based medical education, particularly for invasive procedures like TVP. The objectives of this paper were as follows: first, to enlist a multidisciplinary team of experts to create an assessment tool for TVP using best practices in checklist development; second, to determine the reliability of checklist scoring; and third, to assess EM residents' baseline ability to perform TVP using a dedicated task trainer.

METHODS

This study was conducted at a single 4-year EM residency. A panel of emergency physicians and cardiologists designed a TVP checklist using a modified Delphi approach. After consensus was achieved on a final checklist, EM residents were assessed using a dedicated TVP task trainer. Inter-rater reliability was determined using Cohen's kappa coefficient. Resident performance was determined by number of correctly performed checklist items.

RESULTS

The expert panel achieved consensus on a 30-item checklist after three rounds of revisions. The Cohen's kappa coefficient for the overall checklist score was 0.87, with individual checklist items ranging from 0.63 to 1.00. In total, 58 residents were assessed with a mean score of 13.5 of 30 checklist items. Scores increased with each year of training.

CONCLUSIONS

This study details the rigorous development of a TVP checklist designed by a multidisciplinary team of experts. Checklist scores demonstrated strong inter-rater reliability. The overall poor performance of this cohort suggests the current approach to TVP training does not provide sufficient preparation for EM residents. Competency-based techniques, such as simulation-based mastery learning, should be explored.

Use of a simulation-based mastery learning curriculum for neurology residents to improve the identification and management of status epilepticus

BACKGROUND

Appropriate and timely treatment of status epilepticus (SE) decreases morbidity and mortality. Therefore, skill-based training in the identification and management of SE is crucial for clinicians.

OBJECTIVE

The objective of the study was to develop and evaluate the impact of a simulation-based mastery learning (SBML) curriculum to train neurology residents on the identification and management of SE.

METHODS

We used pretest-posttest design with a retention test on SE skills for this study. Neurology residents in the second postgraduate year (PGY-2) were eligible to participate in the SE SBML curriculum. Learners completed a baseline-simulated SE skills assessment (pretest) using a 26-item dichotomous skills checklist. Next, they participated in a didactic session about the identification and management of SE, followed by deliberate skills practice. Subsequently, participants completed another skills assessment (posttest) using the same 26-item checklist. All participants were required to meet or exceed a minimum passing standard (MPS) determined by a panel of 14 SE experts using the Mastery Angoff standard setting method. After meeting the MPS at posttest, participants were reassessed during an unannounced in situ simulation session on the medical wards. We compared pretest with posttest simulated SE skills performance and posttest with reassessment in situ performance.

RESULTS

The MPS was set at 88% (23/26) checklist items correct. Sixteen neurology residents participated in the intervention. Participant performance improved from a median of 44.23% (Interquartile range (IQR): 34.62-55.77) at pretest to 94.23% (IQR: 92.13-100) at the posttest after SBML (p < .001). There was no significant difference in scores between the posttest and in situ test up to 8 months later (94.23%; IQR: 92.31-100 vs. 92.31%; IQR: 88.46-96.15; p = .13).

CONCLUSIONS

Our SBML curriculum significantly improved residents' SE identification and management skills that were largely retained during an unannounced simulated encounter in the hospital setting.

Creation and Implementation of a Mastery Learning Curriculum for Emergency Department Thoracotomy

Introduction

Emergency department thoracotomy (EDT) is a lifesaving procedure within the scope of practice of emergency physicians. Because EDT is infrequently performed, emergency medicine (EM) residents lack opportunities to develop procedural competency. There is no current mastery learning curriculum for residents to learn EDT. The purpose of this study was to develop and implement a simulation-based mastery learning curriculum to teach and assess EM residents’ performance of the EDT.

Methods

We developed an EDT curriculum using a mastery learning framework. The minimum passing standard (MPS) for a previously developed 22-item checklist was determined using the Mastery Angoff approach. EM residents at a four-year academic EM residency program underwent baseline testing in performing an EDT on a simulation trainer. Performance was scored by two raters using the checklist. Learners then participated in a novel mastery learning EDT curriculum that included an educational video, hands-on instruction, and deliberate practice. After a three-month period, residents then completed initial post testing. Residents who did not meet the minimum passing standard after post testing participated in additional deliberate practice until mastery was obtained. Baseline and post-test scores, and time to completion of the procedure were compared with paired t-tests.

Results

Of 56 eligible EM residents, 54 completed baseline testing. Fifty-two residents completed post-testing until mastery was reached. The minimum passing standard was 91.1%, (21/22 items correct on the checklist). No participants met the MPS at the baseline assessment. After completion of the curriculum, all residents subsequently reached the MPS, with deliberate practice sessions not exceeding 40 minutes. Scores from baseline testing to post-testing significantly improved across all postgraduate years from a mean score of 10.2/22 to 21.4/22 (p <0.001). Mean time to complete the procedure improved from baseline testing (6 minutes [min] and 21 seconds [sec], interquartile range [IQR] = 4 min 54 sec - 7 min 51 sec) to post-testing (5 min 19 seconds, interquartile range 4 min 17sec - 6 min 15 sec; p = 0.001).

Conclusion

This simulation-based mastery learning curriculum resulted in all residents performing an EDT at a level that met or exceeded the MPS with an overall decrease in time needed to perform the procedure.

Development of the Uncertainty Communication Checklist: A Patient-Centered Approach to Patient Discharge From the Emergency Department

Clear communication with patients upon emergency department (ED) discharge is important for patient safety during the transition to outpatient care. Over one-third of patients are discharged from the ED with diagnostic uncertainty, yet there is no established approach for effective discharge communication in this scenario. From 2017 to 2019, the authors developed the Uncertainty Communication Checklist for use in simulation-based training and assessment of emergency physician communication skills when discharging patients with diagnostic uncertainty. This development process followed the established 12-step Checklist Development Checklist framework and integrated patient feedback into 6 of the 12 steps. Patient input was included as it has potential to improve patient-centeredness of checklists related to assessment of clinical performance. Focus group patient participants from 2 clinical sites were included: Thomas Jefferson University Hospital, Philadelphia, PA, and Northwestern University Hospital, Chicago, Illinois.The authors developed a preliminary instrument based on existing checklists, clinical experience, literature review, and input from an expert panel comprising health care professionals and patient advocates. They then refined the instrument based on feedback from 2 waves of patient focus groups, resulting in a final 21-item checklist. The checklist items assess if uncertainty was addressed in each step of the discharge communication, including the following major categories: introduction, test results/ED summary, no/uncertain diagnosis, next steps/follow-up, home care, reasons to return, and general communication skills. Patient input influenced both what items were included and the wording of items in the final checklist. This patient-centered, systematic approach to checklist development is built upon the rigor of the Checklist Development Checklist and provides an illustration of how to integrate patient feedback into the design of assessment tools when appropriate.

Emergency Department Thoracotomy: Development of a Reliable, Validated Checklist for Procedural Training

OBJECTIVES

Emergency department thoracotomy (EDT) is a rare and challenging procedure. Emergency medicine (EM) residents have limited opportunities to perform the procedure in clinical or educational settings. Standardized, reliable, validated checklists do not exist to evaluate procedural competency. The objectives of this project were twofold: 1) to develop a checklist containing the critical actions for performing an EDT that can be used for future procedural skills training and 2) to evaluate the reliability and validity of the checklist for performing EDT.

METHODS

After a literature review, a preliminary 22-item checklist was developed and disseminated to experts in EM and trauma surgery. A modified Delphi method was used to revise the checklist. To assess usability of the checklist, EM and trauma surgery faculty and residents were evaluated performing an EDT while inter-rater reliability was calculated with Cohen's kappa. A Student's t-test was used to compare the performance of participants who had or had not performed a thoracotomy in clinical practice. Item-total correlation was calculated for each checklist item to determine discriminatory ability.

RESULTS

A final 22-item checklist was developed for EDT. The overall inter-rater reliability was strong (κ = 0.84) with individual item agreement ranging from moderate to strong (κ = 0.61 to 1.00). Experts (attending physicians and senior residents) performed well on the checklist, achieving an average score of 80% on the checklist. Participants who had performed EDT in clinical practice performed significantly better than those that had not, achieving an average of 80.7% items completed versus 52.3% (p < 0.05). Seventeen of 22 items had an item-total correlation greater than 0.2.

CONCLUSIONS

A final 22-item consensus-based checklist was developed for the EDT. Overall inter-rater reliability was strong. This checklist can be used in future studies to serve as a foundation for curriculum development around this important procedure.

Simulation-based mastery learning compared to standard education for discussing diagnostic uncertainty with patients in the emergency department: a randomized controlled trial

BACKGROUND

Diagnostic uncertainty occurs frequently in emergency medical care, with more than one-third of patients leaving the emergency department (ED) without a clear diagnosis. Despite this frequency, ED providers are not adequately trained on how to discuss diagnostic uncertainty with these patients, who often leave the ED confused and concerned. To address this training need, we developed the Uncertainty Communication Education Module (UCEM) to teach physicians how to discuss diagnostic uncertainty. The purpose of the study is to evaluate the effectiveness of the UCEM in improving physician communications.

METHODS

The trial is a multicenter, two-arm randomized controlled trial designed to teach communication skills using simulation-based mastery learning (SBML). Resident emergency physicians from two training programs will be randomly assigned to immediate or delayed receipt of the two-part UCEM intervention after completing a baseline standardized patient encounter. The two UCEM components are: 1) a web-based interactive module, and 2) a smart-phone-based game. Both formats teach and reinforce communication skills for patient cases involving diagnostic uncertainty. Following baseline testing, participants in the immediate intervention arm will complete a remote deliberate practice session via a video platform and subsequently return for a second study visit to assess if they have achieved mastery. Participants in the delayed intervention arm will receive access to UCEM and remote deliberate practice after the second study visit. The primary outcome of interest is the proportion of residents in the immediate intervention arm who achieve mastery at the second study visit.

DISCUSSION

Patients' understanding of the care they received has implications for care quality, safety, and patient satisfaction, especially when they are discharged without a definitive diagnosis. Developing a patient-centered diagnostic uncertainty communication strategy will improve safety of acute care discharges. Although use of SBML is a resource intensive educational approach, this trial has been deliberately designed to have a low-resource, scalable intervention that would allow for widespread dissemination and uptake.

TRIAL REGISTRATION

The trial was registered at clinicaltrials.gov (NCT04021771). Registration date: July 16, 2019.

Setting a Minimum Passing Standard for the Uncertainty Communication Checklist Through Patient and Physician Engagement

BACKGROUND

Historically, medically trained experts have served as judges to establish a minimum passing standard (MPS) for mastery learning. As mastery learning expands from procedure-based skills to patient-centered domains, such as communication, there is an opportunity to incorporate patients as judges in setting the MPS.

OBJECTIVE

We described our process of incorporating patients as judges to set the MPS and compared the MPS set by patients and emergency medicine residency program directors (PDs).

METHODS

Patient and physician panels were convened to determine an MPS for a 21-item Uncertainty Communication Checklist. The MPS for both panels were independently calculated using the Mastery Angoff method. Mean scores on individual checklist items with corresponding 95% confidence intervals were also calculated for both panels and differences analyzed using a t test.

RESULTS

Of 240 eligible patients and 42 eligible PDs, 25 patients and 13 PDs (26% and 65% cooperation rates, respectively) completed MPS-setting procedures. The patient-generated MPS was 84.0% (range 45.2-96.2, SD 10.2) and the physician-generated MPS was 88.2% (range 79.7-98.1, SD 5.5). The overall MPS, calculated as an average of these 2 results, was 86.1% (range 45.2-98.1, SD 9.0), or 19 of 21 checklist items.

CONCLUSIONS

Patients are able to serve as judges to establish an MPS using the Mastery Angoff method for a task performed by resident physicians. The patient-established MPS was nearly identical to that generated by a panel of residency PDs, indicating similar expectations of proficiency for residents to achieve skill "mastery."

A Mastery Learning Capstone Course to Teach and Assess Components of Three Entrustable Professional Activities to Graduating Medical Students

PROBLEM

Thirteen measurable Entrustable Professional Activities (EPAs) have been proposed by the Association of American Medical Colleges as a means to operationalize medical school graduates' patient care qualifications. Mastery learning is an effective method for boosting clinical skills, but its applicability to the EPAs remains to be studied. The authors designed this study to evaluate a mastery learning intervention to teach and assess components of 3 of the 13 EPAs in a 4th-year capstone course.

INTERVENTION

The course featured mastery learning principles and addressed three EPA-based skills: (a) obtain informed consent, (b) develop a differential diagnosis and write admission orders, and (c) write discharge prescriptions. All students underwent a baseline skills assessment, received feedback, engaged in deliberate practice with actionable feedback, and completed a similar skills-based posttest assessment. Students continued with practice and testing until the minimum passing standards (MPSs) were reached for each posttest.

CONTEXT

All medical students at a single medical school (Northwestern University, Feinberg School of Medicine) who matriculated in 2012 and graduated with the class of 2016 participated in a required transition to residency course immediately prior to graduation.

OUTCOME

There were 134 students eligible to participate, and 130 (97.0%) completed all curricular requirements and assessments. All 130 medical students who completed the course met or exceeded the MPS for each of the three EPA-based clinical skills. Reliability coefficients for outcome data were uniformly high. Measures for each of the three clinical skills showed statistically significant improvement.

LESSONS LEARNED

The capstone course was an effective approach to teach and assess components of three EPA-based clinical skills to mastery learning standards in a 4th-year capstone course. We learned that this approach for implementation is feasible and results in significant improvement in components of EPA skill performance. Next steps will include developing assessments incorporating the mastery model into components of additional EPAs, identifying the best location within the curriculum to insert this content, and expanding the number of assessments as part of a larger assessment system.

Orientation to the Operating Room: An Introduction to the Surgery Clerkship for Third-Year Medical Students

Introduction

The operating room is a complex environment in which individual team members perform specific tasks according to their role. A simulation activity was created to introduce medical students on the surgery clerkship to issues relating to patient safety, infection control, and regulatory requirements.

Methods

This activity takes place prior to general surgery rotation operative experiences, and addresses the need for students to practice roles they will perform while participating in patient care. The activity includes a simulated operation, an assessment, and a scripted debriefing. Among other tasks, students practice safe patient transfer and monitoring, donning sterile garb, preparing the surgical site, and being active participants in a sign-in and time-out. Students are assessed on assigned tasks, their ability to maintain sterility, and the degree to which they engage with their team.

Results

Students reported the simulation helped them better understand how they could become involved on their first day in the operating room. Students also reported they were more confident when in the operating room. This finding also extended to students who had previously been in the operating room during a prior OB/GYN rotation.

Discussion

Patient safety is paramount when in the operating room, and this simulation activity fills a current gap in student's practical knowledge as they prepare to enter their surgery clerkship. Giving medical students the information and skills needed to be safe and effective members of the operating team prior to entering the operating room is of benefit to the surgical team, students, and patients.

An institution-wide approach to submission, review, and funding of simulation-based curricula

This article describes the development, implementation, and modification of an institutional process to evaluate and fund graduate medical education simulation curricula. The goals of this activity were to (a) establish a standardized mechanism for proposal submission and evaluation, (b) identify simulation-based medical education (SBME) curricula that would benefit from mentored improvement before implementation, and (c) ensure that funding decisions were fair and defensible. Our intent was to develop a process that was grounded in sound educational principles, allowed for efficient administrative oversight, ensured approved courses were high quality, encouraged simulation education research and scholarship, and provided opportunities for medical specialties that had not previously used SBME to receive mentoring and faculty development.

An Investigation of the Relationship Between Emergency Medicine Trainee Burnout and Clinical Performance in a High-fidelity Simulation Environment

OBJECTIVE

Burnout is prevalent among emergency medicine (EM) physicians, with physicians experiencing burnout more likely to report committing medical errors or delivering suboptimal care. The relationship between physician burnout and identifiable differences in clinical care, however, remains unclear. We examined if EM trainee burnout was associated with differences in clinical performance using high-fidelity simulation as a proxy for patient care.

METHODS

In this cross-sectional study across six institutions, we measured trainee performance over four simulation scenarios based on recognized EM milestones. For each scenario a faculty rater assessed whether the trainee performed predefined critical actions specific to each case. A summation of performed actions across all cases resulted in a cumulative task (CT) score (range = 0-85). Raters also assigned an impression score on a 10-point scale (0 = poor; 10 = outstanding) assessing the trainee's overall performance after each scenario, with the mean of the scores resulting in an overall impression (OI) score. After the simulation assessment, we measured trainees' burnout via the Maslach Burnout Inventory through a confidential, electronic survey. Trainee depression, quality of life (QOL) and daytime sleepiness were also evaluated. Survey results were compared to simulation scores using analysis of variance and covariance.

RESULTS

Fifty-eight of 89 (65.2%) eligible participants completed the survey and simulation assessment. Thirty-one of 58 (53.4%, 95% CI = 40.2% to 66.7%) trainees reported burnout. In trainees with burnout compared to those without, mean CT scores (73.4 vs. 75.2, 95% CI of difference = 0.06 to 3.51) and OI scores (6.4 vs 6.8, 95% CI of difference = 0.03 to 0.79) were negatively associated with burnout after controlling for training program. In contrast, QOL were positively associated with CT [F(1,48) = 4.796, p = 0.033] and OI [F(1,48) = 4.561, p = 0.038] scores. There were no significant associations between simulation performance and depression or daytime sleepiness.

CONCLUSION

Emergency medicine trainees with burnout received lower cumulative performance scores over four high-fidelity simulation scenarios than trainees without burnout.

Supplemental Milestones for Emergency Medicine Residency Programs: A Validation Study

Introduction

Emergency medicine (EM) residency programs may be 36 or 48 months in length. The Residency Review Committee for EM requires that 48-month programs provide educational justification for the additional 12 months. We developed additional milestones that EM training programs might use to assess outcomes in domains that meet this accreditation requirement. This study aims to assess for content validity of these supplemental milestones using a similar methodology to that of the original EM Milestones validation study.

Methods

A panel of EM program directors (PD) and content experts at two institutions identified domains of additional training not covered by the existing EM Milestones. This led to the development of six novel subcompetencies: “Operations and Administration,” “Critical Care,” “Leadership and Management,” “Research,” “Teaching and Learning,” and “Career Development.” Subject-matter experts at other 48-month EM residency programs refined the milestones for these subcompetencies. PDs of all 48-month EM programs were then asked to order the proposed milestones using the Dreyfus model of skill acquisition for each subcompetency. Data analysis mirrored that used in the original EM Milestones validation study, leading to the final version of our supplemental milestones.

Results

Twenty of 33 subjects (58.8%) completed the study. No subcompetency or individual milestone met deletion criteria. Of the 97 proposed milestones, 67 (69.1%) required no further editing and remained at the same level as proposed by the study authors. Thirty milestones underwent level changes: 15 (15.5%) were moved one level up and 13 (13.4%) were moved one level down. One milestone (1.0%) in “Leadership and Management” was moved two levels up, and one milestone in “Operations and Administration” was moved two levels down. One milestone in “Research” was ranked by the survey respondents at one level higher than that proposed by the authors; however, this milestone was kept at its original level assignment.

Conclusion

Six additional subcompetencies were generated and assessed for content validity using the same methodology as was used to validate the current EM Milestones. These optional milestones may serve as an additional set of assessment tools that will allow EM residency programs to report these additional educational outcomes using a familiar milestone rubric.

Assessing practice-based learning and improvement

Assessment of practice-based learning and improvement (PBLI) is a core concept identified in several competency frameworks. This paper summarizes the current state of PBLI assessment as presented at the 2012 Academic Emergency Medicine consensus conference on education research in emergency medicine. Based on these findings and consensus achieved at the conference, seven recommendations have been identified for future research.

Language use in the informed consent discussion for emergency procedures

BACKGROUND

The field of health literacy has closely examined the readability of written health materials to optimize patient comprehension. Few studies have examined spoken communication in a way that is comparable to analyses of written communication.

PURPOSE

The study objective was to characterize the structural elements of residents' spoken words while obtaining informed consent.

METHODS

Twenty-six resident physicians participated in a simulated informed consent discussion with a standardized patient. Audio recordings of the discussions were transcribed and analyzed to assess grammar statistics for evaluating language complexity (e.g., reading grade level). Transcripts and time values were used to assess structural characteristics of the dialogue (e.g., interactivity).

RESULTS

Discussions were characterized by physician verbal dominance. The discussions were interactive but showed significant differences between the physician and patient speech patterns for all language complexity metrics.

CONCLUSIONS

In this study, physicians spoke significantly more and used more complex language than the patients.