Assessing patient care: summary of the breakout group on assessment of observable learner performance

Leadership Matters: Needs Assessment and Framework for the International Federation for Emergency Medicine Administrative Leadership Curriculum

OBJECTIVES

The objective was to research and develop a novel curriculum on administrative leadership development within the discipline of emergency medicine (EM) with the goal of establishing and implementing it through the world's EM professional organizations.

METHODS

From 2016 to 2018 an assessment of different administrative and leadership programs was performed by researching and reviewing previously outlined curricula. Using the data from this assessment, a questionnaire was developed, that was subsequently sent to members of the International Federation for Emergency Medicine's (IFEM) listserv.

RESULTS

A total of 377 people from 38 different countries participated in the survey. The majority of respondents identified themselves as EM specialists (81%, 306/377), while others identified themselves as EM resident physicians (9.5%, 36/377) and non-EM specialist physicians (4.5%, 17/377). A large majority of respondents articulated that there was a paucity of developed curricula focusing on leadership, administrative, and management principles within their institution, training program, or professional organization. Across all topic areas, fewer than 30% of polled individuals indicated that they had formal education related to individual and programmatic leadership development, change management, assessment methodology, negotiation skills, financial analysis, media relations, and health care policy. Quality improvement (QI) was the only curricular element that a majority of respondents had integrated into their clinical practice (61%). Qualitative data analysis of the narrative comments was performed with further evaluation of thematic components.

CONCLUSIONS

The results of this study further support the findings that the majority of EM providers queried do not have a longitudinal curriculum that fosters administrative and leadership development nor advocate for its importance in relation to the quality of care. Given this gap, we propose that medical education at all levels-medical schools, EM resident/specialty training programs, and professional organizations-should consider creating administrative and leadership development programs. Additionally, development of any curriculum should require a global understanding of health care systems and awareness of the unique contexts of a given location and its available resources.

Direct Observation Tools in Emergency Medicine: A Systematic Review of the Literature

OBJECTIVES

Direct observation is important for assessing the competency of medical learners. Multiple tools have been described in other fields, although the degree of emergency medicine-specific literature is unclear. This review sought to summarize the current literature on direct observation tools in the emergency department (ED) setting.

METHODS

We searched PubMed, Scopus, CINAHL, the Cochrane Central Register of Clinical Trials, the Cochrane Database of Systematic Reviews, ERIC, PsycINFO, and Google Scholar from 2012 to 2020 for publications on direct observation tools in the ED setting. Data were dual extracted into a predefined worksheet, and quality analysis was performed using the Medical Education Research Study Quality Instrument.

RESULTS

We identified 38 publications, comprising 2,977 learners. Fifteen different tools were described. The most commonly assessed tools included the Milestones (nine studies), Observed Structured Clinical Exercises (seven studies), the McMaster Modular Assessment Program (six studies), Queen's Simulation Assessment Test (five studies), and the mini-Clinical Evaluation Exercise (four studies). Most of the studies were performed in a single institution, and there were limited validity or reliability assessments reported.

CONCLUSIONS

The number of publications on direct observation tools for the ED setting has markedly increased. However, there remains a need for stronger internal and external validity data.

Turning Objective Structured Clinical Examinations into Reality

Objective structured clinical examinations assess learners "showing how" to perform complex clinical tasks. Devised as summative evaluations, these examinations with immediate feedback are useful formative evaluations to improve learner performance. This review describes how objective structured clinical examinations have been used in rheumatology education. Steps for creating an objective structured clinical examination are discussed. Validity and reproducibility are important considerations, especially for high-stakes summative objective structured clinical examinations. Consideration of the potential benefits in clinical education and their hazards are reviewed. When well-designed, formative objective structured clinical examinations have high educational value for learners and medical educators.

Understanding the Struggles to Be a Medical Provider: View Through Medical Student Essays

BACKGROUND

The clinical learning environment helps to shape the professional identity of medical students. This process begins from existing personal identity and is influenced by various factors, including clinical experiences and clinical learning environment.

OBJECTIVE

The purpose of this study was to examine medical students' reflections as a way to identify and better characterize the modern struggles that medical students face, in order to inform the development of professional identity.

METHODS

Students rotating in their emergency medicine clerkship wrote reflections on dilemmas that highlighted common struggles of becoming a doctor. Qualitative analysis was performed to determine common themes from the essays. The research team coded 173 reflections and identified themes and major domains.

RESULTS

The first domain was Patient-Provider Conflict, including challenging patient (34%), difficult communication (25%), competing priorities between patients' interest and trainees need to learn (19%), and bias (13%). The second domain was Provider-Specific Issues, such as the "gray zone," in which there is not a clear standard of practice (29%), end-of-life care (14%), emotional struggle (6%), and fear of litigation (5%). The final domain was Systems Issues, such as cost of care (12%) and role of the emergency department (6%).

CONCLUSIONS

The reflections point to a wide variety of challenges that students confront in practice that will contribute to how they develop into physicians.

Screening residents for infant lumbar puncture readiness with just-in-time simulation-based assessments

Background

Determining when to entrust trainees to perform procedures is fundamental to patient safety and competency development.

Objective

To determine whether simulation-based readiness assessments of first year residents immediately prior to their first supervised infant lumbar punctures (LPs) are associated with success.

Methods

This prospective cohort study enrolled paediatric and other first year residents who perform LPs at 35 academic hospitals from 2012 to 2014. Within a standardised LP curriculum, a validated 4-point readiness assessment of first year residents was required immediately prior to their first supervised LP. A score ≥3 was required for residents to perform the LP. The proportion of successful LPs (<1000 red blood cells on first attempt) was determined. Process measures included success on any attempt, number of attempts, analgesia usage and use of the early stylet removal technique.

Results

We analysed 726 LPs reported from 1722 residents (42%). Of the 432 who underwent readiness assessments, 174 (40%, 95% CI 36% to 45%) successfully performed their first LP. Those who were not assessed succeeded in 103/294 (35%, 95% CI 30% to 41%) LPs. Assessed participants reported more frequent direct attending supervision of the LP (diff 16%; 95% CI 8% to 22%), greater use of topical analgesia (diff 6%; 95% CI 1% to 12%) and greater use of the early stylet removal technique (diff 11%; 95% CI 4% to 19%) but no difference in number of attempts or overall procedural success.

Conclusions

Simulation-based readiness assessments performed in a point-of-care fashion were associated with several desirable behaviours but were not associated with greater clinical success with LP.

Assessing Team Leadership in Emergency Medicine: The Milestones and Beyond

BACKGROUND

Team leadership is a critical skill for emergency medicine physicians that directly affects team performance and the quality of patient care. There exists a robust body of team science research supporting team leadership conceptual models and behavioral skill sets. However, to date, this work has not been widely incorporated into health care team leadership education.

OBJECTIVE

This narrative review has 3 aims: (1) to synthesize the team science literature and to translate important concepts and models to health care team leadership; (2) to describe how team leadership is currently represented in the health care literature and in the Accreditation Council for Graduate Medical Education Milestones for emergency medicine; and (3) to propose a novel, evidence-based framework for the assessment of team leadership in emergency medicine.

METHODS

We conducted a narrative review of the team science and health care literature. We summarized our findings and identified a list of team leadership behaviors that were then used to create a framework for team leadership assessment.

RESULTS

Current health care team leadership measurement tools do not incorporate evidence-based models of leadership concepts from other established domains. The emergency medicine milestones include several team leadership behaviors as part of a larger resident evaluation program. However, they do not offer a comprehensive or cohesive representation of the team leadership construct.

CONCLUSIONS

Despite the importance of team leadership to patient care, there is no standardized approach to team leadership assessment in emergency medicine. Based on the results of our review, we propose a novel team leadership assessment framework that is supported by the team science literature.

A novel mentorship programme for residents integrating academic development, clinical teaching and graduate medical education assessment

INTRODUCTION

Mentorship fosters career development and growth. During residency training, mentorship should support clinical development along with intellectual and academic interests. Reported resident mentoring programmes do not typically include clinical components. We designed a programme that combines academic development with clinical feedback and assessment in a four-year emergency medicine residency programme.

METHODS

Incoming interns were assigned an advisor. At the conclusion of the intern year, residents actively participated in selecting a mentor for the duration of residency. The programme consisted of quarterly meetings, direct clinical observation and specific competency assessment, assistance with lecture preparation, real-time feedback on presentations, simulation coaching sessions, and discussions related to career development. Faculty participation was recognized as a valuable component of the annual review process. Residents were surveyed about the overall programme and individual components.

RESULTS

Over 88 % of the respondents said that the programme was valuable and should be continued. Senior residents most valued the quarterly meetings and presentation help and feedback. Junior residents strongly valued the clinical observation and simulation sessions.

CONCLUSIONS

A comprehensive mentorship programme integrating clinical, professional and academic development provides residents individualized feedback and coaching and is valued by trainees. Individualized assessment of clinical competencies can be conducted through such a programme.

Programmatic assessment of level 1 milestones in incoming interns

OBJECTIVES

With the Accreditation Council for Graduate Medical Education (ACGME) Next Accreditation System, emergency medicine (EM) residency programs will be required to report residents' progress through the EM milestones. The milestones include five progressively advancing skill levels, with Level 1 defining the skill set of a medical school graduate and Level 5, that of an attending physician. The ACGME stresses that multiple forms of assessment should be used to ensure capture of the multifaceted competencies. The objective of this study was to determine the feasibility and results of programmatic assessment of Level 1 milestones using multisource assessments for incoming EM interns in July.

METHODS

The study population was interns starting in 2012 and 2013. Interns' Level 1 milestone assessment was done with four distinct methods: 1) the postgraduate orientation assessment (POA) by the Graduate Medical Education Office for all incoming interns (this multistation examination covers nine of the EM milestones and includes standardized patient cases, task completion, and computer-based stations); 2) direct observation of patient encounters by core faculty using a milestones-based clinical skills competency checklist; 3) the global monthly assessment at the end of the intern orientation month that was updated to reflect the EM milestones; and 4) faculty assessment during procedural labs. These occurred during the July orientation month that included the POA, clinical shifts, didactic sessions, and procedure labs.

RESULTS

In the POA, interns were competent in 48% to 93% of the milestones assessed. Overall, competency was 70% to 80%, with low scores noted in aseptic technique (patient care Milestone 13 [PC13]) and written and verbal hand-off (interpersonal communications skills [ICS]2). In overall communication, 70% of interns demonstrated competency. In excess of 80% demonstrated competency in critical values interpretation (PC3), informed consent (PC9), pain assessment (PC11), and geriatric functional assessment (PC3). On direct observation, almost all Level 1 milestones were achieved (93% to 100%); however, only 78% of interns achieved competency in pharmacotherapy (PC5). On global monthly evaluations, all interns met Level 1 milestones.

CONCLUSIONS

A multisource assessment of EM milestones is feasible and useful to determine Level 1 milestones achievement for incoming interns. A structured assessment program, used in conjunction with more traditional forms of evaluation such as global monthly evaluations and direct observation, is useful for identifying deficits in new trainees and may be able inform the creation of early intervention programs.

Medical education research in the context of translational science

Health care struggles to transfer recent discoveries into high-quality medical care. Therefore, translational science seeks to improve the health of patients and communities by studying and promoting the translation of findings from bench research into clinical care. Similarly, medical education practice may be slow to adopt proven evidence of better learning and assessment. The Academic Emergency Medicine (AEM) consensus conference was designed to promote the dissemination of evidence-based education research and practice. We will pull from the work developed by the consensus conference as a means to create a roadmap for future medical education research using the framework of translational science.

The current state of core competency assessment in emergency medicine and a future research agenda: recommendations of the working group on assessment of observable learner performance

In 2012, the Accreditation Council for Graduate Medical Education (ACGME) introduced the Next Accreditation System (NAS) for residency program accreditation. With implementation of the NAS, residents are assessed according to a series of new emergency medicine (EM)-specific performance milestones, and the frequency of assessment reporting is increased. These changes are driving the development of new assessment tools for the NAS that can be feasibly implemented by EM residency programs and that produce valid and reliable assessment data. This article summarizes the recommendations of the writing group on assessment of observable learner performance at the 2012 Academic Emergency Medicine consensus conference on education research in EM that took place on May 9, 2012, in Chicago, Illinois. The authors define an agenda for future assessment tool research and development that was arrived at by consensus during the conference.