A simulated "Night-onCall" to assess and address the readiness-for-internship of transitioning medical students

Librarian-Led Assessment of Medical Students' Evidence-Based Medicine Competency: Facilitators and Barriers

Introduction

We must ensure, through rigorous assessment that physicians have the evidence-based medicine (EBM) skills to identify and apply the best available information to their clinical work. However, there is limited guidance on how to assess EBM competency. With a better understanding of their current role in EBM education, Health Sciences Librarians (HSLs), as experts, should be able to contribute to the assessment of medical student EBM competence. The purpose of this study is to explore the HSLs perspective on EBM assessment practices, both current state and potential future activities.

Methods

We conducted focus groups with librarians from across the United States to explore their perceptions of assessing EBM competence in medical students. Participants had been trained to be raters of EBM competence as part of a novel Objective Structured Clinical Examination (OSCE). This OSCE was just the starting point and the discussion covered topics of current EBM assessment and possibility for expanded responsibilities at their own institutions. We used a reflexive thematic analysis approach to construct themes from our conversations.

Results

We constructed eight themes in four broad categories that influence the success of librarians being able to engage in effective assessment of EBM: administrative, curricular, medical student, and librarian.

Conclusion

Our results inform medical school leadership by pointing out the modifiable factors that enable librarians to be more engaged in conducting effective assessment. They highlight the need for novel tools, like EBM OSCEs, that can address multiple barriers and create opportunities for deeper integration of librarians into assessment processes.

Viewing Readiness-for-Residency through Binoculars: Mapping Competency-Based Assessments to the AAMC's 13 Core Entrustable Professional Activities (EPAs)

Construct: The construct being assessed is readiness-for-residency of graduating medical students, as measured through two assessment frameworks. Background: Readiness-for-residency of near-graduate medical students should be but is not consistently assessed. To address this, the Association of American Medical Colleges (AAMC), in 2014, identified and described 13 core Entrustable Professional Activities (EPAs), which are tasks that all residents should be able to perform unsupervised upon entering residency. However, the AAMC did not initially provide measurement guidelines or propose standardized assessments. We designed Night-onCall (NOC), an immersive simulation for our near-graduating medical students to assess and address their readiness-for-residency, framed around tasks suggested by the AAMC's core EPAs. In adopting this EPA assessment framework, we began by building upon an established program of competency-based clinical skills assessments, repurposing competency-based checklists to measure components of the EPAs where possible, and designing new checklists, when necessary. This resulted in a blended suite of 14 checklists, which theoretically provide substantive assessment of all 13 core EPAs. In this paper, we describe the consensus-based mapping process conducted to ensure we understood the relationship between competency and EPA-based assessment lenses and could therefore report meaningful feedback on both to transitioning students in the NOC exercise. Approach: Between January-November 2017, five clinician and two non-clinician health professions educators at NYU Grossman School of Medicine conducted a rigorous consensus-based mapping process, which included each rater mapping each of the 310 NOC competency-based checklist items to lists of entrustable behaviors expected of learners according to the AAMC 13 core EPAs. Findings: All EPAs were captured to varying degrees by the 14 NOC checklists (overall Intraclass Correlation Coefficient (ICC) = 0.77). Consensus meetings resolved discrepancies and improved ICC values for three (EPA-9, EPA-10, EPA-12) of the four EPAs that initially showed poor reliability. Conclusions: Findings suggest that with some limitations (e.g., EPA-7 "form clinical questions/retrieve evidence") established competency-based assessments can be repurposed to measure readiness-for-residency through an EPA lens and both can be reported to learners and faculty.

Exploring Perceptions of Competency-Based Medical Education in Undergraduate Medical Students and Faculty: A Program Evaluation

BACKGROUND

There is limited work exploring competency-based medical education (CBME) in undergraduate medical education. We aimed to assess medical students' and faculty's perception of CBME in the undergraduate medicine setting after its implementation at our institution through a Content, Input, Process, Product (CIPP) program evaluation model.

METHODS

We explored the rationale for the transition to a CBME curriculum (Content), the changes to the curriculum and the teams involved in the transition (Input), medical students' and faculty's perception of the current CBME curriculum (Process), and benefits and challenges of implementing undergraduate CBME (Product). A cross-sectional online survey was delivered over 8-weeks in October 2021 to medical students and faculty as part of the Process and Product evaluation.

RESULTS

Medical students displayed greater optimism towards CBME, compared to faculty, in terms of its role in medical education (p<0.05). Faculty were less certain about how CBME was currently implemented (p<0.05), as well as how feedback to students should be delivered (p<0.05). Students and faculty agreed on perceived benefits to CBME implementation. Faculty time commitment to teaching and logistical concerns were reported as perceived challenges.

CONCLUSION

Education leaders must prioritize faculty engagement and continued professional development of faculty to facilitate the transition. This program evaluation identified strategies to aid the transition to CBME in the undergraduate setting.

Global perspectives on trends in health higher education

Today, changes in political and economic conditions, epidemiological and sociological developments, and the advancement of science and technology have necessitated the health and medical education systems to change. Therefore, we conducted a study entitled "Global perspectives on trends in higher education in healthcare," to improve the quality of healthcare so that it can be used as a model for predicting future events related to medical education. This futures study applied the social, technological, economical, ecological, political, values (STEEPV) model to identify and analyze the trends that affect medical education at different levels. To collect and analyze the data, a scoping review of the articles published from the year 2000 was conducted on the World Health Organization (WHO), Web of Science, Scopus, PubMed/MEDLINE, EMBASE, Science Direct, Google Scholar, EBSCO, and Cochrane databases. The review process was performed in five stages: 1- Determining the research question, 2- Identifying relevant studies, 3- Selecting the studies, 4- Charting the data, and 5- Analyzing data. The preferred reporting item for systematic review and meta-analysis (PRISMA) statement was used in the selection and screening of articles. A total of 213 articles were included in the study for qualitative synthesis. A total of 154 trends were identified using the STEEPV model at seven levels of health behaviors and patients, diseases and health problems, healthcare system, medical education system, medical education institutions, medical curricula, and teaching and learning. Considering the results of this study, it is possible to formulate proper and efficient future scenarios for the higher health education system. Also, it will be helpful for medical education policymaking.

Exploring the Impact of Pre-course High-Fidelity Simulation on Professional Socialization of Medical Students in Emergency Medicine Internship Rotation—A Qualitative Approach

Background

Medical students in block clerkships constantly adapt to new environments and learn to interact with new people as they rotate between specialties. This frequent change potentially limited interns' opportunities for participation in real clinical practice. The aims of this study were to explore interns' conceptualization of their learning opportunities and experiences in the workplace during an emergency medicine (EM) block internship. In addition, the study also explored how participating in the pre-rotation high-fidelity simulation (HFS) orientation influenced interns' perception of their transition, participation and learning experiences in the real EM setting.

Methods

We implemented a newly developed pre-EM rotation orientation curriculum for interns. This orientation took place on the first day of the 2-week EM internship rotation. Two focus group discussions were held after each simulation training, one immediately after simulation to understand the students' perception and the educational impact of this activity, the other at the end of EM rotation to explore and compare their roles and perception in both simulation activity and the real clinical practice. A total of 151 seventh-year medical students enrolled in the pre-course HFS and post-hoc focus group discussions between 2017 and 2019. We applied thematic analysis to systemically identify, examine, and construct themes.

Results

Four major themes were constructed from the data; 1. Challenges in finding authentic learning experiences within the context of emergency medicine; 2. Effectiveness of the pre-course HFS 3. Limitations of EM internship rotation curriculum and pre-course simulation. 4. Suggestions for EM block-internship curriculum reforms. Our study's key findings indicate that pre-rotation orientation HFS activity, which offered a psychologically safe space for students to explore facets of EM and gain a contextualized understanding of the emergency work culture and environment, was essential for enhancing students' ability to identify and maximize practice affordances in real workplace.

Conclusion

Simulation, facilitates interns' negotiation of legitimate peripheral participation opportunities as they transition into the EM community of practice during their block internship rotation; putting students at the center of the learning process.

A Preliminary Evaluation of Students' Learning and Performance Outcomes in an Accelerated 3-Year MD Pathway Program

BACKGROUND

Little outcome data exist on 3-year MD (3YMD) programs to guide residency program directors (PDs) in deciding whether to select these graduates for their programs.

OBJECTIVE

To compare performance outcomes of 3YMD and 4-year MD (4YMD) students at New York University Grossman School of Medicine.

METHODS

In 2020, using the Kirkpatrick 4-level evaluation model, outcomes from 3 graduating cohorts of 3YMD students (2016-2018) were compared with the 4YMD counterparts.

RESULTS

Descriptive statistics compared outcomes among consented student cohorts: 92% (49 of 53) 3YMD, 87% (399 of 459) 4YMD-G, and 84% (367 of 437) 4YMD-S. Student survey response rates were 93% (14 of 15), 74% (14 of 19), and 89% (17 of 19) from 2016 to 2018. PDs' response rates were 58% (31 of 53, 3YMD) and 51% (225 of 441, 4YMD). Besides age, 3YMD and 4YMD cohorts did not differ significantly in admissions variables. Other than small statistically significant differences in the medicine shelf examination (3YMD mean 74.67, SD 7.81 vs 4YMD-G mean 78.18, SD 7.60; t test=3.02; P=.003) and USMLE Step 1 (3YMD mean 235.13, SD 17.61 vs 4YMD-S mean 241.70, SD 15.92; t test=2.644; P=.009 and vs 4YMD-G mean 242.39, SD 15.65; t test=2.97; P=.003) and Step 2 CK scores (3YMD mean 242.57, SD 15.58 vs 4YMD-S mean 248.55, SD 15.33; t test=2.55; P=.01 and vs 4YMD-G mean 247.83, SD 15.38; t test=2.97; P=.03), other metrics and overall intern ratings did not differ by pathway.

CONCLUSIONS

Exploratory findings from a single institution suggest that 3YMD students performed similarly to 4YMD students in medical school and the first year of residency.

Questioning medical competence: Should the Covid-19 crisis affect the goals of medical education?

The COVID-19 pandemic has disrupted many societal institutions, including health care and education. Although the pandemic's impact was initially assumed to be temporary, there is growing conviction that medical education might change more permanently. The International Competency-based Medical Education (ICBME) collaborators, scholars devoted to improving physician training, deliberated how the pandemic raises questions about medical competence. We formulated 12 broad-reaching issues for discussion, grouped into micro-, meso-, and macro-level questions. At the individual micro level, we ask questions about adaptability, coping with uncertainty, and the value and limitations of clinical courage. At the institutional meso level, we question whether curricula could include more than core entrustable professional activities (EPAs) and focus on individualized, dynamic, and adaptable portfolios of EPAs that, at any moment, reflect current competence and preparedness for disasters. At the regulatory and societal macro level, should conditions for licensing be reconsidered? Should rules of liability be adapted to match the need for rapid redeployment? We do not propose a blueprint for the future of medical training but rather aim to provoke discussions needed to build a workforce that is competent to cope with future health care crises.

Handoffs and Nurse Calls: Overnight Call Simulation for Fourth-Year Medical Students

INTRODUCTION

Interns must be able to give and receive handoffs and use handoff information to respond to calls from nursing staff regarding patient concerns. Medical students may not receive adequate instruction in these tasks and often feel unprepared in this aspect of transitioning to residency. This program simulated an overnight call experience for fourth-year medical students emphasizing handoffs, nurse calls, and medical emergency response.

METHODS

The program utilized a combination of traditional didactics and simulated handoffs, nurse calls, and patient scenarios to allow groups of fourth-year medical students to independently manage a simulated overnight call. The program was designed for students as part of a larger Transition to Residency capstone course.

RESULTS

We ran four sessions over 3 years, with a total of 105 medical student participants. All students reported increased confidence or comfort in their ability to manage handoffs and respond to nurse calls. Students reported that the sessions were helpful and realistic.

DISCUSSION

This program provided fourth-year medical students with a realistic and useful opportunity to simulate handoffs and response to nurse calls, which increased their confidence and comfort. Minor changes were made between iterations of the course with continued positive feedback from medical students. The course is generalizable and can be adapted to the needs and resources of different institutions.

"Doing it Right" Overnight: a Multi-perspective Qualitative Study Exploring Senior Medical Resident Overnight Call

BACKGROUND

Competency-based medical education (CBME) requires the development of workplace-based assessment tools that are grounded in authentic clinical work. Developing such tools, however, requires a deep understanding of the underlying facets of the competencies being assessed. Gaining this understanding remains challenging in contexts where performance is not readily visible to supervisors such as the senior medical resident (SMR) on-call role in internal medicine.

OBJECTIVE

This study draws on the perspectives of healthcare professionals with whom the SMR interacts with overnight to generate insights into the different components of on-call SMR practice and the range of ways SMRs effectively and less effectively enact these.

APPROACH

We used a constructivist grounded theory (CGT) approach to examine variation in how on-call SMRs carry out their role overnight.

PARTICIPANTS

Six medical students, five junior residents, five internal medicine attending physicians, five emergency physicians, and three emergency nurses conducted observations of their on-call interactions with SMRs. Participants were then interviewed and asked to elaborate on their observations as well as provide comparative reflections on the practices of past SMRs they worked with.

KEY RESULTS

Strong collaboration and organizational skills were identified as critical components to effectively being the on-call SMR. Perceived weaker SMRs, while potentially also having issues with clinical skills, stood out more when they could not effectively manage the realities of collaboration in a busy workplace.

CONCLUSION

What consistently differentiated a perceived effective SMR from a less effective SMR was someone who was equipped to manage the realities of interprofessional collaboration in a busy workplace. Our study invites medical educators to consider what residents, particularly those in more complex roles, need to receive feedback on to support their development as physicians. It is our intention that the findings be used to inform the ways programs approach teaching, assessment, and the provision of feedback.

Exploring why medical students still feel underprepared for clinical practice: a qualitative analysis of an authentic on-call simulation

BACKGROUND

Current research shows that many UK medical graduates continue to feel underprepared to work as a junior doctor. Most research in this field has focused on new graduates and employed the use of retrospective self-rating questionnaires. There remains a lack of detailed understanding of the challenges encountered in preparing for clinical practice, specifically those faced by medical students, where relevant educational interventions could have a significant impact. Through use of a novel on-call simulation, we set out to determine factors affecting perceived preparation for practice in final year medical students and identify ways in which we may better support them throughout their undergraduate training.

METHODS

30 final year medical students from Imperial College London participated in a 90-minute simulation on hospital wards, developed to recreate a realistic on-call experience of a newly qualified doctor. Students partook in pairs, each observed by a qualified doctor taking field notes on their decisions and actions. A 60-minute semi-structured debrief between observer and student pair was audio-recorded for analysis. Field notes and students' clinical documentation were used to explore any challenges encountered. Debrief transcripts were thematically analysed through a general inductive approach. Cognitive Load Theory (CLT) was used as a lens through which to finalise the evolving themes.

RESULTS

Six key themes emerged from the on-call simulation debriefs: information overload, the reality gap, making use of existing knowledge, negative feelings and emotions, unfamiliar surroundings, and learning 'on the job'.

CONCLUSIONS

The combination of high fidelity on-call simulation, close observation and personalised debrief offers a novel insight into the difficulties faced by undergraduates in their preparation for work as a junior doctor. In using CLT to conceptualise the data, we can begin to understand how cognitive load may be optimised within this context and, in doing so, we highlight ways in which undergraduate curricula may be adapted to better support students in their preparation for clinical practice. Recommendations are centred around enhancing the expertise of the learner through 'whole task' training approaches and integrated learning, as well as navigating negative emotions and supporting lifelong 'learning while working'.

Entrustment Decision Making: Extending Miller's Pyramid

The iconic Miller's pyramid, proposed in 1989, characterizes 4 levels of assessment in medical education ("knows," "knows how," "shows how," "does"). The frame work has created a worldwide awareness of the need to have different assessment approaches for different expected outcomes of education and training. At the time, Miller stressed the innovative use of simulation techniques, geared at the third level ("shows how"); however, the "does" level, assessment in the workplace, remained a largely uncharted area. In the 30 years since Miller's conference address and seminal paper, much attention has been devoted to procedures and instrument development for workplace-based assessment. With the rise of competency-based medical education (CBME), the need for approaches to determine the competence of learners in the clinical workplace has intensified. The proposal to use entrustable professional activities as a framework of assessment and the related entrustment decision making for clinical responsibilities at designated levels of supervision of learners (e.g., direct, indirect, and no supervision) has become a recent critical innovation of CBME at the "does" level. Analysis of the entrustment concept reveals that trust in a learner to work without assistance or supervision encompasses more than the observation of "doing" in practice (the "does" level). It implies the readiness of educators to accept the inherent risks involved in health care tasks and the judgment that the learner has enough experience to act appropriately when facing unexpected challenges. Earning this qualification requires qualities beyond observed proficiency, which led the authors to propose adding the level "trusted" to the apex of Miller's pyramid.

Entrustable professional activities and facets of competence in a simulated workplace-based assessment for advanced medical students

Background: Competence-based assessment formats in medical education usually focus on individual facets of competence (FOCs). The concept of 'Entrustable Professional Activities' (EPAs) encompasses supervisors' decisions on which level of supervision a trainee requires to perform a professional activity including several FOCs. How the different FOCs as perceived by clinician raters contribute to entrustment decisions is yet unclear.Objective: How do FOC perceptions relate to entrustment-decisions?Methods: Sixty-seven advanced medical students participated in an assessment simulating the first day of a resident physician. Participants were rated by supervisors for seven FOCs and twelve EPAs.Results: There was a positive correlation between FOC and EPA scores. Each EPA displayed a different correlation pattern with FOC ratings.Discussion: For most EPAs high levels of entrustment were associated with high ratings for selected FOCs. The results are in alignment with the assumption that each EPA encompasses a different set of FOCs.Conclusions: In our simulated workplace-based assessment, entrustment decisions for EPAs reflect the FOCs observed in a trainee. Thus, assessment of FOCs alongside with EPA ratings could add to the understanding of factors contributing to entrustment decisions.

Understanding medical student evidence-based medicine information seeking in an authentic clinical simulation

Objective

Evidence-based medicine practices of medical students in clinical scenarios are not well understood. Optimal foraging theory (OFT) is one framework that could be useful in breaking apart information-seeking patterns to determine effectiveness and efficiency of different methods of information seeking. The aims of this study were to use OFT to determine the number and type of resources used in information seeking when medical students answer a clinical question, to describe common information-seeking patterns, and identify patterns associated with higher quality answers to a clinical question.

Methods

Medical students were observed via screen recordings while they sought evidence related to a clinical question and provided a written response for what they would do for that patient based on the evidence that they found.

Results

Half (51%) of study participants used only 1 source before answering the clinical question. While the participants were able to successfully and efficiently navigate point-of-care tools and search engines, searching PubMed was not favored, with only half (48%) of PubMed searches being successful. There were no associations between information-seeking patterns and the quality of answers to the clinical question.

Conclusion

Clinically experienced medical students most frequently relied on point-of-care tools alone or in combination with PubMed to answer a clinical question. OFT can be used as a framework to understand the information-seeking practices of medical students in clinical scenarios. This has implications for both teaching and assessment of evidence-based medicine in medical students.

Monitoring communication skills progress of medical students: Establishing a baseline has value, predicting the future is difficult

OBJECTIVE

To provide evidence for the validity of an Introductory Clinical Experience (ICE) that was implemented as a baseline assessment of medical students' clinical communication skills to support progression of skills over time.

METHODS

In this longitudinal study of communication skills, medical students completed the ICE, then a Practice of Medicine (POM) Objective Structured Clinical Exam 8 months later, and the Comprehensive Clinical Skills Exam (CCSE) 25 months later. At each experience, trained Standardized Patients assessed students, using the same behaviorally anchored checklist in 3 domains: Information Gathering, Relationship Development, and Patient Education and Counseling (PEC) with good internal reliability (.70-.87). Skills development patterns were described. ICE as a predictor of later performance was explored. Students' perspectives were elicited.

RESULTS

140 (80%) medical students consented to include their data in this study. Overall communication scores increased over time (eta² = .17, medium effect) mostly attributable to increase in PEC skills (eta² = .48, large effect), in 4 patterns. ICE and POM scores predicted future communication skills. Most students recognized the educational value of ICE.

CONCLUSION

Entering medical students' clinical communication skills increase over time on average and may predict future performance.

PRACTICE IMPLICATIONS

Implementing an ICE is likely a valid strategy for monitoring progress and facilitating communication skills development.