Evaluation and treatment of students with difficulties passing the Step examinations

Using Self-Regulated Learning Microanalysis to Examine Regulatory Processes in Clerkship Students Engaged in Practice Questions

Introduction

Self-regulated learning is a cyclical process of forethought, performance, and self-reflection that has been used as an assessment tool in medical education. No prior studies have evaluated SRL processes for answering multiple-choice questions (MCQs) and most evaluated one or two iterations of a non-MCQ task. SRL assessment during MCQs may elucidate reasons why learners are successful or not on these questions that are encountered repeatedly during medical education.

Methods

Internal medicine clerkship students at three institutions participated in a SRL microanalytic protocol that targeted strategic planning, metacognitive monitoring, causal attributions, and adaptive inferences across seven MCQs. Responses were transcribed and coded according to previously published methods for microanalytic protocols.

Results

Forty-four students participated. In the forethought phase, students commonly endorsed prioritizing relevant features as their diagnostic strategy (n = 20, 45%) but few mentioned higher-order diagnostic reasoning processes such as integrating clinical information (n = 5, 11%) or comparing/contrasting diagnoses (n = 0, 0%). However, in the performance phase, students' metacognitive processes included high frequencies of integration (n = 38, 86%) and comparing/contrasting (n = 24, 55%). In the self-reflection phase, 93% (n = 41) of students faulted their management reasoning and 84% (n = 37) made negative references to their abilities. Less than 10% (n = 4) of students indicated that they would adapt their diagnostic reasoning process for these questions.

Discussion

This study describes in detail student self-regulatory processes during MCQs. We found that students engaged in higher-order diagnostic reasoning processes but were not explicit about it and seldom reflected critically on these processes after selecting an incorrect answer. Self-reflections focused almost exclusively on management reasoning and negative references to abilities which may decrease self-efficacy. Encouraging students to identify and evaluate diagnostic reasoning processes and make attributions to controllable factors may improve performance.

Factors Leading to Successful Performance on U.S. National Licensure Exams for Medical Students: A Scoping Review

PURPOSE

To synthesize the evidence of the factors leading to successful performance on knowledge-based national licensure exams (NLEs) for medical students.

METHOD

The authors conducted a scoping review to summarize the peer-reviewed empiric literature that used United States Medical Licensing Examination (USMLE) Step 1 or Step 2 Clinical Knowledge or Comprehensive Osteopathic Medical Licensing Examination (COMLEX) Level 1 or Level 2 Cognitive Evaluation scores as outcomes. The authors searched PubMed and Scopus without date restrictions through April 30, 2021. Two reviewers independently screened and selected studies for inclusion. Data were summarized narratively and with descriptive statistics.

RESULTS

The authors screened 1,185 unique citations and included 233 full-text studies in their review. Of these, 201 (86%) were studies of USMLE exams, 31 (13%) were studies of COMLEX exams, and 1 (0.4%) reported on both. The authors classified 29 studies (12%) as informing NLE preparation, 163 (70%) as attempting to identify predictive variables, and 76 (33%) as using NLE scores for program evaluation. Preparation studies found that the number of practice test items, practice exam scores, and less time in dedicated preparation correlated with higher NLE scores. Use of other commercial resources or study strategies was not consistently associated with higher scores. Predictive studies found the strongest relationships between individuals' performance on past assessments and their NLE scores.

CONCLUSIONS

The factors leading to successful performance on knowledge-based NLEs align with well-known principles from the cognitive sciences. Learners build on existing foundations of knowledge (reflected in their prior academic performance) and are likely to learn more efficiently with testing and spaced learning over time. While commercial test preparation resources are ubiquitous, there is no evidence that a single resource gives students a competitive advantage on NLEs. Developing habits of regular and continuous learning is necessary for clinical practice and successful NLE performance.

One size doesn’t fit all: framework and resources for individualized interventions for struggling medical students

When helping struggling medical students, a “one size fits all” approach is often ineffective, as many different factors affect academic success. In our experience, these factors may overlap or be distinct for each student, and thus require individualized interventions based on student needs. We recommend an individualized model of academic support includes an intake interview, assessment, individualized interventions, and follow up to assess progress. This paper provides a recommended framework of resources for medical school faculty and staff who work with struggling students. Different issues that lead students to struggle are categorized in the main domains of Bloom’s taxonomy: cognitive, affective, and psychomotor. A discussion of the impact of the factors in each domain includes detailed tables with corresponding manifestations, assessments, and support strategies literature for each issue. An application of a proposed framework is presented on a case example. Individualized approaches to improving medical student success are presented to address the complex and interrelated areas of academic success, wellbeing, quality of life, career potential, and satisfaction with medical school.

Principles of Remediation for the Struggling Neurosurgery Resident

BACKGROUND

When resident physicians fail to demonstrate appropriate milestone competencies early in their neurologic surgery residency, a plan of remediation is necessary.

METHODS

Once any psychologic, physical, or behavioral causes of identified knowledge or psychomotor deficiencies have been identified and addressed, the next step is to develop a plan to close these gaps. Specific areas that are assessed for deficits include medical knowledge, clinical reasoning and judgment, clinical skills, time management and organization, interpersonal skills, communication, and professionalism. Specific learning goals and objectives, as well as teaching and learning methods, pertain to the unique areas of deficit, and all of these must be considered with the goal of developing a resident-specific remediation plan.

RESULTS

A plan for assessment of the remediation process is described, including an evaluation of what constitutes individual resident remediation success.

CONCLUSIONS

Finally, a discussion of the prior resident remediation studies across many disciplines is made.

The Evolution of Resident Remedial Teaching at One Institution

Residency program directors and teaching faculty invest an enormous amount of time, energy, and resources in providing underperforming at-risk learners with remedial teaching. A remediation program was created and centralized at the University of Colorado School of Medicine in 2006 and 2012, respectively, that consolidated expertise in and resources for learner assessment and individualized teaching for struggling learners, particularly those placed on probation or receiving letters of warning (called focused review letters) from their residency programs. Since the implementation of the program, the authors have observed a decrease in the number of residents being placed on probation, and, of those on probation, more are graduating and obtaining board certification. In this Article, the authors aim to describe the development and outcomes of the program and to explore possible reasons for the improved outcomes.

Interventions for undergraduate and postgraduate medical learners with academic difficulties: A BEME systematic review: BEME Guide No. 56

Background: Clinical teachers often struggle to report unsatisfactory trainee performance, partly because of a lack of evidence-based remediation options. Objectives: To identify interventions for undergraduate (UG) and postgraduate (PG) medical learners experiencing academic difficulties, link them to a theory-based framework and provide literature-based recommendations around their use. Methods: This systematic review searched MEDLINE, CINAHL, EMBASE, ERIC, Education Source and PsycINFO (1990-2016) combining these concepts: medical education, professional competence/difficulty and educational support. Original research/innovation reports describing intervention(s) for UG/PG medical learners with academic difficulties were included. Data extraction employed Michie's Behavior Change Techniques (BCT) Taxonomy and program evaluation models from Stufflebeam and Kirkpatrick. Quality appraisal used the Mixed Methods Appraisal Tool (MMAT). The authors synthesized extracted evidence by adapting the GRADE approach to formulate recommendations. Results: Sixty-eight articles met the inclusion criteria, most commonly addressing knowledge (66.2%), skills (53.9%) and attitudinal problems (26.2%), or learner personal issues (41.5%). The most common BCTs were Shaping knowledge, Feedback/monitoring, and Repetition/substitution. Quality appraisal was variable (MMAT 0-100%). A thematic content analysis identified 109 interventions (UG: n = 84, PG: n = 58), providing 24 strong, 48 moderate, 26 weak and 11 very weak recommendations. Conclusion: This review provides a repertoire of literature-based interventions for teaching/learning, faculty development, and research purposes.

Struggle and failure on clinical placement: a critical narrative review

BACKGROUND

Clinical placements are crucial to the development of skills and competencies in speech-language pathology (SLP) education and, more generally, a requirement of all health professional training programmes. Literature from medical education provides a context for understanding how the environment can be vital to all students' learning. Given the increasing costs of education and demands on health services, students who struggle or fail on clinical placement place an additional burden on educators. Therefore, if more is known or understood about these students and their experience in relation to the clinical learning environment, appropriate strategies and support can be provided to reduce the burden. However, this literature does not specifically explore marginal or failing students and their experience.

AIMS

To review existing research that has explored failing and struggling health professional students undertaking clinical placements and, in particular, SLP students.

METHODS & PROCEDURES

A critical narrative review was undertaken. Three electronic databases, ProQuest, CINAHL and OVID (Medline 1948-), were searched for papers exploring marginal and failing students in clinical placement contexts across all health professions, published between 1988 and 2017. Data were extracted and examined to determine the breadth of the existing research, and publications were critically appraised and major research themes identified.

MAIN CONTRIBUTION

Sixty-nine papers were included in the review. The majority came from medicine and nursing in the United States and United Kingdom, with other allied health disciplines less well represented. The review identified key themes with the majority of papers focused on identification of at risk students and support and remediation. The review also highlighted the absence of literature relating to the student voice and in the allied health professions.

CONCLUSIONS & IMPLICATIONS

This review highlighted the limited research related to failing/struggling student learning in clinical contexts, and only a handful of papers have specifically addressed marginal or failing students in allied health professions. The complexity of interrelated factors in this field has been highlighted in this review. Further research needs to include the student's voice to develop greater understanding and insights of struggle and failure in clinical contexts.

Study Skills and Test Taking Strategies for Coaching Medical Learners Based on Identified Areas of Struggle

INTRODUCTION

Learners in high-performing contexts such as medical school and residency are presumed to have appropriate study skills to be successful. However, for those learners in academic difficulty who are identified as having weak study skills and poor test taking skills, faculty need tools to use to lead these struggling learners to academic success. In coaching learners on study skills, we frequently found that the study skills that helped them get into medical school or residency were no longer sufficient to make them successful in their new program. Given that there are multiple study strategies available, faculty coaches need mechanisms to first tease out which skills are the issue and then provide targeted strategies specific to each learner.

METHODS

In meeting with a faculty coach, learners are briefly interviewed, complete a self-assessment to explore all possible root weaknesses in their study skills, and then read strategic solutions and review with faculty how they may be implemented. This tool has been offered to 52 students, 76 residents, and 20 fellows and faculty between 2010 and 2015.

RESULTS

One hundred forty-eight individuals participated in this innovation, with more than 91% of all individuals going on to pass the exam that they had either failed or, in the case of the in-training exam, scored below the 30th percentile on.

CONCLUSION

A self-assessment tool is key to individualized insight and action plans for improving study skills. Implementation must be supported with concurrent in-person coaching.

Investigating the Performance of Second Language Medical Students on Lengthy Clinical Vignettes

This study evaluated the extent to which medical students with limited English-language experience are differentially impacted by the additional reading load of test items consisting of long clinical vignettes. Participants included 25,012 examinees who completed Step 2 of the U.S. Medical Licensing Examination®. Test items were categorized into five levels based on the number of words per item, and examinee scores at each level were evaluated as a function of English-language experience (English as a second language [ESL] status and scores on a test of English-speaking proficiency). The longest items were more difficult than the shortest items across all examinee groups, and examinees with more English-language experience scored higher than those with less experience across all five levels of word count. The effect of primary interest-the interaction of word count with English-language experience-was statistically significant, indicating that score declines for longer items were larger for examinees with less English-language experience; however, the magnitude of this interaction effect was barely detectable (η² = .0004, p < .001). Additional analyses supported the conclusion that the differential effect for examinees with less English-language experience was small but worthy of continued monitoring.

Alone in the Crowd: I Failed the ABGC Certification Exam

The American Board of Genetic Counseling (ABGC) certification examination (often referred to as "the board exam") has become a milestone within the field of genetic counseling. For many, it is the final standardized test taken and indicates the examinee has met "the standards of minimal competence to practice as a genetic counselor" (Bulletin 2015). Although voluntary, certification is strongly encouraged, and in some employment situations, required. Although recent statistics indicate the majority of those who take the test pass, each year there are those who sit for the test unsuccessfully. Despite this fact, exam failure and tools for dealing with this experience are not often broached in the literature. This essay recalls my experiences with a failed exam attempt and the subsequent emotional turmoil. It also aims to start the conversation regarding the difficulty of coping with the "secret" shame of public, professional failure, and suggests there is room for further discussion and resource development in this area.

Correlation of the National Emergency Medicine M4 Clerkship Examination with USMLE Examination Performance

Introduction

Assessment of medical students’ knowledge in clinical settings is complex yet essential to the learning process. Clinical clerkships use various types of written examinations to objectively test medical knowledge within a given discipline. Within emergency medicine (EM), a new national standardized exam was developed to test medical knowledge in this specialty. Evaluation of the psychometric properties of a new examination is an important issue to address during test development and use. Studies have shown that student performance on selected standardized exams will reveal students’ strengths and/or weaknesses, so that effective remedial efforts can be implemented. Our study sought to address these issues by examining the association of scores on the new EM national exam with other standardized exam scores.

Methods

From August 2011 to April 2013, average National EM M4 examination scores of fourth-year medical students taken at the end of a required EM clerkship were compiled. We examined the correlation of the National EM M4 examination with the scores of initial attempts of the United States Medical Licensing Exam (USMLE) Step 1 and Step 2 Clinical Knowledge (CK) examinations. Correlation coefficients and 95% confidence intervals of correlation coefficients are reported. We also examined the association between the national EM M4 examination score, final grades for the EM rotation, and USMLE Step 1 and Step 2 CK scores.

Results

133 students were included in the study and achieved a mean score of 79.5 SD 8.0 on the National EM M4 exam compared to a national mean of 79.7 SD 3.89. The mean USMLE Step 1 score was 226.8 SD 19.3. The mean USMLE Step 2 CK score was 238.5 SD 18.9. National EM M4 examination scores showed moderate correlation with both USMLE Step 1 (mean score=226.8; correlation coefficient=0.50; 95% CI [0.28–0.67]) and USMLE Step 2 CK (mean score=238.5; correlation coefficient=0.47; 95% CI [0.25–0.65]). Students scoring below the median on the national EM M4 exam also scored well below their colleagues on USMLE exams.

Conclusion

The moderate correlation of the national EM M4 examination and USMLE Step 1 and Step 2 CK scores provides support for the utilization of the CDEM National EM M4 examination as an effective means of assessing medical knowledge for fourth-year medical students. Identification of students scoring lower on standardized exams allows for effective remedial efforts to be undertaken throughout the medical education process.

Medical students' personal experience of high-stakes failure: case studies using interpretative phenomenological analysis

BACKGROUND

Failing a high-stakes assessment at medical school is a major event for those who go through the experience. Students who fail at medical school may be more likely to struggle in professional practice, therefore helping individuals overcome problems and respond appropriately is important. There is little understanding about what factors influence how individuals experience failure or make sense of the failing experience in remediation. The aim of this study was to investigate the complexity surrounding the failure experience from the student's perspective using interpretative phenomenological analysis (IPA).

METHODS

The accounts of three medical students who had failed final re-sit exams, were subjected to in-depth analysis using IPA methodology. IPA was used to analyse each transcript case-by-case allowing the researcher to make sense of the participant's subjective world. The analysis process allowed the complexity surrounding the failure to be highlighted, alongside a narrative describing how students made sense of the experience.

RESULTS

The circumstances surrounding students as they approached assessment and experienced failure at finals were a complex interaction between academic problems, personal problems (specifically finance and relationships), strained relationships with friends, family or faculty, and various mental health problems. Each student experienced multi-dimensional issues, each with their own individual combination of problems, but experienced remediation as a one-dimensional intervention with focus only on improving performance in written exams. What these students needed to be included was help with clinical skills, plus social and emotional support. Fear of termination of the their course was a barrier to open communication with staff.

CONCLUSIONS

These students' experience of failure was complex. The experience of remediation is influenced by the way in which students make sense of failing. Generic remediation programmes may fail to meet the needs of students for whom personal, social and mental health issues are a part of the picture.

Academic and professional career outcomes of medical school graduates who failed USMLE Step 1 on the first attempt

This study sought to determine the academic and professional outcomes of medical school graduates who failed the United States Licensing Examination Step 1 on the first attempt. This retrospective cohort study was based on pooled data from 2,003 graduates of six Midwestern medical schools in the classes of 1997-2002. Demographic, academic, and career characteristics of graduates who failed Step 1 on the first attempt were compared to graduates who initially passed. Fifty medical school graduates (2.5 %) initially failed Step 1. Compared to graduates who initially passed Step 1, a higher proportion of graduates who initially failed Step 1 became primary care physicians (26/49 [53 %] vs. 766/1,870 [40.9 %]), were more likely at graduation to report intent to practice in underserved areas (28/50 [56 %] vs. 419/1,939 [ 21.6 %]), and more likely to take 5 or more years to graduate (11/50 [22.0 %] vs. 79/1,953 [4.0 %]). The relative risk of first attempt Step 1 failure for medical school graduates was 13.4 for African Americans, 7.4 for Latinos, 3.6 for matriculants >22 years of age, 3.2 for women, and 2.3 for first generation college graduates. The relative risk of not being specialty board certified for those graduates who initially failed Step 1 was 2.2. Our observations regarding characteristics of graduates in our study cohort who initially failed Step 1 can inform efforts by medical schools to identify and assist students who are at particular risk of failing Step 1.

The remediation challenge: theoretical and methodological insights from a systematic review

OBJECTIVES

Remediation is usually offered to medical students and doctors in training who underperform on written or clinical examinations. However, there is uncertainty and conflicting evidence about the effectiveness of remediation. The aim of this systematic review was to synthesise the available evidence to clarify how and why remediation interventions may have worked in order to progress knowledge on this topic.

METHODS

The MEDLINE, EMBASE, CINAHL (Cumulative Index to Nursing and Allied Health Literature), ERIC (Educational Resources Information Centre), Web of Science and Scopus databases were searched for papers published from 1984 to April 2012, using the search terms 'remedial teaching', 'education', 'medical', 'undergraduate'/or 'clinical clerkship'/or 'internship and residency', 'at risk' and 'struggling'. Only studies that included an intervention, then provided retest data, and reported at least one outcome measure of satisfaction, knowledge, skills or effects on patients were eligible for inclusion. Studies of practising doctors were excluded. Data were abstracted independently in duplicate for all items. Coding differences were resolved through discussion.

RESULTS

Thirty-one of 2113 studies met the review criteria. Most studies were published after 2000 (n=24, of which 12 were published from 2009 onwards), targeted medical students (n=22) and were designed to improve performance on an immediately subsequent examination (n=22). Control or comparison groups, conceptual frameworks, adequate sample sizes and long-term follow-up measures were rare. In studies that included long-term follow-up, improvements were not sustained. Intervention designs tended to be highly complex, but their design or reporting did not enable the identification of the active components of the remedial process.

CONCLUSIONS

Most remediation interventions in medical education focus on improving performance to pass a re-sit of an examination or assessment and provide no insight into what types of extra support work, or how much extra teaching is critical, in terms of developing learning. More recent studies are generally of better quality. Rigorous approaches to developing and evaluating remediation interventions are required.

A national cohort study of U.S. medical school students who initially failed Step 1 of the United States Medical Licensing Examination

PURPOSE

To describe educational outcomes for a national cohort of U.S. medical students who initially failed Step 1 of the United States Medical Licensing Examination.

METHOD

The authors analyzed deidentified, individualized records for the 1993-2000 cohort of U.S. medical school matriculants who both initially failed Step l and were no longer in medical school as of March 2, 2009, using multivariable logistic regression to identify factors associated with attempting, and initially passing, Step 2 Clinical Knowledge (CK).

RESULTS

Of 6,594 students who failed Step l, 5,985 (90.8%) took Step 2CK. Women and Asian/Pacific Islander students were more likely to attempt Step 2CK; more recent matriculants and students with lower failing Step 1 scores were less likely. Of the 5,985 students who attempted Step 2CK, 4,168 (69.6%) initially passed. Women, students with higher Medical College Admission Test scores, and more recent matriculants were more likely to pass Step 2CK; Asian/Pacific Islander students, underrepresented minority students, older students, and students with lower failing Step 1 scores were less likely. Ninety percent of students in the study sample (5,952/6,594) ultimately graduated from medical school, including 99.5% (4,148/4,168) of those who initially passed, 96.7% (1,757/1,817) of those who initially failed, and 7.7% (47/609) of those who never attempted Step 2CK.

CONCLUSIONS

The authors identified variables associated with educational outcomes among students who failed Step l. These findings can inform medical schools' efforts to develop tailored interventions to maximize the likelihood that students will take Step 2CK and pass on the first attempt.