Is short-term remediation after OSCE failure sustained? A retrospective analysis of the longitudinal attainment of underperforming students in OSCE assessments

Interventions for undergraduate and postgraduate medical learners with academic difficulties: A BEME systematic review update: BEME Guide No. 85

BACKGROUND

Clinical teachers often struggle to record trainee underperformance due to lacking evidence-based remediation options.

OBJECTIVES

To provide updated evidence-based recommendations for addressing academic difficulties among undergraduate and postgraduate medical learners.

METHODS

A systematic review searched databases including MEDLINE, CINAHL, EMBASE, ERIC, Education Source, and PsycINFO (2016-2021), replicating the original Best Evidence Medical Education 56 review strategy. Original research/innovation reports describing intervention(s) for medical learners with academic difficulties were included. Data extraction used Michie's Behaviour Change Techniques (BCT) Taxonomy and program evaluation models from Stufflebeam and Kirkpatrick. Quality appraised used the Mixed Methods Appraisal Tool (MMAT). Authors synthesized extracted evidence by adapting GRADE approach to formulate recommendations.

RESULTS

Eighteen articles met the inclusion criteria, primarily addressing knowledge (66.7%), skills (66.7%), attitudinal problems (50%) and learner's personal challenges (27.8%). Feedback and monitoring was the most frequently employed BCT. Study quality varied (MMAT 0-100%). We identified nineteen interventions (UG: n = 9, PG: n = 12), introducing twelve new thematic content. Newly thematic content addressed contemporary learning challenges such as academic procrastination, and use of technology-enhanced learning resources. Combined with previous interventions, the review offers a total dataset of 121 interventions.

CONCLUSION

This review offers additional evidence-based interventions for learners with academic difficulties, supporting teaching, learning, faculty development, and research efforts.

Pre- and post-examination reflections of first-year medical students in an integrated medical anatomy course

Due to the rigor and pace of undergraduate medical anatomy courses, it is not uncommon for students to struggle and fail initially. However, repetition of coursework places an additional burden on the student, instructor, and institution. The purpose of this study was to compare the exam preparation strategies of repeating and non-repeating students to identify areas where struggling students can be supported prior to course failure. As part of their integrated anatomy course, first-year medical students at Indiana University completed a metacognitive Practice-Based Learning and Improvement (PBLI) assignment prior to and after their first exam. In the PBLIs, students were asked to reflect on their exam preparation strategies, confidence, and satisfaction, as well as their predicted and actual exam performance. PBLI responses from non-repeating and repeating students were then analyzed quantitatively and qualitatively. A total of 1802 medical students were included in this study, including 1751 non-repeating and 51 repeating students. Based on their PBLI responses, non-repeating students were appropriately confident, somewhat satisfied, and more accurate when predicting their exam performance. Repeating students were overconfident, dissatisfied, and inaccurate when predicting their first exam performance on their initial, unsuccessful attempt but were more successful on their second, repeat attempt. Qualitative analysis revealed that repeating students aimed to improve their studying by modifying their existing study strategies and managing their time more effectively. In conjunction with other known risk factors, these insights into repeater and non-repeater exam preparation practices can help anatomy educators better identify and support potential struggling students.

A phenomenological exploration of the feedback experience of medical students after summative exam failure

BACKGROUND

Preventing medical students entering cycles of underperformance following assessment is a priority due to the consequences for the student, faculty, and wider society. The benefits from feedback may be inadequately accessed by students in difficulty due to the emotional response evoked by examination failure. This study aims to explore medical students' experiences of receiving feedback after summative assessment failure and investigate the role of emotions on motivation for learning after underperformance, to better support remediation and preparation for future assessments.

METHODS

This study used interpretative phenomenological analysis (IPA) to explore the experiences of four medical students who failed summative assessments. Additionally, a content analysis was conducted using Linguistic Inquiry and Word Count (LIWC) to investigate the characteristics and use of language to describe their emotional response.

RESULTS

Anger, fear, anxiety, and sadness were emotions frequently experienced after examination failure. These emotions led to feelings of mistrust of the medical school and subsequent distrust in the university's assessment processes, impacting on the desire to engage with feedback. There was dissonance between the students' perceptions of what feedback should provide and what benefit feedback provided after summative assessments. The linguistic inquiry further confirmed an initial (and sometimes long lived) negative affective state after experiencing failure, and a barrier to engagement with remediation when not effectively managed.

CONCLUSIONS

A range of emotions, directed at themselves and the medical school are experienced by students following exam failure. These emotions lead to a range of negative feelings and responses that affect how students make sense of and move on from the failure experience. There is a need for educators to better understand and support students to manage, reflect and contextualise their emotional responses, minimise external attribution and to enable focus on remediation and learning.

Conscientious, competent and caring: producing the junior doctor of the future

This article is based on the Association for the Study of Medical Education Gold Medal Plenary for 2022, given by the first author. It outlines different ways in which medical training can be approached, based on his career and his work with colleagues. Among the attributes that it would be desirable to promote in future doctors are conscientiousness, competence and care for patients as individuals. This article explores each of these in separate sections. The first demonstrates that the trait of conscientiousness can be observed in first and second year medical students by their compliance in routine low level tasks such as attendance and submission of required work on time. A 'conscientiousness index' calculated on this basis is a statistically significant predictor of later events such as performance in exams, the prescribing safety assessment, and the UK situational judgement test in subsequent years, and also in postgraduate assessments such as Royal college exams and the annual reviews of competence progression. The second proposes that competence in tasks undertaken by junior doctors is better achieved by teaching on medical imaging, clinical skills and living anatomy than by cadaveric dissection. The final section argues that the incorporation of arts and humanities teaching into medical education is likely to lead to better understanding of the patient perspective in later practice.

Remediation practices for health profession students and clinicians: An integrative review

BACKGROUND

The current state of practices in health care remediation is not well known. The purpose of this review is to characterize, assess, and present synthesized results of current student and professional remediation practices described in the literature.

METHODS

This study used an integrative review process including article extraction and review, descriptive characterization and statistics, classification of levels of evidence, assessment of risk of bias, and examination of relationships between factors and types of remediation. Articles were located in a search of PubMed (MEDLINE) and EBSCO (CINAHL Complete) last accessed in May 2022.

INCLUSION CRITERIA

Full text journal articles and Briefs published between January 2001 and May 2022, English language, focus on remediation in health science education programs and professionals, identified key words in title, abstract, or article.

EXCLUSION CRITERIA

Published outside the date range; focus of study or article outside health sciences; main focus not on remediation process or program (defined above), books, presentations and abstracts.

RESULTS

97 articles were included. Design rigor clustered around Level 6 (case-controlled studies, case series, case reports). All programs and activities were reported as successful. There was a statistically significant relationship (p < 0.01) between healthcare discipline and type of remediation.

CONCLUSIONS

A variety of remediation methods for health care students and professionals are reported to be successful. Higher level studies are needed to help define best practices for remediation activities in health care professional knowledge and skill.

Pharmacy education and workforce: strategic recommendations based on expert consensus in Lebanon

Pharmacy in Lebanon has been taught for years, and the profession has known the golden ages in previous years. However, with the recent graduation of hundreds of pharmacists, without prior workforce planning, the oversupply of non-specialized pharmacists caused a mismatch with the needs of the market. The context of severe socioeconomic and sanitary crises has further exacerbated the situation, with hundreds of pharmacists leaving the country. A group of pharmacy experts joined to suggest strategic solutions to face such challenges, suggesting a clear strategy for education and the workforce, overarched by educational and professional values and based on six main pillars: (1) implement a national competency framework (including the core and specialized competency frameworks) to be used as a basis for licensure (colloquium); (2) implement a national pharmacy program accreditation, encompassing standards related to competencies adoption and assessment, curricula, teaching methods, research and innovation, instructors' and preceptors' skills, and experiential training; (3) organize training for students and early-career pharmacists; (4) optimize continuing education and implement continuous professional development, fostering innovation and specialization among working pharmacists; (5) develop and implement a pharmacy workforce strategy based on pharmacy intelligence, job market, and academic capacities; (6) develop and implement a legal framework for the above-mentioned pillars in collaboration with ministries and parliamentary commissions. Under the auspices of the relevant authorities, mainly the Order of Pharmacists of Lebanon and the Ministry of Education and Higher Education, the suggested strategy should be discussed and implemented for a better future for the pharmacy profession.

Warning: medical education is hazardous to your mental health. Medical students should make an informed decision to begin and continue training

For decades there has been ample evidence that training to become a physician and practicing medicine is hazardous to one's health and wellness. In the face of the extremely high rates of suicide, substance abuse, depression and burnout in the medical student, resident, and physician populations, it would be dishonest to suggest medical education and practice is all gain and no pain. This article is directed to members of the medical education community and challenges stakeholders to view their teaching and training of medical students as an intervention requiring free and informed consent. We hope this exercise shifts the paradigm of educators and enables students to enter medical training from a free and informed position.

Technology enhanced assessment: Ottawa consensus statement and recommendations

INTRODUCTION

In 2011, a consensus report was produced on technology-enhanced assessment (TEA), its good practices, and future perspectives. Since then, technological advances have enabled innovative practices and tools that have revolutionised how learners are assessed. In this updated consensus, we bring together the potential of technology and the ultimate goals of assessment on learner attainment, faculty development, and improved healthcare practices.

METHODS

As a material for the report, we used the scholarly publications on TEA in both HPE and general higher education, feedback from 2020 Ottawa Conference workshops, and scholarly publications on assessment technology practices during the Covid-19 pandemic.

RESULTS AND CONCLUSION

The group identified areas of consensus that remained to be resolved and issues that arose in the evolution of TEA. We adopted a three-stage approach (readiness to adopt technology, application of assessment technology, and evaluation/dissemination). The application stage adopted an assessment 'lifecycle' approach and targeted five key foci: (1) Advancing authenticity of assessment, (2) Engaging learners with assessment, (3) Enhancing design and scheduling, (4) Optimising assessment delivery and recording learner achievement, and (5) Tracking learner progress and faculty activity and thereby supporting longitudinal learning and continuous assessment.

How good are medical students at communicating risk? An implementation study at three German medical schools

OBJECTIVE

To assess the objective and subjective risk communication skills of medical students in three universities in Germany METHODS: We developed a risk communication skills Objective Structured Clinical Examination (OSCE) station and implemented it in three medical schools in Germany. 596 students contributed data to a risk communication checklist and a risk communication self-assessment. Multiple linear regression models were used to identify factors associated with the risk communication OSCE performance.

RESULTS

Participants in our study achieved on average 73.5% of the total risk communication skills score, which did not differ between locations (F(2-595) = 1.96; p = 0.142). The mean objective performance of students who assessed their skills as poor was significantly worse than the performance of students who assessed their skills as good (t(520) = -5.01, p < 0.001). The risk communication skills score was associated with native language but not with gender nor General Point Average (p < 0.001).

CONCLUSIONS

Medical students demonstrated acceptable risk communication skills scores and were able to self-assess their performance. However, selected communication techniques should be re-emphasised in the undergraduate medical curriculum.

PRACTICE IMPLICATIONS

Our research identified shortcomings in particular subgroups that can be addressed through tailored curriculum interventions.

Emotion in remediation: A scoping review of the medical education literature

OBJECTIVES

Remediation can be crucial and high stakes for medical learners, and experts agree it is often not optimally conducted. Research from other fields indicates that explicit incorporation of emotion improves education because of emotion's documented impacts on learning. Because this could present an important opportunity for improving remediation, we aimed to investigate how the literature on remediation interventions in medical education discusses emotion.

METHODS

The authors used Arksey and O'Malley's framework to conduct a scoping literature review of records describing remediation interventions in medical education, using PubMed, CINAHL Complete, ERIC, Web of Science and APA PsycInfo databases, including all English-language publications through 1 May 2020 meeting search criteria. They included publications discussing remediation interventions either empirically or theoretically, pertaining to physicians or physician trainees of any level. Two independent reviewers used a standardised data extraction form to report descriptive information; they reviewed included records for the presence of mentions of emotion, described the mentions and analysed results thematically.

RESULTS

Of 1644 records, 199 met inclusion criteria and were reviewed in full. Of those, 112 (56%) mentioned emotion in some way; others focused solely on cognitive aspects of remediation. The mentions of emotion fell into three themes based on when the emotion was cited as present: during regular coursework or practice, upon referral for remediation and during remediation. One-quarter of records (50) indicated potential intentional incorporation of emotion into remediation programme design, but they were non-specific as to how emotions related to the learning process itself.

CONCLUSION

Even though emotion is omnipresent in remediation, medical educators frequently do not factor emotion into the design of remediation approaches and rarely explicitly utilise emotion to improve the learning process. Applications from other fields may help medical educators leverage emotion to improve learning in remediation, including strategies to frame and design remediation.

Remediation and reassessment methods in pharmacy education: A systematic review

BACKGROUND

Colleges of pharmacy are currently required to implement a remediation program within their curricula, but no specifications are provided on the ideal methodology. While the need for successful remediation strategies continues to grow, literature describing positive or negative outcomes of different approaches is significantly lacking. The objective of this literature review was to describe and evaluate remediation methodologies in pharmacy education.

METHODS

This literature review was completed following PRISMA criteria. A search of the PubMed, Cochrane Library, Cumulative Index of Nursing and Allied Health, Academic Search Complete, PsycInfo, Scopus, and ProQuest Central databases was conducted in July 2019. Studies were included if they involved pharmacy student education and described either remediation or reassessment.

RESULTS

The evaluated studies discussed a range of course types being remediated, a large variety of remediation strategies and timeframes, and differing overall outcomes. No studies provided comparison of remediation techniques or provided details on the implementation of their chosen approaches. A consistent finding within the evaluated studies was the inclusion of prevention strategies to attempt to avoid the need for remediation preemptively. Overall outcomes for each remedial program were inconsistent and no clear patterns were evident other than an improvement in student performance following remediation.

IMPLICATIONS

Remediation strategies included course repetition, summer restudy, reassessment, and individualized plans. Outcomes varied significantly between studies, making methodology comparisons difficult. Future studies that include more detail and consistency in the reported outcomes would be beneficial to students and help clarify remediation for colleges of pharmacy.

Guidelines: The dos, don'ts and don't knows of remediation in medical education

INTRODUCTION

Two developing forces have achieved prominence in medical education: the advent of competency-based assessments and a growing commitment to expand access to medicine for a broader range of learners with a wider array of preparation. Remediation is intended to support all learners to achieve sufficient competence. Therefore, it is timely to provide practical guidelines for remediation in medical education that clarify best practices, practices to avoid, and areas requiring further research, in order to guide work with both individual struggling learners and development of training program policies.

METHODS

Collectively, we generated an initial list of Do's, Don'ts, and Don't Knows for remediation in medical education, which was then iteratively refined through discussions and additional evidence-gathering. The final guidelines were then graded for the strength of the evidence by consensus.

RESULTS

We present 26 guidelines: two groupings of Do's (systems-level interventions and recommendations for individual learners), along with short lists of Don'ts and Don't Knows, and our interpretation of the strength of current evidence for each guideline.

CONCLUSIONS

Remediation is a high-stakes, highly complex process involving learners, faculty, systems, and societal factors. Our synthesis resulted in a list of guidelines that summarize the current state of educational theory and empirical evidence that can improve remediation processes at individual and institutional levels. Important unanswered questions remain; ongoing research can further improve remediation practices to ensure the appropriate support for learners, institutions, and society.

Interventions for undergraduate and postgraduate medical learners with academic difficulties: A BEME systematic review: BEME Guide No. 56

Background: Clinical teachers often struggle to report unsatisfactory trainee performance, partly because of a lack of evidence-based remediation options. Objectives: To identify interventions for undergraduate (UG) and postgraduate (PG) medical learners experiencing academic difficulties, link them to a theory-based framework and provide literature-based recommendations around their use. Methods: This systematic review searched MEDLINE, CINAHL, EMBASE, ERIC, Education Source and PsycINFO (1990-2016) combining these concepts: medical education, professional competence/difficulty and educational support. Original research/innovation reports describing intervention(s) for UG/PG medical learners with academic difficulties were included. Data extraction employed Michie's Behavior Change Techniques (BCT) Taxonomy and program evaluation models from Stufflebeam and Kirkpatrick. Quality appraisal used the Mixed Methods Appraisal Tool (MMAT). The authors synthesized extracted evidence by adapting the GRADE approach to formulate recommendations. Results: Sixty-eight articles met the inclusion criteria, most commonly addressing knowledge (66.2%), skills (53.9%) and attitudinal problems (26.2%), or learner personal issues (41.5%). The most common BCTs were Shaping knowledge, Feedback/monitoring, and Repetition/substitution. Quality appraisal was variable (MMAT 0-100%). A thematic content analysis identified 109 interventions (UG: n = 84, PG: n = 58), providing 24 strong, 48 moderate, 26 weak and 11 very weak recommendations. Conclusion: This review provides a repertoire of literature-based interventions for teaching/learning, faculty development, and research purposes.

Twelve tips for clinicians dealing with uncertainty when assessing learners

Clinician educators often experience distress caused by uncertainty regarding how effectively to participate in assessment practices in a way that supports both their programs and their students. Uncertainty is a common state for clinicians, particularly for those who see patients with early or ill-defined illness presentations. While clinicians often feel ill at ease when facing uncertainty in the clinical realm, becoming comfortable with uncertainty and learning to manage such states are now recognized as vital components of clinical practice. Clinicians, as a result, have adopted a series of strategies to lessen the unease that uncertainty can create. While similar experiences plague clinician educators placed in assessment roles, much less attention has been given to how we can support individuals in the education setting. Here, the distress of uncertainty may be greater due to clinician educators having less experience with assessment practices. Fortunately, strategies that are effective in the clinical domain can be translated into the assessment realm to accommodate uncertainty when assessing learners. In this 12 tips article we offer guidance on the translation of such strategies.

A Randomized Trial Comparing Freely Moving and Zonal Instruction of Veterinary Surgical Skills Using Ovariohysterectomy Models

Timely, specific feedback is the most important feature of simulation-based training, but providing adequate instructor supervision is challenging. Students' (n = 76) surgical skills were assessed after training using either the traditional (T) method of large-group teaching by multiple instructors or the alternative method of one instructor assigned (A) to a defined group of students. Instructors rotated to a different group of students for each laboratory session. The instructor-to-student ratio and environment remained identical. No differences were found in raw assessment scores or the number of students requiring remediation, suggesting that students learned in this environment whether they received feedback from one instructor or multiple. Students had no preference between the methods, though 88% of the instructors preferred the assigned method, because they perceived an increased ability to teach and observe individual students. There was no difference in the number of students identified as at-risk of remediation between groups. When both groups were considered together, students identified as at-risk were more likely (40% vs. 10%) to require post-assessment remediation. However, only 22% of students requiring remediation had been identified as at-risk, and A-group instructors were more accurate than T-group instructors at identifying at-risk students. These results suggest that students accept either instructional method, but most instructors prefer to be assigned to a small group of students. Surgical skills were learned similarly well by students in both groups, although assigned instructors were more accurate at identifying at-risk students, which could prove beneficial if early intervention measures can be offered.

Borderline grades in high stakes clinical examinations: resolving examiner uncertainty

BACKGROUND

Objective Structured Clinical Exams are used to increase reliability and validity, yet they only achieve a modest level of reliability. This low reliability is due in part to examiner variance which is greater than the variance of students. This variance often represents indecisiveness at the cut score with apparent confusion over terms such as "borderline pass". It is amplified by a well reported failure to fail.

METHODS

A borderline grade (meaning performance is neither a clear pass nor a clear fail) was introduced in a high stakes undergraduate medical clinical skills exam to replace a borderline pass grade (which was historically resolved as 50%) in a 4 point scale (distinction, pass, borderline, fail). Each Borderline grade was then resolved into a Pass or Fail grade by a formula referencing the difficulty of the station and the performance in the same domain by the student in other stations. Raw pass or fail grades were unaltered. Mean scores and 95%CI were calculated per station and per domain for the unmodified and the modified scores/grades (results are presented on error bars). To estimate the defensibility of these modifications, similar analysis took place for the P and the F grades which resulted from the modification of the B grades.

RESULTS

Of 14,634 observations 4.69% were Borderline. Application of the formula did not impact the mean scores in each domain but the failure rate for the exam increased from 0.7 to 4.1%. Examiners and students expressed satisfaction with the Borderline grade, resolution formula and outcomes. Mean scores (by stations and by domains respectively) of students whose B grades were modified to P were significantly higher than their counterparts whose B grades were modified to F.

CONCLUSIONS

This study provides a feasible and defensible resolution to situations where the examinee's performance is neither a clear pass nor a clear fail, demonstrating the application of the resolution of borderline formula in a high stakes exam. It does not create a new performance standard but utilises real data to make judgements about these small number of candidates. This is perceived as a fair approach to Pass/Fail decisions.

The benefits of sequential testing: Improved diagnostic accuracy and better outcomes for failing students

INTRODUCTION

In recent decades, there has been a move towards standardized models of assessment where all students sit the same test (e.g. OSCE). By contrast, in a sequential test the examination is in two parts, a "screening" test (S1) that all candidates take, and then a second "test" (S2) which only the weaker candidates sit. This article investigates the diagnostic accuracy of this assessment design, and investigates failing students' subsequent performance under this model.

METHODS

Using recent undergraduate knowledge and performance data, we compare S1 "decisions" to S2 overall pass/fail decisions to assess diagnostic accuracy in a sequential model. We also evaluate the longitudinal performance of failing students using changes in percentile ranks over a full repeated year.

FINDINGS

We find a small but important improvement in diagnostic accuracy under a sequential model (of the order 2-4% of students misclassified under a traditional model). Further, after a resit year, weaker students' rankings relative to their peers improve by 20-30 percentile points.

DISCUSSION

These findings provide strong empirical support for the theoretical arguments in favor of a sequential testing model of assessment, particularly that diagnostic accuracy and longitudinal assessment outcomes post-remediation for the weakest students are both improved.

Managing extremes of assessor judgment within the OSCE

CONTEXT

There is a growing body of research investigating assessor judgments in complex performance environments such as OSCE examinations. Post hoc analysis can be employed to identify some elements of "unwanted" assessor variance. However, the impact of individual, apparently "extreme" assessors on OSCE quality, assessment outcomes and pass/fail decisions has not been previously explored. This paper uses a range of "case studies" as examples to illustrate the impact that "extreme" examiners can have in OSCEs, and gives pragmatic suggestions to successfully alleviating problems.

METHOD AND RESULTS

We used real OSCE assessment data from a number of examinations where at station level, a single examiner assesses student performance using a global grade and a key features checklist. Three exemplar case studies where initial post hoc analysis has indicated problematic individual assessor behavior are considered and discussed in detail, highlighting both the impact of individual examiner behavior and station design on subsequent judgments.

CONCLUSIONS

In complex assessment environments, institutions have a duty to maximize the defensibility, quality and validity of the assessment process. A key element of this involves critical analysis, through a range of approaches, of assessor judgments. However, care must be taken when assuming that apparent aberrant examiner behavior is automatically just that.

Critical review: medical students' motivation after failure

About 10 % of students in each years' entrants to medical school will encounter academic failure at some stage in their programme. The usual approach to supporting these students is to offer them short term remedial study programmes that often enhance approaches to study that are orientated towards avoiding failure. In this critical review I will summarise the current theories about student motivation that are most relevant to this group of students and describe how they are enhanced or not by various contextual factors that medical students experience during their programme. I will conclude by suggesting ways in which support programmes for students who have encountered academic failure might be better designed and researched in the future.

Diagnostic Reasoning Assessment Toolkit: Guided Reflection and Standardized Cases for At-Risk Final-Year Medical Students

INTRODUCTION

A failing diagnostic reasoning performance may represent student deficiency in a number of potential areas. However, many standard clinical skills assessments do not offer detailed assessments of diagnostic reasoning ability. This toolkit was designed to identify specific learner deficiencies with respect to diagnostic reasoning by focusing on individual student remedial work and by standardizing faculty evaluation.

METHODS

Educational objectives were derived from institutional patient care competency learning objectives at the Indiana University School of Medicine. Review of existing clinical skills remediation literature yielded a design that combined two learning methods: guided reflection and standardized patient cases.

RESULTS

Over the 2014-2015 academic year, 12 final-year medical students used this resource to help develop an individual remedial learning plan prior to retaking a failed standardized assessment. Students were generally satisfied with the combined guided reflection and standardized case learning methods.

DISCUSSION

Unique final-year medical student scheduling pressures, combined with a reporting time line for both institutional high-stakes OSCE remediation exams and the USMLE Step 2 Clinical Skills exam, incentivized failing students to schedule a retest on a short time line, often leaving little time for critical preparation. This resource offered an opportunity to efficiently spend limited preparation time to individualize exam preparation using a variety of faculty facilitators. The simplistic design was readily deployable to multiple faculty remediation mentors. Our institution can now provide a standardized diagnostic reasoning remedial evaluation using numerous clinical faculty based at any of our nine campuses.

When students struggle with gross anatomy and histology: A strategy for monitoring, reviewing, and promoting student academic success in an integrated preclinical medical curriculum

Gross anatomy and histology are now often taught as parts of an integrated medical or dental curriculum. Although this puts these foundational basic sciences into a wider educational context, students may not fully appreciate their importance as essential components of their medical education and may not develop a sufficient level of competency, as they are not stand-alone courses. The early identification of medical or dental students who struggle with anatomy or histology and the facilitation of adequate didactic support constitute a significant problem in an integrated curriculum. The timely intervention by an academic review board in combination with an individualized faculty-mediated counseling and remediation process may provide an effective solution to this problem.

Medical students' personal experience of high-stakes failure: case studies using interpretative phenomenological analysis

BACKGROUND

Failing a high-stakes assessment at medical school is a major event for those who go through the experience. Students who fail at medical school may be more likely to struggle in professional practice, therefore helping individuals overcome problems and respond appropriately is important. There is little understanding about what factors influence how individuals experience failure or make sense of the failing experience in remediation. The aim of this study was to investigate the complexity surrounding the failure experience from the student's perspective using interpretative phenomenological analysis (IPA).

METHODS

The accounts of three medical students who had failed final re-sit exams, were subjected to in-depth analysis using IPA methodology. IPA was used to analyse each transcript case-by-case allowing the researcher to make sense of the participant's subjective world. The analysis process allowed the complexity surrounding the failure to be highlighted, alongside a narrative describing how students made sense of the experience.

RESULTS

The circumstances surrounding students as they approached assessment and experienced failure at finals were a complex interaction between academic problems, personal problems (specifically finance and relationships), strained relationships with friends, family or faculty, and various mental health problems. Each student experienced multi-dimensional issues, each with their own individual combination of problems, but experienced remediation as a one-dimensional intervention with focus only on improving performance in written exams. What these students needed to be included was help with clinical skills, plus social and emotional support. Fear of termination of the their course was a barrier to open communication with staff.

CONCLUSIONS

These students' experience of failure was complex. The experience of remediation is influenced by the way in which students make sense of failing. Generic remediation programmes may fail to meet the needs of students for whom personal, social and mental health issues are a part of the picture.

Twelve tips for increasing the defensibility of assessment decisions

In an era of increasing scrutiny of the performance of graduates in the workplace, there can be frustrations when decisions about learners with borderline or poor performance in formal assessments are over-turned in appeal processes. This article addresses the approach to reducing the uncertainty about assessment decisions and surviving appeals.

The first OSCE; does students' experience of performing in public affect their results?

BACKGROUND

Personal qualities have been shown to affect students' exam results. We studied the effect of experience, and level, of public performance in music, drama, dance, sport, and debate at the time of admission to medical school as a predictor of student achievement in their first objective structured clinical examination (OSCE).

METHODS

A single medical school cohort (n = 265) sitting their first clinical exam in 2011 as third year students were studied. Pre-admission statements made at the time of application were coded for their stated achievements in the level of public performance; participation in each activity was scored 0-3, where 0 was no record, 1 = leisure time activity, 2 = activity at school or local level, 3 = activity at district, regional or national level. These scores were correlated to OSCE results by linear regression and t-test. Comparison was made between the highest scoring students in each area, and students scoring zero by t-test.

RESULTS

There was a bell shaped distribution in public performance score in this cohort. There was no significant linear regression relationship between OSCE results and overall performance score, or between any subgroups. There was a significant difference between students with high scores in theatre, debate and vocal music areas, grouped together as verbal performance, and students scoring zero in these areas. (p < 0.05, t-test) with an effect size of 0.4.

CONCLUSIONS

We found modest effects from pre-admission experience of verbal performance on students' scores in the OSCE examination. As these data are taken from students' admission statements, we call into question the received wisdom that such statements are unreliable.

Remediation practices in Canadian psychiatry clerkships

OBJECTIVES

There is a lack of information regarding interventions for undergraduate students at Canadian medical schools who require remediation during their psychiatric training. The need for a theoretical framework to guide remediation has been identified. In this study, we sought to characterize remediation taking place in undergraduate psychiatry education, particularly during clerkship. A secondary goal was to ascertain whether those responsible for remediation were aware of the stages of change (transtheoretical) model, and whether they formally incorporated this framework (or another) into their remediation processes.

METHODS

A short six-question survey on remediation practices was e-mailed to educators responsible for undergraduate psychiatry education at all 17 Canadian medical schools as identified through the Canadian Organization of Undergraduate Psychiatry Educators (COUPE).

RESULTS

The response rate was 67 %. Respondents' overall impressions of their school's remediation process were that it was "highly effective" (25 %), or "somewhat effective" (67 %); 8 % reported being unsure of its effectiveness. While 75 % of survey respondents were aware of the stages of change model, only 17 % reported using this framework: no alternate theoretical frameworks were reported.

CONCLUSIONS

The most common form of evaluation was multiple-choice question (MCQ) exams, and the most common form of remediation was exam rewrites. There is little information regarding the long-term outcomes of remediation, and further research would be useful in formulating recommendations regarding best practices.

Prediction and prevention of failure: an early intervention to assist at-risk medical students

BACKGROUND

Consistent identification and prevention of failure for at-risk medical students is challenging, failing courses is costly to all stakeholders, and there is need for further research into duration, timing and structure of interventions to help students in difficulty.

AIMS

To verify the value of a new exam two weeks into medical school as a predictor of failure, and explore the requirements for a preventative intervention.

METHODS

Students who failed the two-week exam were invited to a series of large-group workshops and small-group follow-up meetings. Participants' subsequent exam performance was compared with non-participants.

RESULTS

About 71% of students who performed poorly in the new exam subsequently failed a course. Attendance at the workshops made no difference to short- or long-term pass rates. Attendance at more than three follow-up small group sessions significantly improved pass rates two semesters later, and was influenced by teacher experience.

CONCLUSIONS

Close similarity between predictor task and target task is important for accurate prediction of failure. Consideration should be given to dose effect and class size in the prevention of failure of at-risk students, and we recommend a systemic approach to intervention/remediation programmes, involving a whole semester of mandatory, weekly small group meetings with experienced teachers.

Advancing the objective structured clinical examination: sequential testing in theory and practice

CONTEXT

Models of short-term remediation for failing students are typically associated with improvements in candidate performance at retest. However, the process is costly to deliver, particularly for performance retests with objective structured clinical examinations (OSCEs), and there is increasing evidence that these traditional models are associated with the longitudinal underperformance of candidates.

METHODS

Rather than a traditional OSCE model, sequential testing involves a shorter 'screening' format, with an additional 'sequential' test for candidates who fail to meet the screening standard. For those tested twice, overall pass/fail decisions are then based on results on the full sequence of tests. In this study, the impacts of sequential assessment on student performance, cost of assessment delivery and overall reliability were modelled using data sourced from a final graduating OSCE in an undergraduate medical degree programme.

RESULTS

Initial modelling using pre-existing OSCE data predicted significant improvements in reliability in the critical area, reflected in pilot results: 13.5% of students (n = 228) were required to sit the sequential OSCE. One student (0.4%) was identified as representing a false positive result (i.e. under the previous system this student would have passed the OSCE but failed on extended testing). Nine students (3.9%) who would have required OSCE retests under the prior system passed the full sequence and were therefore able to graduate at the normal time without loss of earnings. Overall reliability was estimated as 0.79 for the full test sequence. Significant cost savings were realised.

CONCLUSIONS

Sequential testing in OSCEs increases reliability for borderline students because the increased number of observations implies that 'observed' student marks are closer to 'true' marks. However, the station-level quality of the assessment needs to be sufficiently high for the full benefits in terms of reliability to be achieved. The introduction of such a system has financial benefits, good validity inferences and has proved acceptable to students and other stakeholders.

The remediation challenge: theoretical and methodological insights from a systematic review

OBJECTIVES

Remediation is usually offered to medical students and doctors in training who underperform on written or clinical examinations. However, there is uncertainty and conflicting evidence about the effectiveness of remediation. The aim of this systematic review was to synthesise the available evidence to clarify how and why remediation interventions may have worked in order to progress knowledge on this topic.

METHODS

The MEDLINE, EMBASE, CINAHL (Cumulative Index to Nursing and Allied Health Literature), ERIC (Educational Resources Information Centre), Web of Science and Scopus databases were searched for papers published from 1984 to April 2012, using the search terms 'remedial teaching', 'education', 'medical', 'undergraduate'/or 'clinical clerkship'/or 'internship and residency', 'at risk' and 'struggling'. Only studies that included an intervention, then provided retest data, and reported at least one outcome measure of satisfaction, knowledge, skills or effects on patients were eligible for inclusion. Studies of practising doctors were excluded. Data were abstracted independently in duplicate for all items. Coding differences were resolved through discussion.

RESULTS

Thirty-one of 2113 studies met the review criteria. Most studies were published after 2000 (n=24, of which 12 were published from 2009 onwards), targeted medical students (n=22) and were designed to improve performance on an immediately subsequent examination (n=22). Control or comparison groups, conceptual frameworks, adequate sample sizes and long-term follow-up measures were rare. In studies that included long-term follow-up, improvements were not sustained. Intervention designs tended to be highly complex, but their design or reporting did not enable the identification of the active components of the remedial process.

CONCLUSIONS

Most remediation interventions in medical education focus on improving performance to pass a re-sit of an examination or assessment and provide no insight into what types of extra support work, or how much extra teaching is critical, in terms of developing learning. More recent studies are generally of better quality. Rigorous approaches to developing and evaluating remediation interventions are required.

The effect of a short integrated study skills programme for first-year medical students at risk of failure: a randomised controlled trial

BACKGROUND

There is a need for outcome-based studies on strategies for supporting at-risk medical students that use long-term follow-up and contemporaneous controls.

AIM

To measure the effect of a short integrated study skills programme (SSP) on the study progress of at-risk medical students.

METHODS

First-year students identified as at-risk of academic failure at 7 months after enrolment were invited to participate in the randomised controlled trial. Participants were randomly assigned to the SSP group or to a control group receiving standard academic support. Effects of SSP were measured on the short (passed first exam after intervention), medium (obtained enough credits to proceed to second year) and long term (completed first-year curriculum within 2 years).

RESULTS

SSP participants (n=43) more often passed the first exam after the intervention than controls (n=41; 30% versus 12%; X2(1)=4.06, p<0.005, effect size=0.22), in particular those who had previously passed at least one exam. No medium or long-term effect was found. Participants who had attended four or five SSP sessions outperformed those who had attended fewer sessions on all outcome measures.

CONCLUSION

A short, integrated SSP benefited some, but not all students. Our advice is to focus support efforts on at-risk students who have demonstrated commitment and academic potential.