Prediction and prevention of failure: an early intervention to assist at-risk medical students

The potential of structured learning diaries for combining the development and assessment of self-regulated learning

Structured Self-Regulated Learning (SSRL) diaries have the potential for combining the development and assessment of a student's SRL processes over time. The aim of this study was to evaluate the extent to which an SSRL diary can develop SRL and provide a reliable longitudinal assessment of SRL development in academically low-achieving undergraduate medical students. We conducted a quasi-experimental study with low-achieving medical students at Tehran University of Medical Sciences. The intervention was a weekly SSRL diary, with 21 items in two parts (before and after studying) that was integrated with weekly explicit SRL training. A repeated measures ANOVA was performed to assess the participants' SRL development across time. We conducted generalizability theory analysis in two designs; the first was to assess the reliability of the total diary score in assessing low-achieving medical students' SRL and the second was to assess the efficacy of the four-week intervention results in improving the low-achieving medical students' SRL. Each participant (n = 20) completed four SSRL diaries. There were significant positive changes during the intervention in the students' measures of total SRL, time estimation of study, time spent on study, concentration, elaboration, organization, self-monitoring, and self-evaluation. The absolute and relative generalizability coefficients for the first design, which indicates the reliability of the students' SRL scores, were 0.71 and 0.77. The absolute and relative generalizability coefficients for the second design, which presents the reliability of the improvement of students' SRL across time were 0.79 and 0.87. The results of this study suggest that SSRL diaries combined with explicit training can provide an effective and reliable method for combining the development and assessment of SRL in low-achieving medical students' SRL over time.

Using Self-Regulated Learning Microanalysis to Examine Regulatory Processes in Clerkship Students Engaged in Practice Questions

Introduction

Self-regulated learning is a cyclical process of forethought, performance, and self-reflection that has been used as an assessment tool in medical education. No prior studies have evaluated SRL processes for answering multiple-choice questions (MCQs) and most evaluated one or two iterations of a non-MCQ task. SRL assessment during MCQs may elucidate reasons why learners are successful or not on these questions that are encountered repeatedly during medical education.

Methods

Internal medicine clerkship students at three institutions participated in a SRL microanalytic protocol that targeted strategic planning, metacognitive monitoring, causal attributions, and adaptive inferences across seven MCQs. Responses were transcribed and coded according to previously published methods for microanalytic protocols.

Results

Forty-four students participated. In the forethought phase, students commonly endorsed prioritizing relevant features as their diagnostic strategy (n = 20, 45%) but few mentioned higher-order diagnostic reasoning processes such as integrating clinical information (n = 5, 11%) or comparing/contrasting diagnoses (n = 0, 0%). However, in the performance phase, students' metacognitive processes included high frequencies of integration (n = 38, 86%) and comparing/contrasting (n = 24, 55%). In the self-reflection phase, 93% (n = 41) of students faulted their management reasoning and 84% (n = 37) made negative references to their abilities. Less than 10% (n = 4) of students indicated that they would adapt their diagnostic reasoning process for these questions.

Discussion

This study describes in detail student self-regulatory processes during MCQs. We found that students engaged in higher-order diagnostic reasoning processes but were not explicit about it and seldom reflected critically on these processes after selecting an incorrect answer. Self-reflections focused almost exclusively on management reasoning and negative references to abilities which may decrease self-efficacy. Encouraging students to identify and evaluate diagnostic reasoning processes and make attributions to controllable factors may improve performance.

Conscientious, competent and caring: producing the junior doctor of the future

This article is based on the Association for the Study of Medical Education Gold Medal Plenary for 2022, given by the first author. It outlines different ways in which medical training can be approached, based on his career and his work with colleagues. Among the attributes that it would be desirable to promote in future doctors are conscientiousness, competence and care for patients as individuals. This article explores each of these in separate sections. The first demonstrates that the trait of conscientiousness can be observed in first and second year medical students by their compliance in routine low level tasks such as attendance and submission of required work on time. A 'conscientiousness index' calculated on this basis is a statistically significant predictor of later events such as performance in exams, the prescribing safety assessment, and the UK situational judgement test in subsequent years, and also in postgraduate assessments such as Royal college exams and the annual reviews of competence progression. The second proposes that competence in tasks undertaken by junior doctors is better achieved by teaching on medical imaging, clinical skills and living anatomy than by cadaveric dissection. The final section argues that the incorporation of arts and humanities teaching into medical education is likely to lead to better understanding of the patient perspective in later practice.

Developing low-achieving medical students' self-regulated learning using a combined learning diary and explicit training intervention

INTRODUCTION

The development of self-regulated learning (SRL) is an essential educational component of remediation for low-achieving students. The aim of this study was to design, implement, and evaluate a longitudinal SRL intervention combining both a structured learning diary and explicit SRL training in a cohort of low-achieving undergraduate medical students.

MATERIALS AND METHODS

A mixed methods quasi-experimental study was conducted, with a pretest-posttest study in the intervention group and comparison of the GPA and course grade of the intervention group with a historical comparison group. A questionnaire and focus group explored the participants' perceptions about the intervention.

RESULTS

The SRL scores (total and rehearsal, organization, critical thinking, metacognitive regulation, time management and environment management) and course grade of participants were significantly improved in the intervention group. The course grade of participants was significantly higher than the comparison group but the GPA was not significantly different. Overall, the participants were positive about the intervention.

CONCLUSIONS

This study was the first in medical education to evaluate the effectiveness and user acceptability of an SRL intervention that combined a structured learning diary and explicit SRL training in low-achieving medical students. Further research is recommended in different contexts and with larger number of students.

Determining baseline anatomy knowledge among professional allied health students

Health professional students often struggle with anatomy coursework despite undergraduate coursework in anatomy. Educators must identify early on whether students may struggle in order to target remediation. The purpose of the study was to elucidate whether an assessment tool administered before the start of a professional anatomy course correlated with allied health students' course performance. Students over four years were given a quiz covering anatomy knowledge they were expected to know upon matriculation to their professional program. A supplemental data form was administered at the course's conclusion to identify prior anatomy experience and topics in which students felt deficient. Pre-quiz scores significantly correlated with examination performance throughout the course. Students reported feeling most deficient in neurobiology (54.9%) and anatomy terminology (39.1%). Videos were created to target these deficient knowledge areas; students who watched the videos did better in course assessments than those who did not. Most respondents (98.0%) recommended students take an undergraduate anatomy course prior to starting a health professional program. These results indicate that a quiz assessing anatomy knowledge among matriculating students may identify students with the potential to struggle in a professional anatomy course early on. Responses outlined areas in which students felt deficient, which allows educators to target topics early with intervention tools such as the review videos in this study. Finally, most respondents strongly recommended undergraduate coursework in anatomy prior to starting a professional health program, which outlines students' recognition that a solid foundation in anatomical knowledge is important to success in professional programs.

'Early identification of struggling pre-clerkship learners using formative clinical skills OSCEs: an assessment for learning program.'

Multiple experts in clinical skills remediation recommend early identification to support struggling learners, but there is minimal documentation on implementation of these programs. We share one school's outcomes-based research utilizing the formative assessment for learning model to early-identify pre-clerkship students struggling with clinical skills using formative OSCEs (F-OSCE). Student scores were monitored over longitudinal F-OSCE experiences as part of a curricular innovation. Points towards early identification accumulated when a student's score fell below the 80% threshold for each section of an OSCE. Students who accumulated enough points were advised of the need for intervention, and coaching was recommended. Students were surveyed about their experiences with the program. The objective was to explore whether this early identification program and coaching intervention had a positive impact on subsequent OSCE performance. Of 184 students in 2 cohorts who completed F-OSCEs, 38 (20.7%) were flagged for early identification. Of these, 17 (44.7%) sought additional help by voluntarily participating in the coaching program. Students who participated in extra clinical skills coaching demonstrated statistically significant improvements in performance on subsequent FOSCEs, as did the early identified students who did not participate in extra coaching. The greatest impact of coaching intervention was noted in the physical examination domain. This program was effective in identifying students struggling with clinical skills on formative OSCEs. Early identified students demonstrated improvements in subsequent OSCE performance, with those who sought coaching faring slightly better. Development of robust early identification programs as formative assessments of clinical skills and follow-up coaching programs to guide skills development are important implications of this work. Monitoring short- and long-term results for students identified through this approach to see if improvement is sustained is planned.

A foundational knowledge assessment tool to predict academic performance of medical students in first-year anatomy and physiology

Misalignments in teaching pedagogies between secondary schools and tertiary institutions have exacerbated educational disparities among students from different backgrounds. Given the variation in students' educational background and competencies, there was a need to develop an Anatomy and Physiology (A&P) Foundational Knowledge Assessment to establish the levels of preparedness of first-year medical students. Previous work that focused on the development of the assessment showed it to be effective in measuring students' foundational knowledge in human anatomy and physiology. The aim of this study was to assess the validity of the A&P Foundational Knowledge Assessment in determining students' prior knowledge and predicting academic performance of first-year students in their anatomy and physiology studies. Three hundred seventy first-year students, across two cohort years, 2017 and 2018, completed the A&P Foundational Knowledge Assessment. Data were analyzed through descriptive statistics, analysis of variance, and Pearson's correlation. Results show that for both cohorts ∼30% of students scored ≤55% and were potentially at risk of performing poorly in their anatomy and physiology studies. Pearson's correlation showed a significant relationship between students' performance on the foundational knowledge assessment and their anatomy and physiology assessments. For both cohorts, >10% of students identified by the A&P Foundational Knowledge Assessment were at risk of either failing the course, entering an extended degree program, or being excluded from the program. Results indicate that the assessment is a good predictor for differentiating medical students' performance in first-year anatomy and physiology.NEW & NOTEWORTHY The development of a foundational knowledge assessment tool to predict academic performance of medical students in first-year anatomy and physiology.

When the wheels fall off - Medical students' experiences of interrupted academic progression

INTRODUCTION

There is limited published research on medical students' perspectives of a significant interruption to their academic progression. This study sought to identify the factors that contribute to difficulties with academic progression and to understand how medical students successfully respond.

METHODS

This interpretive phenomenological study reports on the findings from in-depth interviews of 38 final year medical students who had experienced a significant academic interruption.

RESULTS

The two superordinate themes were: the factors contributing to the interruption and their experience of the interruption. Factors identified as contributing to the interruption were: workload, learning in medicine, motivation for medicine, isolation, adapting to local culture, health and external factors. Their experience of the interruption focused on stages of working through the process: 'what happened,' 'how it felt,' 'managing the failure,' 'accepting the failure' and 'making some changes.'

DISCUSSION

Each factor affected how the participants reacted and responded to the interruption. Regardless of the origins of the interruption, most reacted and responded in a comparable process, albeit with varying timespans. These reactions and responses were in a state of fluctuation. In order to succeed many stated they shifted their motivation from external to internal, in direct response to the interruption, resulting in changed learning behaviours.

CONCLUSIONS

The process of working through an interruption to academic progression for students may benefit from a model of interval debriefing, restorative academic and personal development support. Facilitation of this process could enable students to face an interruption constructively rather than as an insurmountable emotionally burdensome barrier. Medical schools could utilise these findings to implement further support strategies to reduce the number of significant academic disruptions.

Ambivalent professional identity of early remedial medical students from Generation Z: a qualitative study

BACKGROUND

Supporting professional identity development in medical students undergoing remediation in the first few years of their studies is an important topic. However, there is a lack of research on developing an effective and individualised process for successful remediation that targets learner identities. This study examined the identities of Generation Z remedial medical students through the lens of professional identity formation, focusing on the difficulties they faced and the support they sought.

METHODS

An exploratory qualitative case study was conducted within a constructivist paradigm. Twenty-two medical students (14 males and 8 females) who had experienced remediation in their first few years of medical university participated in this study. All participants were members of Generation Z. Qualitative data were collected through face-to-face, semi-structured interviews and analysed using thematic analysis.

RESULTS

Medical students undergoing remediation in the first few years experienced resistance to the medical profession and conflict due to the gap between the ideal and the reality they experienced after entering medical university. Students' professional identities were closely intertwined with their pre-university identities; this affected the process of professional identity formation after entering medical university. They preferred assurances of confidentiality as a prerequisite and immediately sought advice through social networks to support their professional identity development.

CONCLUSIONS

When planning professional identity development support for Generation Z medical students undergoing remediation in the first few years, it is necessary to carefully select integrative interaction methods, focus on the context of individual learners, and collaboratively discuss specific responses between students and faculty. The results of this study could be useful to faculty in developing support systems for future remedial medical students that focuses on professional identity development and mentoring of remedial medical students.

Investigating the risk factors for academic difficulties in the medical programme at a South African university

BACKGROUND

The National Benchmark Test (NBT) that determines academic readiness is widely used by Faculties as an additional measure to select students for the study of medicine. Despite this, many students continue to experience academic challenges that culminate in delayed graduation and sometimes academic exclusion or discontinuation of studies.

AIM

This study aimed to understand academic and non-academic variables linked with academic difficulties in the first three years of medical education.

METHODS

The study sample consisted of six cohorts of medical students for the period 2011 to 2016 (n = 1392). Only the first three of the six-year medical programme were selected for analysis. Survival analysis and Cox Proportional Hazard (CPH) was used to identify academic and non-academic variables associated with academic difficulties.

RESULTS

A total of 475 students (34%) experienced academic difficulty; 221 (16%) in the first year of study, 192 (14%) in the second year and 62 (5%) in the third year of study. The results show that Intermediate Upper, Lower and Basic levels for all NBT domains, living in university residence, rurality and male gender were risk factors for academic difficulty.

CONCLUSION

In mitigating these factors, the NBT must inform the type of support programmes to augment the students' skills and promote academic success. Additionally, existing support programmes should be evaluated to ascertain if they reach students at risk and whether participating in these programmes yield positive academic outcomes.

Emotion in remediation: A scoping review of the medical education literature

OBJECTIVES

Remediation can be crucial and high stakes for medical learners, and experts agree it is often not optimally conducted. Research from other fields indicates that explicit incorporation of emotion improves education because of emotion's documented impacts on learning. Because this could present an important opportunity for improving remediation, we aimed to investigate how the literature on remediation interventions in medical education discusses emotion.

METHODS

The authors used Arksey and O'Malley's framework to conduct a scoping literature review of records describing remediation interventions in medical education, using PubMed, CINAHL Complete, ERIC, Web of Science and APA PsycInfo databases, including all English-language publications through 1 May 2020 meeting search criteria. They included publications discussing remediation interventions either empirically or theoretically, pertaining to physicians or physician trainees of any level. Two independent reviewers used a standardised data extraction form to report descriptive information; they reviewed included records for the presence of mentions of emotion, described the mentions and analysed results thematically.

RESULTS

Of 1644 records, 199 met inclusion criteria and were reviewed in full. Of those, 112 (56%) mentioned emotion in some way; others focused solely on cognitive aspects of remediation. The mentions of emotion fell into three themes based on when the emotion was cited as present: during regular coursework or practice, upon referral for remediation and during remediation. One-quarter of records (50) indicated potential intentional incorporation of emotion into remediation programme design, but they were non-specific as to how emotions related to the learning process itself.

CONCLUSION

Even though emotion is omnipresent in remediation, medical educators frequently do not factor emotion into the design of remediation approaches and rarely explicitly utilise emotion to improve the learning process. Applications from other fields may help medical educators leverage emotion to improve learning in remediation, including strategies to frame and design remediation.

Supporting students with academic difficulties

INTRODUCTION

Up 15% of health professions students experience academic difficulties. We describe an early tracking system that effectively identifies and supports students with performance deficiencies. The objectives of this study were to discuss changes made in an academic success system at our new college of pharmacy, describe effectiveness of the updated program after one-year of use, and present changes that are being discussed based on the faculty and student feedback.

METHODS

We conducted in-person faculty interviews using open-ended questions to evaluate the effectiveness of our original program for supporting students with academic difficulties called the Early Tracking System (ETS). We then developed a new program titled the Individualized Plan for Academic Success System (IPASS). We conducted more faculty interviews and a student survey after the implementation of IPASS to evaluate its effectiveness.

RESULTS

In the 2018-2019 academic year at our institution, IPASS was activated 291 times, of which there were 27 course failures. The results of our study indicated a pass rate of 90.72% post-IPASS for all pharmacy students.

CONCLUSIONS

The goal of this study was to revise and implement an early tracking system that all faculty members, students, and administrators can utilize efficiently and effectively. This study demonstrated that students with academic difficulties have the potential for progression if their deficiencies are detected early. We described a unique system by which we addressed the needs of the at-risk students. The interview of faculty and survey of students indicated satisfaction with IPASS.

Role of formative assessment in predicting academic success among GP registrars: a retrospective longitudinal study

OBJECTIVES

The James Cook University General Practice Training (JCU GPT) programme's internal formative exams were compared with the Royal Australian College of General Practitioners (RACGP) pre-entry exams to determine ability to predict final performance in the RACGP fellowship exams.

DESIGN

A retrospective longitudinal study.

SETTING

General Practice (GP) trainees enrolled between 2016 and 2019 at a Registered Training Organisation in regional Queensland, Australia.

PARTICIPANTS

376 GP trainees enrolled in the training programme.

EXPOSURE MEASURES

The pre-entry exams were Multiple-Mini Interviews (MMI), Situational Judgement Test (SJT) and Candidate Assessment and Applied Knowledge Test. The internal formative exams comprised multiple choice questions (MCQ1 and MCQ2), short answer questions, clinical skills and clinical reasoning.

PRIMARY OUTCOME MEASURE

The college exams were Applied Knowledge Test (AKT), Key Feature Problems (KFP) and Objective Structured Clinical Examination (OSCE).

RESULTS

Correlations (r), coefficients of determination (R²) and OR were used as parameters for estimating strength of relationship and precision of predictive accuracy. SJT and MMI were moderately (r=0.13 to 0.31) and MCQ1 and MCQ2 highly (r=0.37 to 0.53) correlated with all college exams (p<0.05 to p<0.01), with R² ranging from 0.070 to 0.376. MCQ1 was predictive of failure in all college exams (AKT: OR=2.32, KFP: OR=3.99; OSCE: OR=3.46); while MCQ2 predicted failure in AKT (OR=2.83) and KFP (OR=3.15).

CONCLUSION

We conclude that the internal MCQ formative exams predict performance in the RACGP fellowship exams. We propose that our formative assessment tools could be used as academic markers for early identification of potentially struggling trainees.

Guidelines: The dos, don'ts and don't knows of remediation in medical education

INTRODUCTION

Two developing forces have achieved prominence in medical education: the advent of competency-based assessments and a growing commitment to expand access to medicine for a broader range of learners with a wider array of preparation. Remediation is intended to support all learners to achieve sufficient competence. Therefore, it is timely to provide practical guidelines for remediation in medical education that clarify best practices, practices to avoid, and areas requiring further research, in order to guide work with both individual struggling learners and development of training program policies.

METHODS

Collectively, we generated an initial list of Do's, Don'ts, and Don't Knows for remediation in medical education, which was then iteratively refined through discussions and additional evidence-gathering. The final guidelines were then graded for the strength of the evidence by consensus.

RESULTS

We present 26 guidelines: two groupings of Do's (systems-level interventions and recommendations for individual learners), along with short lists of Don'ts and Don't Knows, and our interpretation of the strength of current evidence for each guideline.

CONCLUSIONS

Remediation is a high-stakes, highly complex process involving learners, faculty, systems, and societal factors. Our synthesis resulted in a list of guidelines that summarize the current state of educational theory and empirical evidence that can improve remediation processes at individual and institutional levels. Important unanswered questions remain; ongoing research can further improve remediation practices to ensure the appropriate support for learners, institutions, and society.

Interventions for undergraduate and postgraduate medical learners with academic difficulties: A BEME systematic review: BEME Guide No. 56

Background: Clinical teachers often struggle to report unsatisfactory trainee performance, partly because of a lack of evidence-based remediation options. Objectives: To identify interventions for undergraduate (UG) and postgraduate (PG) medical learners experiencing academic difficulties, link them to a theory-based framework and provide literature-based recommendations around their use. Methods: This systematic review searched MEDLINE, CINAHL, EMBASE, ERIC, Education Source and PsycINFO (1990-2016) combining these concepts: medical education, professional competence/difficulty and educational support. Original research/innovation reports describing intervention(s) for UG/PG medical learners with academic difficulties were included. Data extraction employed Michie's Behavior Change Techniques (BCT) Taxonomy and program evaluation models from Stufflebeam and Kirkpatrick. Quality appraisal used the Mixed Methods Appraisal Tool (MMAT). The authors synthesized extracted evidence by adapting the GRADE approach to formulate recommendations. Results: Sixty-eight articles met the inclusion criteria, most commonly addressing knowledge (66.2%), skills (53.9%) and attitudinal problems (26.2%), or learner personal issues (41.5%). The most common BCTs were Shaping knowledge, Feedback/monitoring, and Repetition/substitution. Quality appraisal was variable (MMAT 0-100%). A thematic content analysis identified 109 interventions (UG: n = 84, PG: n = 58), providing 24 strong, 48 moderate, 26 weak and 11 very weak recommendations. Conclusion: This review provides a repertoire of literature-based interventions for teaching/learning, faculty development, and research purposes.

Predictors of academic performance in the discipline specific bioscience paper: A retrospective quantitative study [corrected]

INTRODUCTION

The cohort of students enrolled in the discipline-specific bioscience paper reflects a structural diversity in that it includes students of multiple ethnicities, varied age groups, differing scholastic and life experiences. These divergent identities of students are known to influence academic performance. The purpose of this retrospective quantitative study was to determine the ability of a set of variables such as age, gender, ethnicity, level of prior education, the place from which prior education was obtained, work experience and prior academic achievement to predict academic performance in the discipline-specific bioscience paper.

METHOD

The sample for this study was a purposive sample of all oral health students who had enrolled in the paper at the Auckland University of Technology from 2011 to 2014. The desensitised empirical data of 116 students from the University's database were subject to multivariable regression analysis. Pearson's correlation coefficients were calculated.

RESULTS

Prior academic achievement was a statistically significant predictor variable (P < 0.001) for the academic performance in the discipline-specific bioscience paper and was also positively correlated (r = 0.641, P < 0.001) to the grades in the discipline-specific bioscience paper.

CONCLUSION

Prior academic achievement was the only variable that was demonstrated to be correlated to and predictive of the academic performance in the discipline-specific bioscience paper.

Learning support interventions for Year 1 medical students: a review of the literature

CONTEXT

The journey through medical school can be challenging, especially for undergraduate medical students who must deal with a demanding curriculum, coupled with the demands of transitioning into adulthood. Despite experiencing learning challenges, most students succeed with appropriate learning support. Many medical schools offer learning support programmes, particularly in the latter years, but it has been suggested that such support could be more beneficial, especially during the initial years.

OBJECTIVES

This review explores learning support intervention programmes used to address learning challenges and deficits in the first year of medical school. Additionally, we propose a potential framework for supporting learning during the first year of medical school.

METHODS

We searched PubMed, Web of Science, MEDLINE, CINAHL (Cumulative Index to Nursing and Allied Health Literature), ERIC (Educational Resources Information Centre), Academic Search Premier and Google Scholar using the search terms 'learning support', 'learning challenge', 'remediation', 'change', 'medical education' and 'first year'. We developed and used a review matrix to record the main elements of each article. We also coded the matrix to identify emerging themes.

RESULTS

The main themes that emerged from the study were 'intervention approaches', 'area of intervention', 'intervention strategies', 'intervention dose' and 'intervention outcomes'.

INTERVENTIONS

(i) used proactive-deficit, reactive-deficit and proactive-developmental approaches; (ii) addressed content knowledge, academic success skills, personal and professional skills and programme-related elements; (iii) utilised faculty staff-facilitated, peer-facilitated, support staff-facilitated, experiential placement, self-study and reduced-load strategies; (iv) varied in length from 5 weeks to 2 years, and (v) generally showed positive results.

CONCLUSIONS

This review has identified the main components of learning support interventions used for Year 1 medical students. Interventions, however, are generally not grounded on empirical assessment that elucidates the nature of the challenges faced by students. Future research should provide empirical understanding of the learning challenges to be addressed.

Situating Remediation: Accommodating Success and Failure in Medical Education Systems

There has been a widespread shift to competency-based medical education (CBME) in the United States and Canada. Much of the CBME discourse has focused on the successful learner, with relatively little attention paid to what happens in CBME systems when learners stumble or fail. Emerging issues, such as the well-documented problem of "failure to fail" and concerns about litigious learners, have highlighted a need for well-defined and integrated frameworks to support and guide strategic approaches to the remediation of struggling medical learners.This Perspective sets out a conceptual review of current practices and an argument for a holistic approach to remediation in the context of their parent medical education systems. The authors propose parameters for integrating remediation into CBME and describe a model based on five zones of practice along with the rules of engagement associated with each zone. The zones are "normal" curriculum, corrective action, remediation, probation, and exclusion.The authors argue that, by linking and integrating theory and practice in remediation with CBME, a more integrated systems-level response to differing degrees of learner difficulty and failure can be developed. The proposed model demonstrates how educational practice in different zones is based on different rules, roles, responsibilities, and thresholds for moving between zones. A model such as this can help medical educators and medical education leaders take a more integrated approach to learners' failures as well as their successes by being more explicit about the rules of engagement that apply in different circumstances across the competency continuum.

Lessons learned from 15 years of non-grades-based selection for medical school

CONTEXT

Thirty years ago, it was suggested in the Edinburgh Declaration that medical school applicants should be selected not only on academic, but also on non-academic, attributes. The main rationale behind extending medical school selection procedures with the evaluation of (non-academic) personal qualities is that this will lead to the selection of students who will perform better as a doctor than those who are selected on the basis of academic measures only. A second rationale is the expectation that this will lead to a representative health workforce as a result of reduced adverse impact. The aims of this paper are (i) to describe what can be learned about the use of selection criteria other than grades from over 15 years of Dutch experience and (ii) to summarise current knowledge on the issue of adverse impact in relation to non-grades-based selection.

METHODS

A narrative review was undertaken of the (published) evidence that has resulted from non-grades-based school-specific selection procedures in the Netherlands and from recent explorations of the effect of the use of non-grades-based selection criteria on student diversity.

RESULTS

The Dutch evidence is grouped into five key themes: the effect of participation in voluntary selection procedures, the assessment of pre-university extracurricular activities, the use of work samples, Dutch experiences with situational judgement tests and the effects of changing circumstances. This is followed by several lessons learned for medical schools that aim to increase their student diversity.

CONCLUSION

Over the last 30 years, important steps towards reliable and valid methods for measuring non-academic abilities have been taken. The current paper describes several lessons that can be learned from the steps taken in the Dutch context. The importance of sharing evidence gathered around the globe and building on this evidence to reach our goal of predicting who will be a good doctor is acknowledged.

The Four-Tier Continuum of Academic and Behavioral Support (4T-CABS) Model: An Integrated Model for Medical Student Success

Not all students cope successfully with the demands of medical school, and students' struggles may result in study delay or dropout. To prevent these outcomes, medical schools need to identify students who are experiencing academic difficul ties and provide them with timely interventions through access to support programs. Although the importance of early identification and intervention is well recognized, less is known about successful strategies for identifying and supporting struggling students.Building on the literature and their own empirical findings, the authors propose an integrated, school-wide model for medical student success comprising a continuum of academic and behavioral support. This Four-Tier Continuum of Academic and Behavioral Support (4T-CABS) model focuses on improving both academic and behavioral outcomes by offering support for students at four levels, which range from adequate instruction for all, to targeted small-group interventions, to individualized support, and also include exit support for students who might be better off in another degree program. Additionally, medical schools should provide both academic and behavioral support; set high, yet realistic expectations and clearly communicate these to students; and intervene early, which requires timely identification of at-risk students who would benefit from the different types and tiers of support. Finally, interventions should be evidence based and fit the needs of the identified groups of students. The authors argue that adopting the core principles of the 4T-CABS model will enable medical schools to maximize academic engagement and performance for all students.

Early predictors of need for remediation in the Australian general practice training program: a retrospective cohort study

Underperforming trainees requiring remediation may threaten patient safety and are challenging for vocational training programs. Decisions to institute remediation are high-stakes-remediation being resource-intensive and emotionally demanding on trainees. Detection of underperformance requiring remediation is particularly problematic in general (family) practice. We sought to establish early-training assessment instruments predictive of general practice (GP) trainees' subsequently requiring formal remediation. We conducted a retrospective cohort study of trainees from a large Australian regionally-based GP training organization. The outcome factor was requirement for formal remediation. Independent variables were demographic factors and a range of formative assessments conducted immediately prior to or during early-stage training. Analyses employed univariate and multivariate logistic regression of each predictor assessment modality with the outcome, adjusting for potential confounders. Of 248 trainees, 26 (10.5 %) required formal remediation. Performance on the Colleague Feedback Evaluation Tool (entailing feedback from a trainee's clinical colleagues on clinical performance, communication and probity) and External Clinical Teaching Visits (half-day sessions of the trainee's clinical consultations observed directly by an experienced GP), along with non-Australian primary medical qualification, were significantly associated with requiring remediation. There was a non-significant trend for association with performance on the Doctors Interpersonal Skills Questionnaire (patient feedback on interpersonal elements of the consultation). There were no significant associations with entry-selection scores or formative exam or assessment scores. Our finding that 'in vivo' assessments of complex behaviour, but not 'in vitro' knowledge-based assessments, predict need for remediation is consistent with theoretical understanding of the nature of remediation decision-making and should inform remediation practice in GP vocational training.

Academic Performance on First-Year Medical School Exams: How Well Does It Predict Later Performance on Knowledge-Based and Clinical Assessments?

Number of appearances in the bottom quartile of 1st-year medical school exams were used to represent the extent to which students were having academic difficulties. Medical educators have long expressed a desire to have indicators of medical student performance that have strong predictive validity. Predictors traditionally used fell into 4 general categories: demographic (e.g., gender), other background factors (e.g., college major), performance/aptitude (e.g., medical college admission test scores), and noncognitive factors (e.g., curiosity). These factors, however, have an inconsistent record of predicting student performance. In comparison to traditional predictive factors, we sought to determine the extent to which academic performance in the 1st-year of medical school, as measured by examination performance in the bottom quartile of the class in 7 required courses, predicted later performance on a variety of assessments, both knowledge based (e.g., United States Medical Licensing Examination Step 1 and Step IICK) and clinical skills based (e.g., clerkship grades and objective structured clinical exam performance). Of all predictors measured, number of appearances in the bottom quartile in Year 1 was the most strongly related to performance in knowledge-based assessments, as well as clinically related outcomes, and, for each outcome, bottom-quartile performance accounted for additional variance beyond that of the traditional predictors. Low academic performance in the 1st year of medical school is a meaningful risk factor with both predictive validity and predictive utility for low performance later in medical school. The question remains as to how we can incorporate this indicator into a system of formative assessment that effectively addresses the challenges of medical students once they have been identified.

Predicting medical students who will have difficulty during their clinical training

BACKGROUND

We aimed to classify the difficulties students had passing their clinical attachments, and explore factors which might predict these problems.

METHODS

We analysed data from regular student progress meetings 2008-2012. Problem categories were: medical knowledge, professional behaviour and clinical skills. For each category we then undertook a predictive risk analysis.

RESULTS

Out of 561 students, 203 were found to have one or more problem category and so were defined as having difficulties. Prevalences of the categories were: clinical skills (67%), knowledge (59%) and professional behaviour (29%). A higher risk for all categories was associated with: male gender, international entry and failure in the first half of the course, but not with any of the minority ethnic groups. Professional and clinical skills problems were associated with lower marks in the Undergraduate Medical Admissions Test paper 2. Clinical skills problems were less likely in graduate students.

CONCLUSIONS

In our students, difficulty with clinical skills was just as prevalent as medical knowledge deficit. International entry students were at highest risk for clinical skills problems probably because they were not selected by our usual criteria and had shorter time to become acculturated.

The validity of student tutors' judgments in early detection of struggling in medical school. A prospective cohort study

Early identification and support of strugglers in medical education is generally recommended in the research literature, though very little evidence of the diagnostic qualities of early teacher judgments in medical education currently exists. The aim of this study was to examine the validity of early diagnosis of struggling in medical school based on informal teacher judgements of in-class behavior. The study design was a prospective cohort study and the outcomes/truth criteria were anatomy failure and medical school drop out. Six weeks into an anatomy course, student tutors attempted to identify medical students, who they reckoned would fail the anatomy course or drop out, based on their everyday experiences with students in a large group educational setting. In addition, they were asked to describe the indicators of struggling they observed. Sixteen student tutors evaluated 429 medical students for signs of struggling. By week six, the student tutors were able to detect approximately 1/4-1/3 of the students who eventually failed or dropped out, and for ¾ of the strugglers they identified, they were correct in their judgments. Informal student tutor's judgements showed incremental validity for both outcomes when controlling for grades obtained in preceeding exams. Lack of participation, lack of commitment, poor academic performance, poor social interactions and general signs of distress were the main indicators of struggling identified. Teachers' informal judgements of in-class behavior may be an untapped source of information in the early identification of struggling medical students with added value above and beyond formal testing.

How well do final year undergraduate medical students master practical clinical skills?

INTRODUCTION

The clinical examination and other practical clinical skills are fundamental to guide diagnosis and therapy. The teaching of such practical skills has gained significance through legislative changes and adjustments of the curricula of medical schools in Germany. We sought to find out how well final year undergraduate medical students master practical clinical skills.

METHODS

We conducted a formative 4-station objective structured clinical examination (OSCE) focused on practical clinical skills during the final year of undergraduate medical education. Participation was voluntary. Besides the examination of heart, lungs, abdomen, vascular system, lymphatic system as well as the neurological, endocrinological or orthopaedic examination we assessed other basic clinical skills (e.g. interpretation of an ECG, reading a chest X-ray). Participants filled-out a questionnaire prior to the exam, inter alia to give an estimate of their performance.

RESULTS

214 final year students participated in our study and achieved a mean score of 72.8% of the total score obtainable. 9.3% of participants (n=20) scored insufficiently (<60%). We found no influence of sex, prior training in healthcare or place of study on performance. Only one third of the students correctly estimated their performance (35.3%), whereas 30.0% and 18.8% over-estimated their performance by 10% and 20% respectively.

DISCUSSION

Final year undergraduate medical students demonstrate considerable deficits performing practical clinical skills in the context of a formative assessment. Half of the students over-estimate their own performance. We recommend an institutionalised and frequent assessment of practical clinical skills during undergraduate medical education, especially in the final year.

Critical review: medical students' motivation after failure

About 10 % of students in each years' entrants to medical school will encounter academic failure at some stage in their programme. The usual approach to supporting these students is to offer them short term remedial study programmes that often enhance approaches to study that are orientated towards avoiding failure. In this critical review I will summarise the current theories about student motivation that are most relevant to this group of students and describe how they are enhanced or not by various contextual factors that medical students experience during their programme. I will conclude by suggesting ways in which support programmes for students who have encountered academic failure might be better designed and researched in the future.

Who is repeating anatomy? Trends in an undergraduate anatomy course

Anatomy courses frequently serve as prerequisites or requirements for health sciences programs. Due to the challenging nature of anatomy, each semester there are students remediating the course (enrolled in the course for a second time), attempting to earn a grade competitive for admissions into a program of study. In this retrospective study, remediation rates and trends in an undergraduate anatomy course with over 400 students enrolled each semester at a large Midwestern university were identified. Demographic data was collected from spring 2004 to spring 2010, including students' age, ethnicity, major of study, class standing, college admission tests (ACT and SAT®) scores, anatomy laboratory and lecture examination scores, and final anatomy grades for each semester. Eleven percent of the students repeated the course at least once. Gender, ethnicity, major of study and SAT scores were all shown to be associated with whether or not a student would need to repeat the course. On average, students who repeated anatomy demonstrated significant improvements in lecture and laboratory scores when comparing first and second enrollments in anatomy, and therefore also saw improved final course grades in their second enrollment. These findings will aid future instructors to identify and assist at-risk students to succeed in anatomy. Instructors from other institutions may also find the results to be useful for identifying students at risk for struggling.

Predicting performance: relative importance of students' background and past performance

CONTEXT

Despite evidence for the predictive value of both pre-admission characteristics and past performance at medical school, their relative contribution to predicting medical school performance has not been thoroughly investigated.

OBJECTIVES

This study was designed to determine the relative importance of pre-admission characteristics and past performance in medical school in predicting student performance in pre-clinical and clinical training.

METHODS

This longitudinal prospective study followed six cohorts of students admitted to a Dutch, 6-year, undergraduate medical course during 2002-2007 (n = 2357). Four prediction models were developed using multivariate logistic regression analysis. Main outcome measures were 'Year 1 course completion within 1 year' (models 1a, 1b), 'Pre-clinical course completion within 4 years' (model 2) and 'Achievement of at least three of five clerkship grades of ≥ 8.0' (model 3). Pre-admission characteristics (models 1a, 1b, 2, 3) and past performance at medical school (models 1b, 2, 3) were included as predictor variables.

RESULTS

In model 1a - including pre-admission characteristics only - the strongest predictor for Year 1 course completion was pre-university grade point average (GPA). Success factors were 'selected by admission testing' and 'age > 21 years'; risk factors were 'Surinamese/Antillean background', 'foreign pre-university degree', 'doctor parent' and male gender. In model 1b, number of attempts and GPA at 4 months were the strongest predictors for Year 1 course completion, and male gender remained a risk factor. Year 1 GPA was the strongest predictor for pre-clinical course completion, whereas being male or aged 19-21 years were risk factors. Pre-clinical course GPA positively predicted clinical performance, whereas being non-Dutch or a first-generation university student were important risk factors for lower clinical grades. Nagelkerke's R(2) ranged from 0.16 to 0.62.

CONCLUSIONS

This study not only confirms the importance of past performance as a predictor of future performance in pre-clinical training, but also reveals the importance of a student's background as a predictor in clinical training. These findings have important practical implications for selection and support during medical school.

Implementation of a study skills program for entering at-risk medical students

While the first year of medical school is challenging for all students, there may be specific issues for students from rural areas, economically disadvantaged backgrounds, ethnic minorities, or nontraditional age groups. A Summer Prematriculation Program (SPP) was created to prepare entering at-risk students for the demands of medical school. For the past 2 yr, an emphasis was placed on the development of appropriate study plans and skills. On presurveys, students predicted an increase in their number of study hours per lecture hour, from 7.6 h in undergraduate coursework to 9.1 h in medical school coursework (n = 35). These study plans were infeasible given the rigorous didactic lecture schedule in medical school. Interventions were made through lectures on study plans and modeling of appropriate study habits using engaging lectures in the SPP physiology course. At the end of the program, a postsurvey was given, and students reported a reduction in the planned hours of study to a more realistic 3.9 h of study time per hour of lecture. Furthermore, students planned to decrease their use of textbooks while increasing their use of concept mapping, videos, and peer teaching. The majority of students completing the SPP program with a study skills emphasis performed well in the Medical Physiology course, with 4 students honoring in the course, 27 students passing, and 2 students remediating the course after an initial failure. These results indicate that at-risk medical students may have inappropriate study plans that can be improved through participation in a program that emphasizes study skills development.