Struggling with strugglers: using data from selection tools for early identification of medical students at risk of failure

Relationship between admission selection tools and student attrition in the early years of medical school

Objectives

Placement in medical schools is highly sought after worldwide with fierce competition among applicants. However, some of the best students withdraw after being accepted to medical school. The aim of this study was to investigate early student attrition within the first 2 years of medical school and determine its relationship to admission selection tools.

Methods

Quantitative research was conducted at the College of Medicine and Health Sciences from 2016 until 2020, during which time routine admission data and students' examination results for the first 2 years were collected and analyzed.

Results

The attrition rate during the study period was 31.7%. High school and college written examination scores were significantly related to completing the premedical program (p = 0.001 and p = 0.002, respectively). Female students scored significantly higher in multiple mini interviews (MMIs) compared with male counterparts (p < 0.001). However, the difference in MMI score was not related to student attrition (p = 0.148).

Conclusion

The cause of early attrition is complex and cannot be attributed to a single factor.Undergraduate high school score and written admission examination results were statistically significant factors in relation to student attrition rate and low academic performance. The results of this study showed that the female students scored significantly higher in the multiple MMI tests compared to male students. However, MMI score alone was not significantly related to student attrition.

Predicting students' academic progress and related attributes in first-year medical students: an analysis with artificial neural networks and Naïve Bayes

BACKGROUND

Dropout and poor academic performance are persistent problems in medical schools in emerging economies. Identifying at-risk students early and knowing the factors that contribute to their success would be useful for designing educational interventions. Educational Data Mining (EDM) methods can identify students at risk of poor academic progress and dropping out. The main goal of this study was to use machine learning models, Artificial Neural Networks (ANN) and Naïve Bayes (NB), to identify first year medical students that succeed academically, using sociodemographic data and academic history.

METHODS

Data from seven cohorts (2011 to 2017) of admitted medical students to the National Autonomous University of Mexico (UNAM) Faculty of Medicine in Mexico City were analysed. Data from 7,976 students (2011 to 2017 cohorts) of the program were included. Information from admission diagnostic exam results, academic history, sociodemographic characteristics and family environment was used. The main dataset included 48 variables. The study followed the general knowledge discovery process: pre-processing, data analysis, and validation. Artificial Neural Networks (ANN) and Naïve Bayes (NB) models were used for data mining analysis.

RESULTS

ANNs models had slightly better performance in accuracy, sensitivity, and specificity. Both models had better sensitivity when classifying regular students and better specificity when classifying irregular students. Of the 25 variables with highest predictive value in the Naïve Bayes model, percentage of correct answers in the diagnostic exam was the best variable.

CONCLUSIONS

Both ANN and Naïve Bayes methods can be useful for predicting medical students' academic achievement in an undergraduate program, based on information of their prior knowledge and socio-demographic factors. Although ANN offered slightly superior results, Naïve Bayes made it possible to obtain an in-depth analysis of how the different variables influenced the model. The use of educational data mining techniques and machine learning classification techniques have potential in medical education.

Stress, Grit, Satisfaction With Life, and Remediation of Prelicensure Nursing Students

BACKGROUND

Stress and poor academic performance often lead to high levels of nursing school attrition.

PURPOSE

To explore nursing students' perceived stress, grit, and satisfaction with life, and assess the effectiveness of a remediation program for students at risk for poor academic performance.

METHODS

Perceived stress, grit, and satisfaction with life were measured, and the effect of remediation given to at-risk students was studied. Examination and psychometric scores were compared between remediation and nonremediation groups.

RESULTS

The remediation group had higher levels of perceived stress and a lower satisfaction with life compared with the rest of the cohort. Remediation significantly improved examination scores of at-risk students although scores remained lower than those among nonremediation students.

CONCLUSIONS

Perceived stress and grit scores were high in nursing students, and satisfaction varied among age groups. Remediation based on metacognitive theory significantly improved at-risk students' examination scores.

The validity of Engagement and Feedback Assessments (EFAs): identifying students at risk of failing

BACKGROUND

Imperial College School of Medicine, London UK, introduced a new curriculum in 2019, with a focus on the GMC outcomes for graduates, and pedagogy best practice. The new curriculum included formative assessments, named engagement and feedback assessments (EFAs), to support learning, and attainment in the summative examinations. The aims of this study were to assess the validity of EFAs and to determine whether they have utility as a modified form of programmatic assessment to inform decision-making regarding possible interventions by measuring and analysing attendance at and performance in these formative events.

METHODS

Seven hundred and sixty-one students were included in the study and assessment results were included for academic years 2019/20 to 2020/21. Forty-one data points per student, (27 in Year 1 and 14 in Year 2) were used, to compare EFA scores with the summative performance. Attendance was monitored through engagement with the EFAs.

RESULTS

Cohort 1 (enrolled 2019): In year 1, EFAs were associated with summative exam scores (overall r = 0.63, p < 0.001). Year 2, EFA scores were also associated with summative scores (overall r = 0.57, p < 0.001), including the clinical practical assessment (r = 0.45, p < 0.001). Missing two or more EFAs was associated with a significant increase in the likelihood of failing one or more summative examinations in the first year (OR: 7.97, 95% CI 2.65-34.39) and second year (OR: 3.20, 95% CI 1.74-5.95). Missing more than two EFAs in their first year was also associated with a higher risk of failing a summative examination in the second year (OR: 2.47, 95% CI 1.33-4.71). Students who increased their attendance between year 1 and 2 fared better in summative assessment than those who maintained poor attendance, whereas those that reduced their attendance fared worse than those that maintained high attendance. Cohort 2 (enrolled 2020): Analysis of cohort 2 supported these findings and in this cohort missing two or more EFAs was again associated with an increased likelihood of failing a summative examination (OR = 4.00, 95% CI = 2.02-7.90).

CONCLUSION

Our EFA model has validity in predicting performance in summative assessments and can inform prospective interventions to support students' learning. Enhancing attendance and engagement can improve outcomes.

Prompt identification of struggling candidates in near peer-led basic life support training: piloting an online performance scoring system

BACKGROUND

Bristol Medical School has adopted a near peer-led teaching approach to deliver Basic Life Support training to first year undergraduate medical students. Challenges arose when trying to identify early in the course which candidates were struggling with their learning, in sessions delivered to large cohorts. We developed and piloted a novel, online performance scoring system to better track and highlight candidate progress.

METHODS

During this pilot, a 10-point scale was used to evaluate candidate performance at six time-points during their training. The scores were collated and entered on an anonymised secure spreadsheet, which was conditionally formatted to provide a visual representation of the score. A One-Way ANOVA was performed on the scores and trends analysed during each course to review candidate trajectory. Descriptive statistics were assessed. Values are presented as mean scores with standard deviation (x̄±SD).

RESULTS

A significant linear trend was demonstrated (P < 0.001) for the progression of candidates over the course. The average session score increased from 4.61 ± 1.78 at the start to 7.92 ± 1.22 at the end of the final session. A threshold of less than 1SD below the mean was used to identify struggling candidates at any of the six given timepoints. This threshold enabled efficient highlighting of struggling candidates in real time.

CONCLUSIONS

Although the system will be subject to further validation, our pilot has shown the use of a simple 10-point scoring system in combination with a visual representation of performance helps to identify struggling candidates earlier across large cohorts of students undertaking skills training such as Basic Life Support. This early identification enables effective and efficient remedial support.

Improving Learning and Study Strategies in Undergraduate Medical Students: A Pre-Post Study

We aimed to describe the impact of a structured interventional program to improve learning and study skills in undergraduate medical students from a Latin American medical school. The interventional program's design was based on diagnostic/prescriptive assessment test scores measuring ten scales. The program consisted of five tailored workshops. The cohort studied consisted of 81 third-year medical students. The outcome variable was the difference between "pre" and "post" test scores. The unadjusted score percentiles were used to compare improvement in learning and study skills. In addition, a sensitivity analysis was conducted to assess variation in the mean difference of the test scores by the number of workshops attended. The response rate was 100% (81/81) for the pre test and ~77% (62/81) for the post test. After the interventional program, nine out of ten scales showed statistical improvement, except for the scale of motivation. The scales with the highest and lowest percent change improvement were time management (66%, p-value: <0.001) and motivation (14.9%, p-value: 0.06). The students who attended more workshops obtained a higher percent change improvement in the post test. These findings suggest that through a well-designed interventional program, it is possible to improve learning and study skills among medical students.

The effect of face-to-face versus online learning on student performance in anatomy: an observational study using a causal inference approach

Introduction

This study aimed to estimate the causal effect of face-to-face learning on student performance in anatomy, compared to online learning, by analysing examination marks under a causal structure.

Methods

We specified a causal graph to indicate how the mode of learning affected student performance. We sampled purposively to obtain end-semester examination marks of undergraduate and postgraduate students who learned using face-to-face (pre-COVID, 2019) or online modes (post-COVID, 2020). The analysis was informed by the causal graph. Marks were compared using linear regression, and sensitivity analyses were conducted to assess if effects were robust to unmeasured confounding.

Results

On average, face-to-face learning improved student performance in the end-semester examination in undergraduate students (gain of mean 8.3%, 95% CI 3.3 to 13.4%; E-value 2.77, lower limit of 95% CI 1.80) but lowered performance in postgraduate students (loss of 8.1%, 95% CI 3.6 to 12.6%; E-value 2.89, lower limit of 95% CI 1.88), compared to online learning.

Discussion

Under the assumed causal graph, we found that compared to online learning, face-to-face learning improved student performance in the end-semester examination in undergraduate students, but worsened student performance in postgraduate students. These findings suggest that different modes of learning may suit different types of students. Importantly, this is the first attempt to estimate causal effects of the mode of learning on student performance under a causal structure. This approach makes our assumptions transparent, informs data analysis, and is recommended when using observational data to make causal inferences.

Factors leading to the academic failure of undergraduate medical students - Predict early to prevent

OBJECTIVES

The main objective of the study was to explore the factors to predict academic failure before the first major assessment.

METHODS

An exploratory qualitative study was conducted from March 2021 to August 2021 at Riphah International University. Using the purposive sampling technique, 16 students and seven teachers were included in the study. Eight online interviews were conducted with students who were academic failures and two focus group discussions were held with eight high achievers and seven teachers. The data was analyzed and thematic analysis was done.

RESULTS

Thematic analyses deduced relevant themes which were: Educational Journey Does past matter? Essential for learning, Personality and psychological make-up, and assessment of behaviour. The factors identified were performance record, learning difficulty, educational dislocation, missionless and purposeless, against free will, tuition, poor self-regulation, low effort, procrastination, non-reflective practice, cognitive load mismanagement, limited remediation, hesitant help-seeking, low self-efficacy, introvert behaviour, demotivation, emotional imbalance, observation of student behaviour, assessment of assigned task.

CONCLUSION

Academic failure can be predicted early and salvaged.

Extending growth curves: a trajectory monitoring approach to identification and interventions in struggling medical student learners

Given gaps in both identifying and providing targeted interventions to struggling learners, the purpose of this study is to both improve rapid identification and to improve individualized academic advising for learners using this visual representation of performance. Across three graduating classes, individual growth curves were calculated for each student on National Board of Medical Examiners customized assessments during the pre-clerkship period using their deviation from the class average at each assessment point. These deviation scores were cumulatively summed over time and were regressed onto the sequence of exams. We analyzed the difference between the regression slopes of those students placed on Academic Probation (AP) versus not, as well as differences in slopes based on the timing of when a struggling learner was placed on AP to explore learner trajectory after identification. Students on AP had an average growth slope of - 6.06 compared to + 0.89 for those not on AP. Findings also suggested that students who were placed on AP early during pre-clerkship showed significant improvement (positive changes in trajectory) compared to students identified later in the curriculum. Our findings suggest that earlier academic probation and intervention with struggling learners may have a positive effect on academic trajectory. Future research can better explore how academic trajectory monitoring and performance review can be regularly used in advising sessions with students.

Assessment of dropout rates in the preclinical years and contributing factors: a study on one Thai medical school

BACKGROUND

The highest dropout rate for medical students is during the preclinical years of education. Several studies have reported possible contributing factors, however, studies regarding the dropout rate from medical education among Thai medical students and its associated factors are still lacking. This study aimed to identify the prevalence of dropout from medical education within the preclinical period and its associated factors among Thai medical students.

METHODS

We collected data from preclinical medical students who entered one medical school in Bangkok, Thailand, between 2017-2019. Demographic data, admission program, pre-admission mental health status assessed by the Thai Mental Health Indicator 66, mental health records from the faculty-based counseling services, and academic achievement were extracted. Data were reported in a descriptive fashion. We analyzed the association between dropout and these factors by robust Poisson regression.

RESULTS

In total, 914 students were recruited. Dropout was only 1.5%, which was lower than the previous studies. Regression analysis showed a significant association between dropout and mental health problems [Prevalence ratio (PR) 58.20, 95%CI 13.72- 246.95] but not admission program [PR 0.32, 95%CI 0.09 - 1.16] or failing examinations [PR 0.59, 95%CI 0.18 - 1.90]. However, other contributing factors such as financial status, family problems, medical illness, and students' motivation, were not evaluated in this study.

CONCLUSIONS

Mental health problems during medical education were associated with dropout after adjusting for other confounding variables. Further longitudinal studies are needed to identify the impacts of academic failure on dropout in higher clinical years.

Investigating the risk factors for academic difficulties in the medical programme at a South African university

BACKGROUND

The National Benchmark Test (NBT) that determines academic readiness is widely used by Faculties as an additional measure to select students for the study of medicine. Despite this, many students continue to experience academic challenges that culminate in delayed graduation and sometimes academic exclusion or discontinuation of studies.

AIM

This study aimed to understand academic and non-academic variables linked with academic difficulties in the first three years of medical education.

METHODS

The study sample consisted of six cohorts of medical students for the period 2011 to 2016 (n = 1392). Only the first three of the six-year medical programme were selected for analysis. Survival analysis and Cox Proportional Hazard (CPH) was used to identify academic and non-academic variables associated with academic difficulties.

RESULTS

A total of 475 students (34%) experienced academic difficulty; 221 (16%) in the first year of study, 192 (14%) in the second year and 62 (5%) in the third year of study. The results show that Intermediate Upper, Lower and Basic levels for all NBT domains, living in university residence, rurality and male gender were risk factors for academic difficulty.

CONCLUSION

In mitigating these factors, the NBT must inform the type of support programmes to augment the students' skills and promote academic success. Additionally, existing support programmes should be evaluated to ascertain if they reach students at risk and whether participating in these programmes yield positive academic outcomes.

Why do students struggle in their first year of medical school? A qualitative study of student voices

BACKGROUND

Struggling at medical school incorporates academic failure, course disruption and early course exit. Struggling is usually multi-factorial involving academic, personal, financial and health factors. Struggling students may fail to engage with available support. First year students are particularly susceptible as they transition to university and a professional career.

METHODS

The study aim was to explore medical students' own voices on struggling and assess how they match up to existing literature. During one academic year, all first year medical students at the University of Birmingham (UK) who opted to leave or were required to withdraw (n = 52) were asked to participate in an individual exit interview. Fifteen students responded and fourteen (27%) agreed to be interviewed. Interviews were face to face (n = 10), telephone (n = 3) and via email (n = 1). Interviews were unstructured and led by a general open question. Framework analysis identified key data themes.

RESULTS

Students described year one of medical school as a critical transition. They simultaneously needed to adapt to being a university student, a medical student and a doctor. A six-group typology of students emerged, each of which struggled with one or more of these adaptations. The groups were: wrong degree choice, mental health problems, acute crisis, at capacity, slow starter and family rock. Some students experienced an isolated problem from within this typology. Most had a multi-factorial story of struggling. Mental health problems and acute crises were the most common issues. Early professional identity formation was a key hurdle. Help-seeking behaviours were varied.

CONCLUSIONS

This study explores the narratives of medical students who struggled from an early stage and presents a data-driven typology of their issues. It advances existing qualitative understanding of this topic, which to date is predominantly derived from educator perceptions and not specific to early course issues. Although our results broadly cohere with existing knowledge, we also present novel findings which may reflect our focus on first year students. Issues around early professional identity formation may reflect the increasing emphasis on professionalism in medical school curricula. Listening to these narratives could help university staff to identify students at risk of struggling for targeted support.

Mental health among Thai medical students: Preadmission evaluation and service utilization

BACKGROUND AND AIMS

Mental illness among medical students is common. Identifying at-risk students can be beneficial in terms of prevention and treatment. This study examined the association between preadmission mental health evaluation and mental health service utilization among Thai medical students.

METHODS

The authors collected data from all first- to sixth-year students at one university hospital and three affiliated hospitals during the 2014 to 2019 academic year (n = 1642). Preadmission mental health was evaluated using the Thai Mental Health Indicators 66 (TMHI-66) questionnaire. Data from mental health service utilization included the dates of service and the clinical diagnosis. Data were analyzed using Cox regression, adjusted for sex, hometown, and affiliated hospital.

RESULTS

The prevalence of poor mental health was found to be 3.7% based on TMHI-66 results. The proportion of all students who sought mental health services was 8.3%, and they were mostly diagnosed with depressive disorder (3%), adjustment disorder (1.9%), and anxiety disorder (1.9%). Students with positive TMHI-66 results were more likely to seek out services at the faculty counseling center for one or more mental health problems (Hazard ratio [HR] = 2.11, 95% CI 1.11-4.04, P = .024).

CONCLUSION

Preadmission mental health was found to be associated with mental health service utilization. Depressive disorder was the most frequent mental illness among Thai medical students utilizing the faculty counseling services. Still, the number was far lower than the reported prevalence of mental disorders due to various reasons. Further studies are needed to investigate risk and protective factors for mental disorders to better promote mental health and encourage mental health service utilization in targeted students throughout medical education.

Predicting failure before it happens: A 5-year, 1042 participant prospective study

PURPOSE OF THE ARTICLE

Students who fail assessments are at risk of negative consequences, including emotional distress and cessation of studies. Identifying students at risk of failure before they experience difficulties may considerably improve their outcomes.

METHODS

Using a prospective design, we collected simple measures of engagement (formative assessment scores, compliance with routine administrative tasks, and attendance) over the first 6 weeks of Year 1. These measures were combined to form an engagement score which was used to predict a summative examination sat 14 weeks after the start of medical school. The project was repeated for five cohorts, giving a total sample size of 1042.

RESULTS

Simple linear regression showed engagement predicted performance (R²_adj = 0.03, F(1,1040) = 90.09, p < 0.001) with a small effect size. More than half of failing students had an engagement score in the lowest two deciles.

CONCLUSIONS

At-risk medical students can be identified with some accuracy immediately after starting medical school using routinely collected, easily analysed data, allowing for tailored interventions to support students. The toolkit provided here can reproduce the predictive model in any equivalent educational context. Medical educationalists must evaluate how the advantages of early detection are balanced against the potential invasiveness of using student data.

Remediation and reassessment methods in pharmacy education: A systematic review

BACKGROUND

Colleges of pharmacy are currently required to implement a remediation program within their curricula, but no specifications are provided on the ideal methodology. While the need for successful remediation strategies continues to grow, literature describing positive or negative outcomes of different approaches is significantly lacking. The objective of this literature review was to describe and evaluate remediation methodologies in pharmacy education.

METHODS

This literature review was completed following PRISMA criteria. A search of the PubMed, Cochrane Library, Cumulative Index of Nursing and Allied Health, Academic Search Complete, PsycInfo, Scopus, and ProQuest Central databases was conducted in July 2019. Studies were included if they involved pharmacy student education and described either remediation or reassessment.

RESULTS

The evaluated studies discussed a range of course types being remediated, a large variety of remediation strategies and timeframes, and differing overall outcomes. No studies provided comparison of remediation techniques or provided details on the implementation of their chosen approaches. A consistent finding within the evaluated studies was the inclusion of prevention strategies to attempt to avoid the need for remediation preemptively. Overall outcomes for each remedial program were inconsistent and no clear patterns were evident other than an improvement in student performance following remediation.

IMPLICATIONS

Remediation strategies included course repetition, summer restudy, reassessment, and individualized plans. Outcomes varied significantly between studies, making methodology comparisons difficult. Future studies that include more detail and consistency in the reported outcomes would be beneficial to students and help clarify remediation for colleges of pharmacy.