Measuring differential attainment: a longitudinal analysis of assessment results for 1512 medical students at four Scottish medical schools

The Utility of Multiple-Choice Assessment in Current Medical Education: A Critical Review

In recent years, healthcare education providers have boasted about a conscious shift towards increasing clinical competence via assessment tests that promote more active learning. Despite this, multiple-choice questions remain amongst the most prevalent forms of assessment. Various literature justifies the use of multiple-choice testing by its high levels of validity and reliability. Education providers also benefit from requiring fewer resources and costs in the development of questions and easier adaptivity of questions to compensate for neurodiversity. However, when testing these (and other) variables via a structured approach in terms of their utility, it is elucidated that these advantages are largely dependent on the quality of the questions that are written, the level of clinical competence that is to be attained by learners and the impact of negating confounding variables such as differential attainment. Attempts at improving the utility of multiple-choice question testing in modern healthcare curricula are discussed in this review, as well as the impact of these modifications on performance.

Medical student experiences of equality, diversity, and inclusion: content analysis of student feedback using Bronfenbrenner's ecological systems theory

BACKGROUND

Issues relating to equality, diversity, and inclusion (EDI) significantly impact on medical student achievement and wellbeing. Interventions have been introduced at curricular and organisational levels, yet progress in addressing these issues remains limited. Timely evaluation is needed to assess effectiveness of interventions, and to explore issues and interactions in learning environments impacting on student experience. We introduced an anonymous question concerning students' experiences of EDI into routine online student feedback questionnaires, to scope the nature of ongoing issues and develop greater understanding of students' experiences in our programme environment. Ecological systems theory, which conceptualizes learning as a function of complex social interactions, determined by characteristics of individual learners and their environment, provides a framework for understanding.

METHODS

Free-text responses regarding experiences of EDI gathered over 20 months from all programme years (n = 760) were pooled for analysis, providing a holistic overview of experiences in the learning environment. A counting exercise identified broad categories reported by students. Content analysis of the qualitative dataset was undertaken. Bronfenbrenner's ecological systems theory was applied as a framework to demonstrate interdependencies between respondents' experiences and environments, and associated impacts.

RESULTS

Three hundred and seventy-six responses were received relating to wide-ranging EDI issues, most frequently gender or ethnicity. Responses mapped onto all areas of the ecological systems model, with frequent links between subsystems, indicating considerable complexity and interdependencies. Interpersonal interactions and associated impacts like exclusion were frequently discussed. Differential experiences of EDI-related issues in medical school compared to clinical settings were reported. Impacts of institutional leadership and wider societal norms were considered by respondents. Respondents discussed their need for awareness of EDI with reference to future professional practice.

CONCLUSIONS

Implementation of a regular free-text evaluation question allowed data-gathering across cohorts and throughout several stages of the curriculum, illuminating student experience. Connections established demonstrated intersectionality, and how environment and other factors interact, impacting on student experiences. Students experience EDI-related issues on multiple levels within the educational environment, with consequent impacts on learning. Any successful approach towards tackling issues and promoting equity of opportunity for all requires multi-level actions and widespread culture change. Students can offer fresh and distinct perspectives regarding change needed, to complement and diversify perspectives provided by staff and organisational leadership. Student voice should be enabled to shape change.

Is the awarding gap at UK medical schools influenced by ethnicity and medical school attended? A retrospective cohort study

OBJECTIVES

To better characterise the Awarding Gap (AG) between black, Asian and other minority ethnic (BAME) and white students in UK undergraduate medical education by examining how it affects eight minority ethnicity subgroups (Bangladeshi, black, Chinese, Indian, Pakistani, mixed, other Asian background and other ethnic background) and whether the AG varies by medical school attended.

DESIGN

Retrospective cohort study.

SETTING

Data extracted from the UK Medical Education Database on students enrolled at 33 UK medical schools in the academic years starting 2012, 2013 and 2014.

PARTICIPANTS

16 020 'Home' tuition fee status students who sat the University Clinical Aptitude Test on entry to university and obtained a UK Foundation Programme (UKFP) application score on exit.

PRIMARY OUTCOME MEASURE

UKFP Z-scores on exit from medical school.

RESULTS

There were significant differences in UKFP Z-scores between ethnicity subgroups. After white students, mixed ethnicity students performed best (coefficient -0.15 standard deviations [SD]) compared with white students, (95% confidence interval [CI] -0.23 to -0.08, p<0.001) and Pakistani students scored lowest (coefficient -0.53 SD, 95% CI -0.60 to -0.46, p<0.001). In pairwise comparisons of scores between all nine individual ethnicity subgroups, 15/36 were statistically significant. The AG varied considerably across medical schools. The largest gap showed the coefficient for BAME was -0.83 SD compared with white students (95% CI -1.18 to -0.49, p<0.001), while the smallest demonstrated no statistically significant difference in performance between BAME and white students (+0.05 SD, 95% CI -0.32 to 0.42, p=0.792).

CONCLUSIONS

BAME students are significantly disadvantaged by the current UK medical education system. There are clear differences in medical school outcomes between students from different ethnicity subgroups, and the size of the AG also varies by medical school attended. Urgent and effective action must be taken to address the AG and achieve an equal learning environment for our future doctors.

Tackling differential attainment in specialist GP training in England and Scotland

In this article NHS England and NHS Education for Scotland describe practical ways we are tackling differences in the attainment of people training as general practitioners (GPs).Trainees from minority ethnic groups and international medical graduates are less likely than others to qualify as GPs. It is difficult to change systemic inequalities, but over the past five years we have made practical changes to GP speciality training. Educators recognise there is an issue and are trying to tackle it.For example, people who had not successfully qualified had an opportunity to return to GP training. When we provided individualised targeted support, the proportion who completed training significantly increased (76%).This was a catalyst for reviewing unconscious bias in GP training. We implemented a national programme to tackle differential attainment and system-level bias. Educators now work with all GP trainees to identify their individual needs. Supervisors are trained to recognise bias and provide targeted support. There is mental health support and regular reviews to see whether trainees are ready to sit exams. Trainee representatives are championing the learner voice in national committees. Exams are being altered to reduce unconscious bias. We are monitoring attainment over time.The key message is that differential attainment should not be in the 'too hard basket'. The narrative is changing from 'can't do' to 'must do', supported by appropriate leadership, promotion and resourcing. There is much more to do, but we are making changes, evaluating and applying our learning. We have moved from talking to taking action.

Do medical students with a disability experience adverse educational outcomes on UK medical courses?

BACKGROUND

International data demonstrate that medical students with disabilities experience differential awarding. One cross-sectional study finds lower graduation rates and scores in written exams, with no impact on clinical exams. Disabled students graduated with lower decile scores. This quantitative, retrospective, longitudinal study explored the impact of disability on exam performance, course performance and course discontinuation.

METHOD

Anonymised data were obtained for 1743 students on a UK graduate medical programme from 2011 to present. Statistical tests, including t-tests and one-way ANOVA were conducted for main effects of demographic variables on exam results and categorical outcomes. Regression models established the effects of variables and sub-categories of variables on results and categorical outcomes.

RESULTS

Significant main effects of disability on exam scores were identified, as well as failure probability. Regressions showed significant differences in outcomes between different types of disability, with mental health conditions predicting course discontinuation. A significant amplifying effect was found for BAME students with disability.

CONCLUSION

Disability had a significant negative impact on all course outcomes, illustrating inequity in medical training and an area of focus for curriculum development. Intersectional data identified a key disadvantaged subgroup of medical students.

Differential attainment in specialty training recruitment in the UK: an observational analysis of the impact of psychometric testing assessment in Public Health postgraduate selection

OBJECTIVES

To determine how current psychometric testing approaches used in selection of postgraduate training in UK Public Health are associated with socioeconomic and sociocultural background of applicants (including ethnicity).

DESIGN

Observational study using contemporaneous data collected during recruitment and psychometric test scores.

SETTING

Assessment centre of UK national Public Health recruitment for postgraduate Public Health training. The assessment centre element of selection comprises three psychometric assessments: Rust Advanced Numerical Reasoning, Watson-Glaser Critical Thinking Assessment II and Public Health situational judgement test.

PARTICIPANTS

629 applicants completed the assessment centre in 2021. 219 (34.8%) were UK medical graduates, 73 (116%) were international medical graduates and 337 (53.6%) were from backgrounds other than medicine.

MAIN OUTCOME MEASURE

Multivariable-adjusted progression statistics in the form of adjusted OR (aOR), accounting for age, sex, ethnicity, professional background and surrogate measures of familial socioeconomic and sociocultural status.

RESULTS

357 (56.8%) candidates passed all three psychometric tests. Candidate characteristics negatively associated with progression were black ethnicity (aOR 0.19, 0.08 to 0.44), Asian ethnicity (aOR 0.35, 0.16 to 0.71) and coming from a non-UK medical graduate background (aOR 0.05, 0.03 to 0.12); similar differential attainment was observed in each of the psychometric tests. Even within the UK-trained medical cohort, candidates from white British backgrounds were more likely to progress than those from ethnic minorities (89.2% vs 75.0%, p=0.003).

CONCLUSION

Although perceived to mitigate the risks of conscious and unconscious bias in selection to medical postgraduate training, these psychometric tests demonstrate unexplained variation that suggests differential attainment. Other specialties should enhance their data collection to evaluate the impact of differential attainment on current selection processes and take forward opportunities to mitigate differential attainment where possible.

Technology enhanced assessment: Ottawa consensus statement and recommendations

INTRODUCTION

In 2011, a consensus report was produced on technology-enhanced assessment (TEA), its good practices, and future perspectives. Since then, technological advances have enabled innovative practices and tools that have revolutionised how learners are assessed. In this updated consensus, we bring together the potential of technology and the ultimate goals of assessment on learner attainment, faculty development, and improved healthcare practices.

METHODS

As a material for the report, we used the scholarly publications on TEA in both HPE and general higher education, feedback from 2020 Ottawa Conference workshops, and scholarly publications on assessment technology practices during the Covid-19 pandemic.

RESULTS AND CONCLUSION

The group identified areas of consensus that remained to be resolved and issues that arose in the evolution of TEA. We adopted a three-stage approach (readiness to adopt technology, application of assessment technology, and evaluation/dissemination). The application stage adopted an assessment 'lifecycle' approach and targeted five key foci: (1) Advancing authenticity of assessment, (2) Engaging learners with assessment, (3) Enhancing design and scheduling, (4) Optimising assessment delivery and recording learner achievement, and (5) Tracking learner progress and faculty activity and thereby supporting longitudinal learning and continuous assessment.

Understanding Healthcare Students' Experiences of Racial Bias: A Narrative Review of the Role of Implicit Bias and Potential Interventions in Educational Settings

Implicit racial bias is a persistent and pervasive challenge within healthcare education and training settings. A recent systematic review reported that 84% of included studies (31 out of 37) showed evidence of slight to strong pro-white or light skin tone bias amongst healthcare students and professionals. However, there remains a need to improve understanding about its impact on healthcare students and how they can be better supported. This narrative review provides an overview of current evidence regarding the role of implicit racial bias within healthcare education, considering trends, factors that contribute to bias, and possible interventions. Current evidence suggests that biases held by students remain consistent and may increase during healthcare education. Sources that contribute to the formation and maintenance of implicit racial bias include peers, educators, the curriculum, and placements within healthcare settings. Experiences of implicit racial bias can lead to psychosomatic symptoms, high attrition rates, and reduced diversity within the healthcare workforce. Interventions to address implicit racial bias include an organizational commitment to reducing bias in hiring, retention, and promotion processes, and by addressing misrepresentation of race in the curriculum. We conclude that future research should identify, discuss, and critically reflect on how implicit racial biases are enacted and sustained through the hidden curriculum and can have detrimental consequences for racial and ethnic minority healthcare students.