1
|
Yeates P, Maluf A, Kinston R, Cope N, McCray G, Cullen K, O'Neill V, Cole A, Goodfellow R, Vallender R, Chung CW, McKinley RK, Fuller R, Wong G. Enhancing authenticity, diagnosticity and equivalence (AD-Equiv) in multicentre OSCE exams in health professionals education: protocol for a complex intervention study. BMJ Open 2022; 12:e064387. [PMID: 36600366 PMCID: PMC9730346 DOI: 10.1136/bmjopen-2022-064387] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/04/2022] [Accepted: 10/12/2022] [Indexed: 12/12/2022] Open
Abstract
INTRODUCTION Objective structured clinical exams (OSCEs) are a cornerstone of assessing the competence of trainee healthcare professionals, but have been criticised for (1) lacking authenticity, (2) variability in examiners' judgements which can challenge assessment equivalence and (3) for limited diagnosticity of trainees' focal strengths and weaknesses. In response, this study aims to investigate whether (1) sharing integrated-task OSCE stations across institutions can increase perceived authenticity, while (2) enhancing assessment equivalence by enabling comparison of the standard of examiners' judgements between institutions using a novel methodology (video-based score comparison and adjustment (VESCA)) and (3) exploring the potential to develop more diagnostic signals from data on students' performances. METHODS AND ANALYSIS The study will use a complex intervention design, developing, implementing and sharing an integrated-task (research) OSCE across four UK medical schools. It will use VESCA to compare examiner scoring differences between groups of examiners and different sites, while studying how, why and for whom the shared OSCE and VESCA operate across participating schools. Quantitative analysis will use Many Facet Rasch Modelling to compare the influence of different examiners groups and sites on students' scores, while the operation of the two interventions (shared integrated task OSCEs; VESCA) will be studied through the theory-driven method of Realist evaluation. Further exploratory analyses will examine diagnostic performance signals within data. ETHICS AND DISSEMINATION The study will be extra to usual course requirements and all participation will be voluntary. We will uphold principles of informed consent, the right to withdraw, confidentiality with pseudonymity and strict data security. The study has received ethical approval from Keele University Research Ethics Committee. Findings will be academically published and will contribute to good practice guidance on (1) the use of VESCA and (2) sharing and use of integrated-task OSCE stations.
Collapse
Affiliation(s)
- Peter Yeates
- School of Medicine, Keele University, Keele, Staffordshire, UK
| | - Adriano Maluf
- School of Medicine, Keele University, Keele, Staffordshire, UK
| | - Ruth Kinston
- School of Medicine, Keele University, Keele, Staffordshire, UK
| | - Natalie Cope
- School of Medicine, Keele University, Keele, Staffordshire, UK
| | - Gareth McCray
- School of Medicine, Keele University, Keele, Staffordshire, UK
| | - Kathy Cullen
- School of Medicine, Dentistry and Biomedical Sciences, Queen's University Belfast, Belfast, UK
| | - Vikki O'Neill
- School of Medicine, Dentistry and Biomedical Sciences, Queen's University Belfast, Belfast, UK
| | - Aidan Cole
- School of Medicine, Dentistry and Biomedical Sciences, Queen's University Belfast, Belfast, UK
| | | | | | - Ching-Wa Chung
- School of Medicine, Medical Sciences and Nutrition, University of Aberdeen, Aberdeen, Scotland, UK
| | | | - Richard Fuller
- School of Medicine, University of Liverpool Faculty of Health and Life Sciences, Liverpool, UK
| | - Geoff Wong
- Nuffield Department of Primary Care Health Sciences, University of Oxford Division of Public Health and Primary Health Care, Oxford, Oxfordshire, UK
| |
Collapse
|
2
|
Yeates P, Moult A, Cope N, McCray G, Fuller R, McKinley R. Determining influence, interaction and causality of contrast and sequence effects in objective structured clinical exams. MEDICAL EDUCATION 2022; 56:292-302. [PMID: 34893998 PMCID: PMC9304241 DOI: 10.1111/medu.14713] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/23/2021] [Revised: 11/03/2021] [Accepted: 12/08/2021] [Indexed: 06/14/2023]
Abstract
INTRODUCTION Differential rater function over time (DRIFT) and contrast effects (examiners' scores biased away from the standard of preceding performances) both challenge the fairness of scoring in objective structured clinical exams (OSCEs). This is important as, under some circumstances, these effects could alter whether some candidates pass or fail assessments. Benefitting from experimental control, this study investigated the causality, operation and interaction of both effects simultaneously for the first time in an OSCE setting. METHODS We used secondary analysis of data from an OSCE in which examiners scored embedded videos of student performances interspersed between live students. Embedded video position varied between examiners (early vs. late) whilst the standard of preceding performances naturally varied (previous high or low). We examined linear relationships suggestive of DRIFT and contrast effects in all within-OSCE data before comparing the influence and interaction of 'early' versus 'late' and 'previous high' versus 'previous low' conditions on embedded video scores. RESULTS Linear relationships data did not support the presence of DRIFT or contrast effects. Embedded videos were scored higher early (19.9 [19.4-20.5]) versus late (18.6 [18.1-19.1], p < 0.001), but scores did not differ between previous high and previous low conditions. The interaction term was non-significant. CONCLUSIONS In this instance, the small DRIFT effect we observed on embedded videos can be causally attributed to examiner behaviour. Contrast effects appear less ubiquitous than some prior research suggests. Possible mediators of these finding include the following: OSCE context, detail of task specification, examiners' cognitive load and the distribution of learners' ability. As the operation of these effects appears to vary across contexts, further research is needed to determine the prevalence and mechanisms of contrast and DRIFT effects, so that assessments may be designed in ways that are likely to avoid their occurrence. Quality assurance should monitor for these contextually variable effects in order to ensure OSCE equivalence.
Collapse
Affiliation(s)
- Peter Yeates
- School of MedicineKeele UniversityKeeleUK
- Fairfield General HospitalPennine Acute Hospitals NHS TrustBuryUK
| | | | | | | | | | | |
Collapse
|
3
|
Moult A, McKinley RK, Yeates P. Understanding patient involvement in judging students' communication skills in OSCEs. MEDICAL TEACHER 2021; 43:1070-1078. [PMID: 34496725 DOI: 10.1080/0142159x.2021.1915467] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
INTRODUCTION Communication skills are assessed by medically-enculturated examiners using consensus frameworks which were developed with limited patient involvement. Assessments consequently risk rewarding performance which incompletely serves patients' authentic communication needs. Whilst regulators require patient involvement in assessment, little is known about how this can be achieved. We aimed to explore patients' perceptions of students' communication skills, examiner feedback and potential roles for patients in assessment. METHODS Using constructivist grounded theory we performed cognitive stimulated, semi-structured interviews with patients who watched videos of student performances in communication-focused OSCE stations and read corresponding examiner feedback. Data were analysed using grounded theory methods. RESULTS A disconnect occurred between participants' and examiners' views of students' communication skills. Whilst patients frequently commented on students' use of medical terminology, examiners omitted to mention this in feedback. Patients' judgements of students' performances varied widely, reflecting different preferences and beliefs. Participants viewed variability as an opportunity for students to learn from diverse lived experiences. Participants perceived a variety of roles to enhance assessment authenticity. DISCUSSION Integrating patients into communications skills assessments could help to highlight deficiencies in students' communication which medically-enculturated examiners may miss. Overcoming the challenges inherent to this is likely to enhance graduates' preparedness for practice.
Collapse
Affiliation(s)
- Alice Moult
- School of Medicine, Keele University, Keele, UK
| | | | | |
Collapse
|