Licensing exams and judicial review: the closing of one door and opening of others?

Predictive validity of a tool to resolve borderline grades in OSCEs

There is inconclusive evidence suggesting which standard setting method yields the highest validity for pass/fail decisions in examinations. The Objective Borderline Method 2 (OBM2) is a decision-making tool for reclassification of borderline grades to clear pass or clear fail grades to resolve examiner uncertainty for high-stakes pass/fail decisions. This study evaluated the predictive validity of OBM2 pass/fail decisions, using consecutive years' Objective Structured Clinical Examination (OSCE) results within a medical cohort (n=271) at the University of New South Wales, Australia. OBM2 decisions in one OSCE (n=687) were compared to marks obtained in a subsequent OSCE via independent samples T-tests and analysis of variance (ANOVA). The extent of the relationship between these two variables determines the predictive validity of OBM2 decisions, given that past student grades are capable of predicting future performance. OBM2 decisions in an initial OSCE were found to have a statistically significant predictive nature for subsequent OSCE marks (p=.005). For initial decisions which reclassified to a pass grade, subsequent OSCE marks were significantly higher than for the cases where initial decisions were reclassified to a fail grade. Stronger associations were identified between related assessment domains/criteria compared to unrelated domains/criteria (Cohen's d=.469 vs Cohen's d=.388 respectively). Through demonstrating the OBM2 decisions' predictive association across exams there is support for the OBM2's predictive validity, deeming it a promising method to be used for resolving examiner uncertainty when making pass/fail decisions within OSCEs.

International medical graduates and general practice training: How do educational leaders facilitate the transition from new migrant to local family doctor?

Objectives: To document medical educators' experience and initiatives in training international medical graduates (IMGs) to become general practitioners (GP). Design: Qualitative social-constructivist emergent design with descriptive and interpretive analyses. Setting: GP vocational training in Australia, Canada, Ireland, New Zealand, the Netherlands, and UK. Participants: Twenty-eight leaders of GP training. Intervention: Data collected from public documents, published literature and 27 semi-structured interviews. Main outcome measures: Tensions in training and innovations in response to these tensions. Results: Medical educators identified tension in teaching IMGs as it could be different to teaching domestic graduates in any or all aspects of a training program. They felt an ethical responsibility to support IMGs to provide quality health care in their adopted country but faced multiple challenges to achieve this. They described initiatives to address these throughout GP training. Conclusions: IMG's differing educational needs will benefit from flexible individualized adaptation of training programs.

Exploring reasons for differences in performance between UK and international medical graduates in the Membership of the Royal College of General Practitioners Applied Knowledge Test: a cognitive interview study

OBJECTIVES

International medical graduates (IMGs) perform less well in national postgraduate licensing examinations compared with UK graduates, even in computer-marked multiple-choice licensing examinations. We aimed to investigate thought processes of candidates answering multiple- choice questions, considering possible reasons for differential attainment between IMGs and UK graduates.

DESIGN

We employed a semistructured qualitative design using cognitive interviews. Systematic grounded theory was used to analyse data from 'think aloud' interviews of general practitioner specialty trainees (GPSTs) while answering up to 15 live questions from the UK Membership of the Royal College of General Practitioners Applied Knowledge Test (AKT).

SETTING

East Midlands, UK.

PARTICIPANTS

21 GPSTs including 13IMGs and 8 UK-trained doctors.

OUTCOMES

Perceptions of participants on how they answered AKT questions together with strategies used or difficulties experienced.

RESULTS

We interviewed 21 GPSTs (8 female, 13 male, 13 IMGs, 14 from black and minority ethnic groups, age 24-64 years) in years 1-3 of training between January and April 2017. Four themes were identified. 'Theoretical versus real-life clinical experience': participants reported difficulties recalling information and responding to questions from theoretical learning compared with clinical exposure; rote learning helped some IMGs recall rare disease patterns. Recency, frequency, opportunity and relevance: participants reported greater difficulty answering questions not recently studied, less frequently encountered or perceived as less relevant. Competence versus insight: some participants were over optimistic about their performance despite answering incorrectly. Cultural barriers: for IMGs included differences in undergraduate experience, lack of familiarity with UK guidelines and language barriers which overlapped with the other themes.

CONCLUSIONS

The difficulties we identified in candidates when answering AKT questions may be addressed through training. IMGs face additional difficulties which impede examination success due to differences in educational experience, content familiarity and language, which are also potentially amenable to additional training support.

Has there been a change in the knowledge of GP registrars between 2011 and 2016 as measured by performance on common items in the Applied Knowledge Test?

The aim of this study was to assess whether the absolute standard of candidates sitting the MRCGP Applied Knowledge Test (AKT) between 2011 and 2016 had changed. It is a descriptive study comparing the performance on marker questions of a reference group of UK graduates taking the AKT for the first time between 2011 and 2016. Using aggregated examination data, the performance of individual 'marker' questions was compared using Pearson's chi-squared tests and trend-line analysis. Binary logistic regression was used to analyse changes in performance over the study period. Changes in performance of individual marker questions using Pearson's chi-squared test showed statistically significant differences in 32 of the 49 questions included in the study. Trend line analysis showed a positive trend in 29 questions and a negative trend in the remaining 23. The magnitude of change was small. Logistic regression did not demonstrate any evidence for a change in the performance of the question set over the study period. However, candidates were more likely to get items on administration wrong compared with clinical medicine or research. There was no evidence of a change in performance of the question set as a whole.

Person-centred medicines optimisation policy in England: an agenda for research on polypharmacy

Aim To examine how patient perspectives and person-centred care values have been represented in documents on medicines optimisation policy in England.

BACKGROUND

There has been growing support in England for a policy of medicines optimisation as a response to the rise of problematic polypharmacy. Conceptually, medicines optimisation differs from the medicines management model of prescribing in being based around the patient rather than processes and systems. This critical examination of current official and independent policy documents questions how central the patient is in them and whether relevant evidence has been utilised in their development.

METHODS

A documentary analysis of reports on medicines optimisation published by the Royal Pharmaceutical Society (RPS), The King's Fund and National Institute for Health and Social Care Excellence since 2013. The analysis draws on a non-systematic review of research on patient experiences of using medicines. Findings The reports varied in their inclusion of patient perspectives and person-centred care values, and in the extent to which they drew on evidence from research on patients' experiences of polypharmacy and medicines use. In the RPS report, medicines optimisation is represented as being a 'step change' from medicines management, in contrast to the other documents which suggest that it is facilitated by the systems and processes that comprise the latter model. Only The King's Fund report considered evidence from qualitative studies of people's use of medicines. However, these studies are not without their limitations. We suggest five ways in which researchers could improve this evidence base and so inform the development of future policy: by facilitating reviews of existing research; conducting studies of patient experiences of polypharmacy and multimorbidity; evaluating medicines optimisation interventions; making better use of relevant theories, concepts and tools; and improving patient and public involvement in research and in guideline development.