Has the OSCE Met Its Final Demise? Rebalancing Clinical Assessment Approaches in the Peri-Pandemic World

Feasibility and reliability of the pandemic-adapted online-onsite hybrid graduation OSCE in Japan

Objective structured clinical examination (OSCE) is widely used to assess medical students' clinical skills. Virtual OSCEs were used in place of in-person OSCEs during the COVID-19 pandemic; however, their reliability is yet to be robustly analyzed. By applying generalizability (G) theory, this study aimed to evaluate the reliability of a hybrid OSCE, which admixed in-person and online methods, and gain insights into improving OSCEs' reliability. During the 2020-2021 hybrid OSCEs, one examinee, one rater, and a vinyl mannequin for physical examination participated onsite, and a standardized simulated patient (SP) for medical interviewing and another rater joined online in one virtual breakout room on an audiovisual conferencing system. G-coefficients and 95% confidence intervals of the borderline score, namely border zone (BZ), under the standard 6-station, 2-rater, and 6-item setting were calculated. G-coefficients of in-person (2017-2019) and hybrid OSCEs (2020-2021) under the standard setting were estimated to be 0.624, 0.770, 0.782, 0.759, and 0.823, respectively. The BZ scores were estimated to be 2.43-3.57, 2.55-3.45, 2.59-3.41, 2.59-3.41, and 2.51-3.49, respectively, in the score range from 1 to 6. Although hybrid OSCEs showed reliability comparable to in-person OSCEs, they need further improvement as a very high-stakes examination. In addition to increasing clinical vignettes, having more proficient online/on-demand raters and/or online SPs for medical interviews could improve the reliability of OSCEs. Reliability can also be ensured through supplementary examination and by increasing the number of online raters for a small number of students within the BZs.

Linking assessment to real life practice - comparing work based assessments and objective structured clinical examinations using mystery shopping

Objective Structured Clinical Examinations (OSCEs) and Work Based Assessments (WBAs) are the mainstays of assessing clinical competency in health professions' education. Underpinned by the extrapolation inference in Kane's Validity Framework, the purpose of this study is to determine whether OSCEs translate to real life performance by comparing students' OSCE performance to their performance in real-life (as a WBA) using the same clinical scenario, and to understand factors that affect students' performance. A sequential explanatory mixed methods approach where a grade comparison between students' performance in their OSCE and WBA was performed. Students were third year pharmacy undergraduates on placement at a community pharmacy in 2022. The WBA was conducted by a simulated patient, unbeknownst to students and indistinguishable from a genuine patient, visiting the pharmacy asking for health advice. The simulated patient was referred to as a 'mystery shopper' and the process to 'mystery shopping' in this manuscript. Community pharmacy is an ideal setting for real-time observation and mystery shopping as staff can be accessed without appointment. The students' provision of care and clinical knowledge was assessed by the mystery shopper using the same clinical checklist the student was assessed from in the OSCE. Students who had the WBA conducted were then invited to participate in semi-structured interviews to discuss their experiences in both settings. Overall, 92 mystery shopper (WBA) visits with students were conducted and 36 follow-up interviews were completed. The median WBA score was 41.7% [IQR 28.3] and significantly lower compared to the OSCE score 80.9% [IQR 19.0] in all participants (p < 0.001). Interviews revealed students knew they did not perform as well in the WBA compared to their OSCE, but reflected that they still need OSCEs to prepare them to manage real-life patients. Many students related their performance to how they perceived their role in OSCEs versus WBAs, and that OSCEs allowed them more autonomy to manage the patient as opposed to an unfamiliar workplace. As suggested by the activity theory, the performance of the student can be driven by their motivation which differed in the two contexts.

Development and validation of the Overall Fidelity Enactment Scale for Complex Interventions (OFES-CI)

BACKGROUND

In many quality improvement (QI) and other complex interventions, assessing the fidelity with which participants 'enact' intervention activities (ie, implement them as intended) is underexplored. Adapting the evaluative approach used in objective structured clinical examinations, we aimed to develop and validate a practical approach to assessing fidelity enactment-the Overall Fidelity Enactment Scale for Complex Interventions (OFES-CI).

METHODS

We developed the OFES-CI to evaluate enactment of the SCOPE QI intervention, which teaches nursing home teams to use plan-do-study-act (PDSA) cycles. The OFES-CI was piloted and revised early in SCOPE with good inter-rater reliability, so we proceeded with a single rater. An intraclass correlation coefficient (ICC) was used to assess inter-rater reliability. For 27 SCOPE teams, we used ICC to compare two methods for assessing fidelity enactment: (1) OFES-CI ratings provided by one of five trained experts who observed structured 6 min PDSA progress presentations made at the end of SCOPE, (2) average rating of two coders' deductive content analysis of qualitative process evaluation data collected during the final 3 months of SCOPE (our gold standard).

RESULTS

Using Cicchetti's classification, inter-rater reliability between two coders who derived the gold standard enactment score was 'excellent' (ICC=0.93, 95% CI=0.85 to 0.97). Inter-rater reliability between the OFES-CI and the gold standard was good (ICC=0.71, 95% CI=0.46 to 0.86), after removing one team where open-text comments were discrepant with the rating. Rater feedback suggests the OFES-CI has strong face validity and positive implementation qualities (acceptability, easy to use, low training requirements).

CONCLUSIONS

The OFES-CI provides a promising novel approach for assessing fidelity enactment in QI and other complex interventions. It demonstrates good reliability against our gold standard assessment approach and addresses the practicality problem in fidelity assessment by virtue of its suitable implementation qualities. Steps for adapting the OFES-CI to other complex interventions are offered.

Twelve tips for improving the quality of assessor judgements in senior medical student clinical assessments

Assessment of senior medical students is usually calibrated at the level of achieving expected learning outcomes for graduation. Recent research reveals that clinical assessors often balance two slightly different perspectives on this benchmark. The first is the formal learning outcomes at graduation, ideally as part of a systematic, program-wide assessment approach that measures learning achievement, while the second is consideration of the candidate's contribution to safe care and readiness for practice as a junior doctor. The second is more intuitive to the workplace, based on experience working with junior doctors. This perspective may enhance authenticity in assessment decisions made in OSCEs and work-based assessments to better align judgements and feedback with professional expectations that will guide senior medical students and junior doctors' future career development. Modern assessment practices should include consideration of qualitative as well as quantitative information, overtly including perspectives of patients, employers, and regulators. This article presents 12 tips for how medical education faculty might support clinical assessors by capturing workplace expectations of first year medical graduates and develop graduate assessments based on a shared heuristic of 'work-readiness'. Peer-to-peer assessor interaction should be facilitated to achieve correct calibration that 'merges' the differing perspectives to produce a shared construct of an acceptable candidate.

Structures, processes and outcomes of objective structured clinical examinations in dental education during the COVID-19 pandemic: A scoping review

INTRODUCTION

Objective structured clinical examinations (OSCEs) are an essential examination tool within undergraduate dental education. Fear of spread of the COVID-19 virus led to dental institutions exploring alternative means of conducting OSCEs. The aim of this scoping review was to investigate what structures, processes and outcomes of dental OSCEs were reported during the COVID-19 pandemic.

MATERIALS AND METHODS

This scoping review was conducted and reported adhering to the Preferred Reporting Items for Systematic Reviews and Meta-analyses extension for scoping review guidelines (PRISMA-ScR). Published literature was identified through a systematic search of PubMed, Embase, Cumulative Index to Nursing and Allied Health Literature (CINAHL), Education Resources Information Center (Eric), ProQuest and Google Scholar. Identified articles were independently reviewed by two authors (KS, AD), followed by synthesis in terms of the reported structures, processes and outcomes. Articles reporting cancellation or rescheduling were also included, extracting data on reasons and any suggestions/recommendations.

RESULTS

The search yielded a total of 290 studies of which 239 sources were excluded after removal of duplicates, leaving 51 studies for title and abstract evaluation. Thirty-four articles were excluded as they did not report on the topic of interest, leaving 17 for full-text evaluation, of which nine were analysed according to the pre-set themes. All dental OSCEs taking place (n = 6) were conducted online whilst the remaining (n = 3) were either cancelled or rescheduled. Data on structures reported specific online videoconferencing software used and provision of staff and student training. Processes on the execution of online OSCEs varied significantly from one study to the other, providing rich data on how dental institutions may carry out such assessments tailored to their need. Information regarding outcomes was sparse, as little attention was paid to the results of the students compared to pre-pandemic, lacking investigation into reliability and validity of online dental OSCEs.

CONCLUSION

Dental OSCEs could be conducted online implementing well-planned structures and processes; however, further evidence is needed to prove its reliability and validity based on outcomes. Dental institutions may need to consider alternative methods to assess practical competencies if online OSCEs are to take place.

Prior degree and academic performance in medical school: evidence for prioritising health students and moving away from a bio-medical science-focused entry stream

BACKGROUND

Given the importance of the selection process, many medical schools are reviewing their selection criteria. The traditional pathway for post-graduate medicine has been from science-based undergraduate degrees, however some programs are expanding their criteria. In this study we investigated academic success across all years and themes of the Deakin University medical degree, based on the type of degree undertaken prior to admission. We evaluated whether the traditional pathway of biomedical science into medicine should remain the undergraduate degree of choice, or whether other disciplines should be encouraged.

METHODS

Data from 1159 students entering the degree from 2008 to 2016 was collected including undergraduate degree, grade point average (GPA), Graduate Medical Schools Admission Test (GAMSAT) score and academic outcomes during the 4 years of the degree. Z-scores were calculated for each assessment within each cohort and analysed using a one sample t-test to determine if they differed from the cohort average. Z-scores between groups were analysed by 1-way ANOVA with LSD post-hoc analysis correcting for multiple comparisons.

RESULTS

The majority of students had Science (34.3%) or Biomedical Science (31.0%) backgrounds. 27.9% of students had a Health-related undergraduate degree with smaller numbers of students from Business (3.5%) and Humanities (3.4%) backgrounds. At entry, GPA and GAMSAT scores varied significantly with Biomedical Science and Science students having significantly higher scores than Health students. Health students consistently outperformed students from other disciplines in all themes while Biomedical Science students underperformed.

CONCLUSIONS

Our data suggest that a Health-related undergraduate degree results in the best performance throughout medical school, whereas a Biomedical Science background is associated with lower performance. These findings challenge the traditional Biomedical Science pathway into medicine and suggest that a health background might be more favourable when determining the selection criteria for graduate entry into medicine.

Patient involvement in assessment: How useful is it?

The author unveils a strategy for enquiry that can facilitate identification of best practices for involving real patients in OSCE and WBA competency‐based assessments