Are remote clinical assessments a feasible and acceptable method of assessment? A systematic review

Feasibility and reliability of the pandemic-adapted online-onsite hybrid graduation OSCE in Japan

Objective structured clinical examination (OSCE) is widely used to assess medical students' clinical skills. Virtual OSCEs were used in place of in-person OSCEs during the COVID-19 pandemic; however, their reliability is yet to be robustly analyzed. By applying generalizability (G) theory, this study aimed to evaluate the reliability of a hybrid OSCE, which admixed in-person and online methods, and gain insights into improving OSCEs' reliability. During the 2020-2021 hybrid OSCEs, one examinee, one rater, and a vinyl mannequin for physical examination participated onsite, and a standardized simulated patient (SP) for medical interviewing and another rater joined online in one virtual breakout room on an audiovisual conferencing system. G-coefficients and 95% confidence intervals of the borderline score, namely border zone (BZ), under the standard 6-station, 2-rater, and 6-item setting were calculated. G-coefficients of in-person (2017-2019) and hybrid OSCEs (2020-2021) under the standard setting were estimated to be 0.624, 0.770, 0.782, 0.759, and 0.823, respectively. The BZ scores were estimated to be 2.43-3.57, 2.55-3.45, 2.59-3.41, 2.59-3.41, and 2.51-3.49, respectively, in the score range from 1 to 6. Although hybrid OSCEs showed reliability comparable to in-person OSCEs, they need further improvement as a very high-stakes examination. In addition to increasing clinical vignettes, having more proficient online/on-demand raters and/or online SPs for medical interviews could improve the reliability of OSCEs. Reliability can also be ensured through supplementary examination and by increasing the number of online raters for a small number of students within the BZs.

Online medical student OSCE examinations during the first three years of the COVID-19 pandemic compared to three years pre-pandemic: An Australian experience in psychiatry and addiction medicine

PURPOSE

We have evaluated the final-year Psychiatry and Addiction Medicine (PAM) summative Objective Structured Clinical Examination (OSCE) examinations in a four-year graduate medical degree program, for the previous three years as a baseline comparator, and during three years of the COVID-19 pandemic (2020-2022).

METHODS

A de-identified analysis of medical student summative OSCE examination performance, and comparative review for the 3 years before, and for each year of the pandemic.

RESULTS

Internal reliability in test scores as measured by R-squared remained the same or increased following the start of the pandemic. There was a significant increase in mean test scores after the start of the pandemic compared to pre-pandemic for combined OSCE scores for all final-year disciplines, as well as for the PAM role-play OSCEs, but not for the PAM mental state examination OSCEs.

CONCLUSIONS

Changing to online OSCEs during the pandemic was related to an increase in scores for some but not all domains of the tests. This is in line with a nascent body of literature on medical teaching and examination following the start of the pandemic. Further research is needed to optimise teaching and examination in a post-pandemic medical school environment.

A level playing field? Evaluation of the virtual Objective Structured Clinical Examination in Psychiatry and Addiction Medicine: A mixed methods study

OBJECTIVE

The Objective Structured Clinical Examination (OSCE) has been used for clinical assessment of a broad range of medical student competencies in Psychiatry and Addiction Medicine. However, there has been little research into online assessments. We investigated the virtual OSCE (v-OSCE) from the user perspective to better understand its acceptability, usefulness, benefits, challenges and potential improvements.

METHODS

At the conclusion of the v-OSCE, all participants (medical students, examiners and simulated patients) were invited to participate in a brief online survey, based on the Technology and Acceptance Model. Freeform qualitative feedback was also obtained to explore participants' experiences and attitudes.

RESULTS

Participants reported the v-OSCE was acceptable, efficient, convenient and easy to use. It was perceived as useful for demonstrating students' interviewing skills and interacting with the simulated patient. Benefits included greater convenience, reduced stress and travel time. Challenges were similar to those experienced in 'real world' telepsychiatry, primarily related to assessment of non-verbal cues and emotional prosody.

CONCLUSIONS

Our findings inform recommendations for improving online examinations. These include increased preparation, practice and professionalism, to better simulate the in-person experience. Study credibility was strengthened by the triangulation of qualitative, quantitative and psychometric data.

The use of objective structured clinical examination in dental education- a narrative review

The Objective Structured Clinical Examination (OSCE) is a performance-based assessment intended to assess medical students' clinical competency in a simulated, standardized environment. Because it measures the student's ability to use clinical knowledge, diagnostic skill, and decision-making, the OSCE is thought to be more objective than traditional tests. OSCE exams have been increasingly employed in dentistry schools, particularly in the last decade, and it is crucial to investigate instructors' and dental students' experiences with this evaluation approach.

A comparison of the American Board of Anesthesiology's in-person and virtual objective structured clinical examinations

BACKGROUND

The American Board of Anesthesiology's Objective Structured Clinical Examination (OSCE), as a component of its initial certification process, had been administered in-person in a dedicated assessment center since its launch in 2018 until March 2020. Due to the COVID-19 pandemic, a virtual format of the exam was piloted in December 2020 and was administered in 2021. This study aimed to compare candidate performance, examiner grading severity, and scenario difficulty between these two formats of the OSCE.

METHODS

The Many-Facet Rasch Model was utilized to estimate candidate performance, examiner grading severity, and scenario difficulty for the in-person and virtual OSCEs separately. The virtual OSCE was equated to the in-person OSCE by common examiners and common scenarios. Independent-samples t-test was used to compare candidate performance, and partially overlapping samples t-tests were applied to compare examiner grading severity and scenario difficulty between the in-person and virtual OSCEs.

RESULTS

The in-person (n = 3235) and virtual (n = 2934) first-time candidates were comparable in age, sex, race/ethnicity, and whether U.S. medical school graduates. The virtual scenarios (n = 35, mean [0.21] ± SD [0.38] in logits) were more difficult than the in-person scenarios (n = 93, 0.00 ± 0.69, Welch's partially overlapping samples t-test, p = 0.01); there were no statistically significant differences in examiner severity (n = 390, -0.01 ± 0.82 vs. n = 304, -0.02 ± 0.93, Welch's partially overlapping samples t-test, p = 0.81) or candidate performance (2.19 ± 0.93 vs. 2.18 ± 0.92, Welch's independent samples t-test, p = 0.83) between the in-person and virtual OSCEs.

CONCLUSIONS

Our retrospective analyses of first-time OSCEs found comparable candidate performance and examiner grading severity between the in-person and virtual formats, despite the virtual scenarios being more difficult than the in-person scenarios. These results provided assurance that the virtual OSCE functioned reasonably well in a high-stakes setting.

Innovations in assessment in health professions education during the COVID-19 pandemic: A scoping review

BACKGROUND

After an initial period of uncertainty during the COVID-19 pandemic, medical universities responded to the pandemic by innovating their methods of teaching-learning and assessment. This scoping review aimed to identify innovations in assessment in medical, dental and nursing education at the undergraduate and postgraduate levels during the pandemic.

METHODS

This review was conducted utilising Arksey and O'Malley's framework where three electronic databases-PubMed, Embase and Web of Science-were systematically searched in February 2022 for articles in English describing innovations in assessments published in or after January 2020.

RESULTS

A total of 70 articles were included in the review. Among the primary research articles, 82.1% reported online assessment, 76.1% were related to medical speciality, 74.6% described assessment at the undergraduate level, 68.7% described summative assessment and 58.2% reported assessment of practical skills. All assessments of theoretical knowledge were done virtually with MCQ being the most used assessment tool and virtual OSCE being the most popular assessment tool to assess practical skills; however, it was acknowledged that the assessment of procedural skills was its limitation. Although the studies described the newer forms of assessments as feasible and acceptable, few studies reported the reliability and validity of the assessments.

CONCLUSIONS

This scoping review identified several innovations in assessment during the COVID-19 pandemic with an overall shift towards virtual or hybrid forms of assessment. However, there was a dearth of evidence regarding the effectiveness of these assessment tools suggesting a need for more research before the tools can be implemented post-pandemic.

Adaptation to Virtual Assessment during the COVID-19 Pandemic: Clinical Case Presentation Examination

BACKGROUND

Case presentation assessment is common in both medicine and dentistry and is known under various names depending on the country and institution. It relates mainly to aspects of diagnosis and treatment planning and is considered highly authentic and useful. The COVID-19 pandemic necessitated the movement of this assessment from face-to-face to online. The aim of this investigation was to explore the students' impressions of the two different examination modalities. With this information, a decision on future diets of this examination can be made to accommodate the students' perspectives.

METHODS

Quantitative and qualitative data were gathered using an online, self-administered survey.

RESULTS

The students were split 50/50 regarding which assessment modality they preferred. Overall, they considered the online examination to be fair, and the majority agreed that the online format allowed them to display their knowledge as well as face-to-face.

CONCLUSIONS

The delivery of case presentation examination is possible online. An online case presentation is a fair, useful, and authentic assessment that is appropriate to the needs of the faculty and students. Satisfaction with the two possible methods of conducting this assessment suggests it would be reasonable to conduct this examination online in the future.