Virtual Objective Structured Clinical Examination Experiences and Performance in Physical Medicine and Rehabilitation Residency

A Comparison Between In-Person and Virtual Communication Skills OSCE for Medical Students

Objectives

This study investigates the effectiveness of a virtual format of an advanced communication skills observed structured clinical examination (OSCE) for senior medical students in comparison to an in-person format. The study also examines the emotional support students experience in the virtual setting. Our analysis was based on quantitative data collected through objective checklists and post-OSCE survey results.

Methods

The virtual OSCE was a revision of an earlier in-person formative advanced communication skills OSCE for fourth-year medical students. Student performances were assessed by self and peers using objective checklists-the modified Master Interview Rating Scale (mMIRS) and Communication Behavior Checklist (CBC). The mMIRS measured interview process such as avoiding jargon and demonstrating empathy. The CBC examined interview content which included tasks specific to the content of the case. The OSCE was followed by a faculty-led debrief and quantitative survey. The virtual OSCE was conducted in 2021, and the results of the checklists and survey were compared with those collected from two earlier in-person OSCEs.

Results

Eighty-three students participated in the virtual OSCE. There was no difference in mMIRS scores between the virtual and in-person OSCE. Overall CBC scores were lower in the virtual OSCE compared to in-person (p < 0.05). Sixty-seven out of 83 (80.7%) students completed the post-OSCE survey. There were no differences between the virtual and in-person OSCE in terms of educational value, whether the OSCE would change the way participants talk to patients, and preparedness to have serious conversations with patients. All respondents somewhat or strongly agreed with feeling emotionally supported during the virtual OSCE.

Conclusion

The virtual format was a suitable alternative to an in-person, formative, advanced communication skills OSCE for medical students. The virtual OSCE was educationally effective and was met with student satisfaction and a sense of emotional support. Future virtual iterations must ensure adequate instruction on interview content.

Implementation of virtual OSCE in health professions education: A systematic review

INTRODUCTION

The Objective Structured Clinical Examination (OSCE) has been widely used in health professions education since the 1970s. The global disruption caused by the COVID-19 pandemic restricted in-person assessments and medical educators globally sought alternative means to assess and certify students and trainees to meet the acute demand for health-care workers. One such solution was through virtual OSCE (vOSCE), which modified traditional in-person OSCE using videoconference platforms. This meta-ethnography sought to synthesise qualitative literature on candidates' and assessors' experiences of vOSCE to evaluate whether it may have a role in future assessment practices.

METHODS

In June 2022, we systematically searched PsycINFO, Medline and ERIC for peer-reviewed qualitative and mixed-methods articles that described candidates' and assessors' experiences of virtual OSCE in health professions education. Of 1069 articles identified, 17 were synthesised using meta-ethnography.

RESULTS

The final synthesis represented 1190 candidates and assessors from faculties of medicine, dentistry, nursing, pharmacy and osteopathy. We developed our findings into four key concepts. 'Strengthening confidence in a virtual environment' highlighted attempts to overcome and mitigate concerns associated with transitioning from in-person to virtual assessment. 'Understanding the scope of use as an assessment' reflected on the suitability of vOSCE in assessing various skills. 'Refining operational processes' emphasised the technical challenges of implementing vOSCE and impacts on accessibility and resources. 'Envisioning its future role' considered the applicability of vOSCE in the climate of rapid development in telehealth.

CONCLUSION

This meta-ethnography highlighted that although vOSCE was primarily considered a temporary and crisis response, candidates and assessors recognised positive, as well as negative, consequences of the transition towards them. Moving forward, medical education policymakers should carefully consider the extent to which elements of vOSCE could be incorporated into assessment systems, particularly in light of the rise of telehealth in clinical practice.

Comparability of Objective Structured Clinical Examinations (OSCEs) and Written Tests for Assessing Medical School Students' Competencies: A Scoping Review

Objective Structured Clinical Examinations (OSCEs) and written tests are commonly used to assess health professional students, but it remains unclear whether the additional human resources and expenses required for OSCEs, both in-person and online, are worthwhile for assessing competencies. This scoping review summarized literature identified by searching MEDLINE and EMBASE comparing 1) OSCEs and written tests and 2) in-person and online OSCEs, for assessing health professional trainees' competencies. For Q1, 21 studies satisfied inclusion criteria. The most examined health profession was medical trainees (19, 90.5%), the comparison was most frequently OSCEs versus multiple-choice questions (MCQs) (18, 85.7%), and 18 (87.5%) examined the same competency domain. Most (77.5%) total score correlation coefficients between testing methods were weak (r < 0.40). For Q2, 13 articles were included. In-person and online OSCEs were most used for medical trainees (9, 69.2%), checklists were the most prevalent evaluation scheme (7, 63.6%), and 14/17 overall score comparisons were not statistically significantly different. Generally low correlations exist between MCQ and OSCE scores, providing insufficient evidence as to whether OSCEs provide sufficient value to be worth their additional cost. Online OSCEs may be a viable alternative to in-person OSCEs for certain competencies where technical challenges can be met.

Student Performance on an Objective Structured Clinical Exam Delivered Both Virtually and In-Person

OBJECTIVE

Passing a milestone objective structured clinical examination (OSCE) is a graduation requirement for the University of Waterloo Pharmacy students. In January 2021, the milestone OSCE was offered concurrently both virtually and in-person, with students being able to choose their desired format. The purpose of this study was to compare student performance between the 2 formats and to identify factors that may have predicted student choice of format.

METHODS

Objective structured clinical examination scores for in-person and virtual exam-takers were compared using 2-tailed independent t tests with Bonferroni correction. Pass rates were compared using χ2 analysis. Prior academic performance variables were analyzed to identify predictors of the chosen exam format. Student and exam personnel surveys were used to capture OSCE feedback.

RESULTS

A total of 67 students (56%) participated in the in-person OSCE, and 52 students (44%) participated virtually. There were no significant differences in overall exam averages or pass rates between the 2 groups. However, virtual exam-takers scored lower in 2 of 7 cases. Previous academic performance did not predict the choice of exam format. Feedback surveys indicated that the exam organization was perceived as a strength regardless of format, but in-person students felt more prepared for the exam than virtual exam-takers with technical challenges and difficulty navigating station resources being noted as barriers in the virtual offering.

CONCLUSION

Virtual and in-person administration of a milestone OSCE resulted in similar student performance, with slightly lower performance on 2 individual case scores with virtual delivery. These results may inform the future development of virtual OSCEs.

Real-Time Dual Video Conferencing of Simulated Tracheostoma Maturation During Residency Interviews

BACKGROUND

There are few reports of dexterity tests being done in a distance telecommunication setting for residency applicant evaluation.

OBJECTIVE

To report the feasibility and suitability of a virtual suturing skills assessment during residency interviews when added to the standard assessment process.

METHODS

A suturing simulation was developed and implemented during otolaryngology-head and neck surgery (OHNS) residency interviews for the 2020-2021 cycle at one program. On the day of the interview, the activity was completed in real time using 2-camera video conferencing with the 2 resident assessors providing a numerical assessment based on an adapted scoring rubric from prior suturing activities at the institution. The exercise involved suturing a 3/4-inch Penrose drain circumferentially with half-vertical mattress stitches to simulate the maturation of a tracheostoma. The residency selection committee then completed a 7-item Likert-type survey, developed by the authors, to evaluate the simulation exercise.

RESULTS

Fifty-one applicants representing all interviewees in the cycle successfully completed this assessment without technologic disruptions. The total cost associated with obtaining and providing the necessary supplies to applicants was $34.78 per interviewee. Time required to complete the suturing task was estimated to range from 10 to 20 minutes. The residency selection committee viewed this exercise as a success (14 of 16, 87.5%) and viewed the results as a valuable adjunct in the overall assessment of candidates (15 of 16, 93.8%).

CONCLUSIONS

A simple motor exercise completed over real-time telecommunication was feasible and perceived as helpful to the residency selection committee when assessing OHNS residency candidates.