Students' and Examiners' Experiences of Their First Virtual Pharmacy Objective Structured Clinical Examination (OSCE) in Australia during the COVID-19 Pandemic

Comparing an In-Person and Online Continuing Education Intervention to Improve Professional Decision-Making: A Mixed Methods Study

Purpose: This paper compares two iterations (in-person and online) of a multi-stage continuing education program for improving high-risk decision-making among mental health workers. Methods: The mixed-methods study analyzed the following: (1) physiological and psychological arousal during simulated patient interviews; (2) physiological and psychological arousal recorded during real-time decision-making over four months; and (3) thoughts on the process and outcomes of the intervention raised in reflective interviews. Findings: Quantitatively, there were no statistical differences in stress measures between in-person and online simulated interviews or decision-making logs, suggesting they were effective in eliciting reactions commonly found in challenging clinical situations. Qualitatively, participants in both iterations indicated that the intervention caused them to reflect on practice, consider a wider range of factors related to the decisions, and enact approaches to improve decision-making. Conclusions: A carefully constructed online continuing education experience can result in outcomes for experienced social workers that are equivalent to an in-person iteration.

Analysis of satisfaction levels and perceptions of clinical competency: a mixed method study on objective structured clinical examinations in undergraduate dental students

OBJECTIVE

To analyze the satisfaction levels, perceptions of developing clinical competencies through objective structured clinical examination and to explore the experiences, challenges, and suggestions of undergraduate dental students.

METHODS

The study adopted a mixed-method convergent design. Quantitative data were collected from 303 participants through surveys, evaluating satisfaction levels with objective structured clinical examination (OSCE). Additionally, qualitative insights were gathered through student focus group interviews, fundamental themes were developed from diverse expressions on various aspects of OSCE assessments. The Chi-Square tests, was performed to assess associations between variables. Data integration involved comparing and contrasting quantitative and qualitative findings to derive comprehensive conclusions.

RESULTS

The satisfaction rates include 69.4% for the organization of OSCE stations and 57.4% for overall effectiveness. However, a crucial challenge was identified, with only 36.7% of students receiving adequate post-OSCE feedback. Furthermore, a majority of students (50%) expressed concerns about the clinical relevance of OSCEs. The study showed a significant associations (p < 0.05) between satisfaction levels and years of study as well as previous OSCE experience. Student focus group interviews revealed diverse perspectives on OSCE assessments. While students appreciate the helpfulness of OSCEs, concerns were raised regarding time constraints, stress, examiner training, and the perceived lack of clinical relevance.

CONCLUSION

The students anticipated concerns about the clinical relevance of OSCEs, highlighting the need for a more aligned assessment approach. Diverse perspectives on OSCE assessments reveal perceived helpfulness alongside challenges such as lack of feedback, examiner training, time constraints, and mental stress.

Assessing the equivalency of face-to-face and online simulated patient interviews in an educational intervention

BACKGROUND

In adapting to COVID-19, many health professional training programs moved abruptly from in-person to online simulated patient interviews for teaching and evaluation without the benefit of evidence regarding the efficacy of this mode of delivery. This paper reports on a multi-methods research project comparing in-person and online simulated patient interviews conducted by allied health professionals as part of an educational intervention offered at a large university teaching hospital.

METHODS

Twenty-three participants conducted two 15-min interviews with simulated patients using previously validated scenarios of patients presenting with suicide risk. In order to assess the equivalency of the two modalities, physiological and psychological stress were measured using heart rate variability parameters and the State-Trait Anxiety Inventory respectively, and then were compared across cohorts using t-tests. Reflective interviews elicited qualitative impressions of the simulations that were subject to thematic qualitative analysis.

RESULTS

There were no statistical differences in measures of psychological stress or physiological arousal of participant health care professionals who engaged with in-person versus online simulated interviews, suggesting they were equally effective in eliciting reactions commonly found in challenging clinical situations. In reflective interviews, participants commented on the realism of both modalities of simulated patient encounters and that simulated interviews provoked emotional and physiological responses consistent with actual patient encounters.

CONCLUSIONS

These findings provide developing evidence that carefully designed online clinical simulations can be a useful tool for the education and assessment of healthcare professionals.

Impact of a Mock OSCE on Student Confidence in Applying the Pharmacists' Patient Care Process

The Medical and Pharmacy Student Collaboration (MAPSC) student organization at the University of Southern California, Alfred E. Mann School of Pharmacy and Pharmaceutical Sciences, created an extracurricular, peer-led, virtual group mock objective structured clinical examination (MOSCE) to expose first-year pharmacy students (P1s) to the Pharmacists' Patient Care Process (PPCP). The purpose of this study is to evaluate the impact of a MAPSC MOSCE on P1s self-reported confidence in applying the PPCP and on patient communication, medication knowledge, and clinical skills. An anonymous, optional, self-reported survey was administered to P1s before and after the event, where they rated their confidence on a scale of 0-100 (0 = not confident, 100 = certainly confident). The statistical analysis was a paired two-tailed t-test with a significance level of p < 0.05. A total of 152 P1s and 30 facilitators attended the MOSCE. One hundred thirty-nine students met the inclusion criteria and were included in the data analysis. There was a statistically significant difference in the change in self-reported confidence for all PPCP components and learning outcomes. The results of our study strongly indicate that introducing P1 students to the PPCP through a MAPSC MOSCE format is a valuable experience.

A Telehealth Module and Virtual Objective Structured Clinical Examination of Health Literacy in Pharmacy Education

OBJECTIVE

To implement an online health literacy (HL) module and evaluate its impact through virtual Objective Structured Clinical Examination (OSCE) on students' abilities to manage patients with low HL.

METHODS

Students virtually engaged in several learning activities related to HL, including practicing using HL assessment tools, designing an information booklet suitable for low HL patients, using readability formulas to lower text readability level to ≤ sixth grade, role-playing scenarios related to HL, and undergoing a virtual OSCE. Student performance on course assessments was tested using Spearman's rank-order. Students evaluated their OSCE experience in terms of case content, virtual assessment, and logistics; and effectiveness of the HL module and confidence related to HL.

RESULTS

A total of 90 students completed the virtual OSCE; the mean score was 8.8 out of 10 (88%), which correlated with similar course assessments. The average score was 3.46 out of 3.7 for the "gathering information" domain (recognizing risk factors and behaviors, assessing HL and adherence, etc.), and 4.06 out of 4.9 for the "patient management" domain (counseling patients on taking medications, focusing on repeating key messages, offering adherence interventions, etc.). Students responded positively regarding case content and virtual assessment, but less favorably regarding logistics. Feedback on effectiveness of the HL module and confidence in managing patients with low HL was positive.

CONCLUSION

An HL module delivered online was effective in improving student knowledge, abilities, and confidence related to HL. Students achieved high scores in all items in the virtual OSCE, and considered it a measure of communication and clinical skills that is comparable to traditional in-room assessments.

Implementation of virtual OSCE in health professions education: A systematic review

INTRODUCTION

The Objective Structured Clinical Examination (OSCE) has been widely used in health professions education since the 1970s. The global disruption caused by the COVID-19 pandemic restricted in-person assessments and medical educators globally sought alternative means to assess and certify students and trainees to meet the acute demand for health-care workers. One such solution was through virtual OSCE (vOSCE), which modified traditional in-person OSCE using videoconference platforms. This meta-ethnography sought to synthesise qualitative literature on candidates' and assessors' experiences of vOSCE to evaluate whether it may have a role in future assessment practices.

METHODS

In June 2022, we systematically searched PsycINFO, Medline and ERIC for peer-reviewed qualitative and mixed-methods articles that described candidates' and assessors' experiences of virtual OSCE in health professions education. Of 1069 articles identified, 17 were synthesised using meta-ethnography.

RESULTS

The final synthesis represented 1190 candidates and assessors from faculties of medicine, dentistry, nursing, pharmacy and osteopathy. We developed our findings into four key concepts. 'Strengthening confidence in a virtual environment' highlighted attempts to overcome and mitigate concerns associated with transitioning from in-person to virtual assessment. 'Understanding the scope of use as an assessment' reflected on the suitability of vOSCE in assessing various skills. 'Refining operational processes' emphasised the technical challenges of implementing vOSCE and impacts on accessibility and resources. 'Envisioning its future role' considered the applicability of vOSCE in the climate of rapid development in telehealth.

CONCLUSION

This meta-ethnography highlighted that although vOSCE was primarily considered a temporary and crisis response, candidates and assessors recognised positive, as well as negative, consequences of the transition towards them. Moving forward, medical education policymakers should carefully consider the extent to which elements of vOSCE could be incorporated into assessment systems, particularly in light of the rise of telehealth in clinical practice.

Student Performance on an Objective Structured Clinical Exam Delivered Both Virtually and In-Person

OBJECTIVE

Passing a milestone objective structured clinical examination (OSCE) is a graduation requirement for the University of Waterloo Pharmacy students. In January 2021, the milestone OSCE was offered concurrently both virtually and in-person, with students being able to choose their desired format. The purpose of this study was to compare student performance between the 2 formats and to identify factors that may have predicted student choice of format.

METHODS

Objective structured clinical examination scores for in-person and virtual exam-takers were compared using 2-tailed independent t tests with Bonferroni correction. Pass rates were compared using χ2 analysis. Prior academic performance variables were analyzed to identify predictors of the chosen exam format. Student and exam personnel surveys were used to capture OSCE feedback.

RESULTS

A total of 67 students (56%) participated in the in-person OSCE, and 52 students (44%) participated virtually. There were no significant differences in overall exam averages or pass rates between the 2 groups. However, virtual exam-takers scored lower in 2 of 7 cases. Previous academic performance did not predict the choice of exam format. Feedback surveys indicated that the exam organization was perceived as a strength regardless of format, but in-person students felt more prepared for the exam than virtual exam-takers with technical challenges and difficulty navigating station resources being noted as barriers in the virtual offering.

CONCLUSION

Virtual and in-person administration of a milestone OSCE resulted in similar student performance, with slightly lower performance on 2 individual case scores with virtual delivery. These results may inform the future development of virtual OSCEs.

Shifting to Authentic Assessments? A Systematic Review of Student Perceptions of High-Fidelity Assessments in Pharmacy

OBJECTIVE

Despite becoming increasingly popular, there is no comprehensive review on high-fidelity assessments in pharmacy education that has a focus on the perceptions and experiences of students. This systematic review investigates the student acceptability of high-fidelity simulation for summative assessments in pharmacy education and provides recommendations regarding high-fidelity simulation practices.

FINDINGS

The search yielded 37 studies in total. The articles fell into 3 distinct categories: objective structured clinical examinations (N = 25); face-to-face simulation assessments (N = 9), and augmented reality assessments (N = 3). Most high-fidelity assessments were well received with most students agreeing they had a vital role in assessing the application of clinical knowledge, even though they are stressful. Students prefer high-fidelity assessments to be face-to-face rather than online and also prefer when simulated patients whom they are unfamiliar with are used. Students also expressed a need to be well-prepared for the assessment regarding the logistics of the exam and technology use.

SUMMARY

High-fidelity simulation is likely to be increasingly important in the assessment of the knowledge and skills of pharmacy students and student perception is an important factor to consider when developing such assessments. Reducing stress associated with high-fidelity assessments could include familiarizing students with task logistics or technology used prior to the assessment, using external simulated patients, and having face-to-face assessments and practice sessions.

Are we ready yet for digital transformation? Virtual versus on-campus OSCE as assessment tools in pharmacy education. A randomized controlled head-to-head comparative assessment

Background

The global COVID-19 pandemic has influenced pharmacy education including learning, assessment, and exams. In the UAE, pharmacy instructors have adapted several innovative teaching methods to strive for quality learning outcomes. The current trial presented a head-to-head comparative assessment between on-campus versus virtual Objective Structured Clinical Examination (OSCE) with examiners' and students' perspectives.

Aim

The main aim was to compare fourth-year students' and examiners' perceptions of the feasibility (time and logistics), stress, performance, and satisfaction between on-campus versus virtual OSCE.

Method

A randomized controlled head-to-head comparative assessment between the On-campus and virtual OSCE was conducted to explore performance and satisfaction of pharmacy students and examiners towards the two OSCE settings. The virtual OSCE was carried out directly after the on-campus -OSCE and the setting was designed in a way that aligned with the on-campus OSCE but in a virtual way. Microsoft Teams® breakout room was used as a virtual stations. Respondus-lockdown-browse and Google Meet® were used for proctoring purposes.

Results

Students who sat for the on-campus assessment were more satisfied with the instructions, the orientation session, the time management, and the overall exam setting, the ability of the exam to assess their communication and clinical skills, professionalism and attitude, and the interactivity of the exam compared to the students who sat for the virtual assessment. Examiners' perceptions for both settings were the same with the exception of interaction with students (p less than 0.05) as the on-campus OSCE was more interactive.

Conclusion

Students still prefer the on-campus OSCE to the virtual OSCE format in many aspects. Nevertheless, virtual OSCE is still a feasible and satisfactory method of assessment when on-campus OSCE is not possible. There is a need of a specialized platform to conduct the virtual OSCE from A to Z rather than maximizing the use of options in the current digital platforms.

Feasibility of large-scale eOSCES: the simultaneous evaluation of 500 medical students during a mock examination

The COVID-19 pandemic has led health schools to cancel many on-site training and exams. Teachers were looking for the best option to carry out online OSCEs, and Zoom was the obvious choice since many schools have used it to pursue education purposes.

METHODS

We conducted a feasibility study during the 2020-2021 college year divided into six pilot phases and the large-scale eOSCEs on Zoom on June 30th, 2021. We developed a specific application allowing us to mass create Zoom meetings and built an entire organization, including a technical support system (an SOS room and catching-up rooms) and teachers' training sessions. We assessed satisfaction via an online survey.

RESULTS

On June 30th, 531/794 fifth-year medical students (67%) participated in a large-scale mock exam distributed in 135 Zoom meeting rooms with the mobilization of 298 teachers who either participated in the Zoom meetings as standardized patients (N =135, 45%) or examiners (N =135, 45%) or as supervisors in the catching-up rooms (N =16, 6%) or the SOS room (N =12, 4%). In addition, 32/270 teachers (12%) experienced difficulties connecting to their Zoom meetings and sought the help of an SOS room member. Furthermore, 40/531 students (7%) were either late to their station or had technical difficulties and declared those issues online and were welcomed in one of the catching-up rooms to perform their eOSCE stations. Additionally, 518/531 students (98%) completed the entire circuit of three stations, and 225/531 students (42%) answered the online survey. Among them, 194/225 (86%) found eOSCES helpful for training and expressed their satisfaction with this experience.

CONCLUSION

Organizing large-scale eOSCEs on Zoom is feasible with the appropriate tools. In addition, eOCSEs should be considered complementary to on-site OSCEs and to train medical students in telemedicine.

Supporting pharmacy students' preparation for an entry-to-practice OSCE using video cases

BACKGROUND AND PURPOSE

Objective structured clinical examinations (OSCEs) are widely used, including in licensure examinations. OSCEs assess diverse skills in a structured manner, but can be stressful for students and have a significant organisational burden for faculty. Case-based video OSCE preparation resources were developed for students preparing for a licensure OSCE. The study aimed to examine student engagement, compare scores awarded by students to performances at specific competence standards with faculty scores, and examine usability, usefulness, and acceptability of the video cases.

EDUCATIONAL ACTIVITY AND SETTING

Final year pharmacy students (n = 149) enrolled in an integrated master of pharmacy programme in Ireland were invited to participate. Six sets of recorded OSCE-based video cases were developed, comprising multiple recordings of the same case scenario, with each pitched at a different level of performance. Students watched and scored the video cases. Usability, usefulness, and acceptability were evaluated via questionnaire.

FINDINGS

One or more video cases were accessed by 70.5% of students. Score ratings awarded by the students, when compared to faculty ratings, showed an overall trend towards inter-rater agreement between students and faculty. Students felt positively in terms of the usability, usefulness, and acceptability of the videos.

SUMMARY

Video cases designed to support OSCE preparation for a high-stakes national pharmacy licensure examination were widely used by students and were perceived to be usable, useful, and acceptable. Such video cases may be a feasible alternative to additional mock OSCEs to support student preparation.

Students and examiners perception on virtual medical graduation exam during the COVID-19 quarantine period: A cross-sectional study

BACKGROUND

With the emergence of the COVID-19 pandemic and lockdown approach that was adopted all over the world, conducting assessments while maintaining integrity became a big challenge. This article aims at sharing the experience of conducting an online assessment with the academic community and to assess its effectiveness from both examiners' and students' perspectives.

METHODS

An online assessment was carried out for the final year medical students of Hawler Medical University/Iraq during the lockdown period of the COVID-19 pandemic, June 2020. Then, an online questionnaire was sent to a sample of 61 examiners and 108 students who have been involved in evaluating the mentioned assessment process. Mann-Whitney and Kruskal-Wallis tests were used to compare the mean ranks of the overall satisfaction scores between categories of the students and examiners. Categorical data were summarized and presented as frequencies and percentages.

RESULTS

The response rates among examiners and students were 69.4% and 88.5% respectively. The majority of the examiners were generally satisfied with the online examination process compared to only around a third of the students. However, both examiners and students agreed that online examination was not suitable for assessing the physical examination skills.

CONCLUSION

The online assessment can be considered a good alternative and acceptable method for medical students' assessment in unpredicted emergencies, yet it was not applicable in testing physical examination skills.

COVID-19-Driven Improvements and Innovations in Pharmacy Education: A Scoping Review

The COVID-19 pandemic led to many colleges of pharmacy having to make major changes relating to their infrastructure and delivery of their curriculum within a very short time frame, including the transition of many components to an online setting. This scoping review sought to summarize what is known about the impact of COVID-19 on pharmacy education and the effectiveness of adaptation strategies which were put in place. PubMed, Web of Science, OVID Medline, and MedEdPortal were searched to identify pharmacy education-related articles published since the beginning of the COVID-19 pandemic. For article inclusion, the following criteria had to be met: described original research, related directly to PharmD or PharmBS education, related to the impact of COVID-19 on pharmacy education, and was available in English. Out of a total of 813 articles, 50 primary research articles were selected for inclusion. Our review of these identified four domains relating to the impact of COVID-19 on pharmacy education and/or effectiveness of adaptation strategies: (1) lab-based courses and activities (including interprofessional education activities), (2) experiential education, (3) didactic education, and (4) student well-being. The key research findings are summarized and discussed. While the COVID-19 pandemic has clearly brought many challenges to pharmacy education, it has also led to key improvements and innovations.