Feasibility and acceptability of virtual mock oral examinations for senior vascular surgery trainees and implications for the certifying exam

Feasibility and Acceptability of Web-Based Structured Oral Examinations for Postgraduate Certification: Mixed Methods Preliminary Evaluation

BACKGROUND

The COVID-19 pandemic disrupted postgraduate certification examinations globally. The Colleges of Medicine of South Africa continued hosting certification examinations through the pandemic. This was achieved by effecting a rapid transition from in-person to web-based certification examinations.

OBJECTIVE

This formative evaluation explored candidates' acceptability of web-based structured oral examinations (SOEs) hosted via Zoom (Zoom Communications Inc). We also reported the audiovisual quality and technical challenges encountered while using Zoom and candidates' overall experience with these examinations conducted during the early part of the COVID-19 pandemic. Additionally, performance in web-based certification examinations was compared with previous in-person certification examinations.

METHODS

This mixed methods, single-arm evaluation anonymously gathered candidates' perceptions of web-based SOE acceptability, audiovisual quality, and overall experience with Zoom using a web-based survey. Pass rates of web-based and previous in-person certification examinations were compared using chi-square tests, with a Yates correction. A thematic analysis approach was adopted for qualitative data.

RESULTS

Between June 2020 and June 2021, 3105 candidates registered for certification examinations, 293 (9.4%) withdrew, 2812 (90.6%) wrote, and 2799 (99.9%) passed, and 1525 (54.2%) were invited to a further web-based SOE. Examination participation was 96.2% (n=1467). During the first web-based examination cycle (2020), 542 (87.1%) of 622 web-based SOE candidates completed the web-based survey. They reported web-based SOEs as fair (374/542, 69%) and adequately testing their clinical reasoning and insight (396/542, 73.1%). Few would have preferred real patient encounters (173/542, 31.9%) or in-person oral examinations (152/542, 28%). Most found Zoom acceptable (434/542, 80%) and fair (396/542, 73.1%) for hosting web-based SOEs. SOEs resulted in financial (434/542, 80%) and time (428/542, 79%) savings for candidates. Many (336/542, 62%) supported the ongoing use of web-based certification examinations. Only 169 technical challenges in using Zoom were reported, which included connectivity-related issues, poor audio quality, and poor image quality. The thematic analysis identified 4 themes of positive and negative experiences related to web-based SOE station design and content, examination station environment, examiner-candidate interactions, and personal benefits for candidates. Our qualitative analysis identified 10 improvements for future web-based SOEs. Candidates achieved high pass rates in web-based certification examinations in 2020 (1583/1732, 91.39%) and 2021 (850/1067, 79.66%). These were significantly higher (2020: N=8635; χ21=667; P<.001; 2021: N=7988; χ21=178; P<.001) than the previous in-person certification examination pass rate of 58.23% (4030/6921; 2017-2019).

CONCLUSIONS

Web-based SOEs conducted by the Colleges of Medicine of South Africa during the COVID-19 pandemic were well received by candidates, and few technical difficulties were encountered while using Zoom. Better performance was observed in web-based examinations than in previous in-person certification examinations. These early findings support the ongoing use of this assessment method.

Innovations in assessment in health professions education during the COVID-19 pandemic: A scoping review

BACKGROUND

After an initial period of uncertainty during the COVID-19 pandemic, medical universities responded to the pandemic by innovating their methods of teaching-learning and assessment. This scoping review aimed to identify innovations in assessment in medical, dental and nursing education at the undergraduate and postgraduate levels during the pandemic.

METHODS

This review was conducted utilising Arksey and O'Malley's framework where three electronic databases-PubMed, Embase and Web of Science-were systematically searched in February 2022 for articles in English describing innovations in assessments published in or after January 2020.

RESULTS

A total of 70 articles were included in the review. Among the primary research articles, 82.1% reported online assessment, 76.1% were related to medical speciality, 74.6% described assessment at the undergraduate level, 68.7% described summative assessment and 58.2% reported assessment of practical skills. All assessments of theoretical knowledge were done virtually with MCQ being the most used assessment tool and virtual OSCE being the most popular assessment tool to assess practical skills; however, it was acknowledged that the assessment of procedural skills was its limitation. Although the studies described the newer forms of assessments as feasible and acceptable, few studies reported the reliability and validity of the assessments.

CONCLUSIONS

This scoping review identified several innovations in assessment during the COVID-19 pandemic with an overall shift towards virtual or hybrid forms of assessment. However, there was a dearth of evidence regarding the effectiveness of these assessment tools suggesting a need for more research before the tools can be implemented post-pandemic.

Virtual OSCE examinations during COVID-19 A 360 satisfaction assessment from examiners and candidates

INTRODUCTION

We sought to determine the satisfaction rates of examiners and candidates in a virtual Objective Structured Clinical Exam (OSCE) of graduating Canadian urology residents.

METHODS

An annual mock exam, aimed at simulating the licencing urology exam for Canadian graduates, was moved to an online format for the 2020 cohort. This exam consists of an OSCE, and a written multiple-choice exam. The Telemedicine Satisfaction Questionnaire (TSQ), a previously validated tool for clinical encounters with three sub-domains (quality of care provided, similarity to face-to-face encounter, and perception of the interaction) was modified for the purposes of evaluating the OSCE encounter. The TSQ was sent electronically to all examiners and candidates after the exam.

RESULTS

There were 14/16 responses from examiners (87.5%) and 24/39 responses from candidates (61.5%). Overall, the format was judged to be a good experience by 13/14 (92.9%) of examiners and 21/24 (87.5%) of candidates; however, when asked specifically if the virtual OSCE was an acceptable way to determine a candidate's competency to practice urology independently, only 8/14 (57.1%) of examiners and 15/24 (62.5%) of candidates agreed.

CONCLUSIONS

This study demonstrates an overall good satisfaction rate among both examiners and candidates when using a teleconference format for a mock OSCE.

The silver lining of the pandemic in surgical education: virtual surgical education and recommendations for best practices

Virtual education is an evolving field within the realm of surgical training. Since the onset of the COVID-19 pandemic, the application of virtual technologies in surgical education has undergone significant exploration and advancement. While originally developed to supplement in-person curricula for the development of clinical decision-making, virtual surgical education has expanded into the realms of clinical decision-making, surgical, and non-surgical skills acquisition. This manuscript aims to discuss the various applications of virtual surgical education as well as the advantages and disadvantages associated with each education modality, while offering recommendations on best practices and future directions.

Health worker education during the COVID-19 pandemic: global disruption, responses and lessons for the future-a systematic review and meta-analysis

BACKGROUND

This systematic review and meta-analysis identified early evidence quantifying the disruption to the education of health workers by the COVID-19 pandemic, ensuing policy responses and their outcomes.

METHODS

Following a pre-registered protocol and PRISMA/AMSTAR-2 guidelines, we systematically screened MEDLINE, EMBASE, Web of Science, CENTRAL, clinicaltrials.gov and Google Scholar from January 2020 to July 2022. We pooled proportion estimates via random-effects meta-analyses and explored subgroup differences by gender, occupational group, training stage, WHO regions/continents, and study end-year. We assessed risk of bias (Newcastle-Ottawa scale for observational studies, RοB2 for randomized controlled trials [RCT]) and rated evidence certainty using GRADE.

RESULTS

Of the 171 489 publications screened, 2 249 were eligible, incorporating 2 212 observational studies and 37 RCTs, representing feedback from 1 109 818 learners and 22 204 faculty. The sample mostly consisted of undergraduates, medical doctors, and studies from institutions in Asia. Perceived training disruption was estimated at 71.1% (95% confidence interval 67.9-74.2) and learner redeployment at 29.2% (25.3-33.2). About one in three learners screened positive for anxiety (32.3%, 28.5-36.2), depression (32.0%, 27.9-36.2), burnout (38.8%, 33.4-44.3) or insomnia (30.9%, 20.8-41.9). Policy responses included shifting to online learning, innovations in assessment, COVID-19-specific courses, volunteerism, and measures for learner safety. For outcomes of policy responses, most of the literature related to perceptions and preferences. More than two-thirds of learners (75.9%, 74.2-77.7) were satisfied with online learning (postgraduates more than undergraduates), while faculty satisfaction rate was slightly lower (71.8%, 66.7-76.7). Learners preferred an in-person component: blended learning 56.0% (51.2-60.7), face-to-face 48.8% (45.4-52.1), and online-only 32.0% (29.3-34.8). They supported continuation of the virtual format as part of a blended system (68.1%, 64.6-71.5). Subgroup differences provided valuable insights despite not resolving the considerable heterogeneity. All outcomes were assessed as very-low-certainty evidence.

CONCLUSION

The COVID-19 pandemic has severely disrupted health worker education, inflicting a substantial mental health burden on learners. Its impacts on career choices, volunteerism, pedagogical approaches and mental health of learners have implications for educational design, measures to protect and support learners, faculty and health workers, and workforce planning. Online learning may achieve learner satisfaction as part of a short-term solution or integrated into a blended model in the post-pandemic future.

Cross-institutional Virtual Mock Oral Examination: A New Paradigm?

Mock oral examinations (MOEs) are shown to be valuable in preparation for national board oral examinations. Although traditionally conducted in-person, improved technological advances have supported exploring virtual alternatives. Furthermore, the cornoavirus disease 2019 pandemic has necessitated virtual learning. We hypothesized that the virtual platform would improve and expand the MOE experience to include collaborative institutions while approximating a board examination environment.

Methods

Sixteen senior plastic surgery residents and 14 faculty from three separate programs participated in a cross-institutional virtual MOE. Over a single day, each trainee was evaluated by two faculty from a separate institution on six scenarios, including digitally interactive photographs. Immediate postexamination debriefing with feedback was performed. All participants were subsequently invited to complete an anonymous survey regarding MOE experience, accuracy, and stress level, with responses graded on a Likert scale (1-5).

Results

Twenty-three participants completed the survey, with 87% having prior MOE experience; however, only 26.1% of participants had virtual MOE experience. Most found the virtual platform more convenient (4.18 + 1.18; 5-far more convenient, 1-far less convenient) and less stressful (2.32 + 0.65; 5-far more stressful, 1-far less stressful) than in-person. All participants found the examination fair, and participants found the examination valuable (4.65 + 0.57) in preparing trainees for the American Board of Plastic Surgery oral examination.

Conclusions

The majority of participants found the virtual MOE experience valuable and comparable to in-person. These results reinforce that a virtual MOE provides an acceptable alternative with greater convenience and improved cross-institutional collaborative efforts.

The Advanced Certifying Exam Simulation-Pro assessment instrument: evaluating surgical trainee examsmanship in virtual oral exams

Purpose

In response to the COVID-19 pandemic, many educational activities in general surgery residency have shifted to a virtual environment, including the American Board of Surgery (ABS) Certifying Exam. Virtual exams may become the new standard. In response, we developed an evaluation instrument, the ACES-Pro, to assess surgical trainee performance with a focus on examsmanship in virtual oral board examinations. The purpose of this study was two-fold: (1) to assess the utility and validity of the evaluation instrument, and (2) to characterize the unique components of strong examsmanship in the virtual setting, which has distinct challenges when compared to in-person examsmanship.

Methods

We developed a 15-question evaluation instrument, the ACES-Pro, to assess oral board performance in the virtual environment. Nine attending surgeons viewed four pre-recorded oral board exam scenarios and scored examinees using this instrument. Evaluations were compared to assess for inter-rater reliability. Faculty were also surveyed about their experience using the instrument.

Results

Pilot evaluators found the ACES-Pro instrument easy to use and felt it appropriately captured key professionalism metrics of oral board exam performance. We found acceptable inter-rater reliability in the domains of verbal communication, non-verbal communication, and effective use of technology (Guttmann's lambda-2 were 0.796, 0.916, and 0.739, respectively).

Conclusions

The ACES-Pro instrument is an assessment with evidence for validity as understood by Kane's framework to evaluate multiple examsmanship domains in the virtual exam setting. Examinees must consider best practices for virtual examsmanship to perform well in this environment.

Supplementary Information

The online version contains supplementary material available at 10.1007/s44186-023-00107-7.

Feasibility of radiology online structured oral examination for undergraduate medical students

Background

Online summative assessment has emerged during the COVID-19 pandemic as an alternative to traditional examinations, bringing opportunities and challenges. The study aims to evaluate the feasibility and effectiveness of online structured oral examination (SOE) in radiology clerkships. The study identifies measures taken to successfully implement online SOE and minimize chances of cheating. It also discusses the challenges encountered and how they were addressed.

Methods

SOE percent scores of fourth-year medical students from two institutions were correlated with students’ grade point average (GPA). The scores were compared among different institutions, students’ genders, students’ batches, examination versions, and examiners with different experience levels. Students’ perceived satisfaction and concerns were captured using anonymous self-administered questionnaire. Technical problems and success rate of SOE implementation were recorded. Results were analyzed using descriptive and inferential statistics.

Results

A total of 79 students participated in the study, out of which 81.0% (n = 64) responded to the survey. SOE scores showed poor positive correlation with the students’ GPAs (r = 0.22, and p = .09). Scores showed no significant difference between the two institutions or genders. Scores were also not significantly different between students who were examined by junior or senior examiners. All but one version of examination showed no significant difference in students’ scores. No significant difference was observed in students’ scores between each two subsequent batches who were exposed to the same examination version.

Conclusion

Online summative SOE is a feasible alternative whenever face-to-face SOE could not be implemented provided that appropriate measures are taken to ensure its successful execution.

Implementation of the Vascular Surgery Board Virtual Certifying Examination

Objective

The onset of the COVID-19 (coronavirus disease 2019) pandemic mandated postponement of the in-person Vascular Surgery Board 2020 certifying examination (CE). Vascular surgery virtual CEs (VVCEs) were developed for the scheduled 2020 CEs (rescheduled to January 2021) and 2021 CEs (rescheduled to July 2021) to avoid postponing the certification testing. In the present study, we have reported the development, implementation, and outcomes of the first two VVCEs.

Methods

The VVCE was similar to the in-person format (three 30-minutes sessions, two examiners, four questions) but required a proctor and a host. In contrast to the general surgery VCEs, the VVCE also incorporated images. The candidates and examiners were instructed on the format, and technology checks were performed before the VVCE. The candidates were given the opportunity to invalidate their examination for technology-related reasons immediately after the examination. Postexamination surveys were administered to all the participants.

Results

The VVCEs were completed by 356 of 357 candidates (99.7%). The pass rates for the January 2021 and July 2021 examinations were 97.6% (first time, 99.4%; retake, 70%) and 94.7% (first time, 94.6%; retake, 100%), respectively. The pass rates were not significantly different from the 2019 in-person CE (χ² = 2.30; P = .13; and χ² = 0.01; P = .91, for the January 2021 and July 2021 examinations, respectively). None of the candidates had invalidated their examination. The candidates (162 of 356; 46%), examiners (64 of 118; 54%), proctors (25 of 27; 93%), and hosts (8 of 9; 89%) completing the survey were very satisfied with the examination (Likert score 4 or 5: candidates, 92.6%; noncandidates, 96.9%) and found the technology domains (Zoom, audio, video, viewing images) to be very good (Likert score 4 or 5), with candidate and other responder scores of 73% to 84% and >94%, respectively. Significantly more of the candidates had favored a future VVCE compared with the examiners (87% vs 32%; χ² = 67.1; P < .001). The free text responses from all responders had commented favorably on the organization and implementation of the examination. However, some candidates had expressed concerns about image sizes, and some examiners had expressed concern about the time constraints for the question format. The candidates appreciated the convenience of an at-home examination, especially the avoidance of travel costs.

Conclusions

The two Vascular Surgery Board VCEs were shown to be psychometrically sound and were overwhelmingly successful, demonstrating that image-based virtual examinations are feasible and could become the standard for the future.