Is There Variability in Scoring of Student Surgical OSCE Performance Based on Examiner Experience and Expertise?

[Systematic analysis of evaluations of Objective Structured Clinical Exam (OSCE) multiple circuits: Explanatory variables and inter-rater correlations]

INTRODUCTION

Objective Structured Clinical Examinations (OSCEs) assess professional performance in a simulated environment. Following their integration into the reform of the 2nd cycle of medical studies (R2C), this pedagogical modality was implemented in France. This study investigates the variability of students' OSCE scores, as well as their inter-rater reproducibility.

METHODS

This single-center retrospective study covered several sessions of evaluative OSCE circuits conducted between January 2022 and June 2023. Variables collected were: baseline situation family, competency domain, presence of a standardized participant for stations; gender and professional status for evaluators; scores (global, clinical and communication skills), number of previously completed OSCE circuits and faculty scores for students.

RESULTS

The variability of the overall score was explained mainly (79.7%, CI95% [77.4; 82.0]) by the station factor. The student factor and the circuit factor explained 7.5% [12.9; 20.2] and<0.01% [2.10-13; 2.10-9] respectively. The inter-rater intra-class correlation coefficient was 87.2% [86.4; 87.9] for the global score. Station characteristics (starting situation, domain) and evaluator characteristics (gender, status) were significantly associated with score variations.

CONCLUSION

This first study on the variability of OSCE circuit scores in France shows good reproducibility with influence of station characteristics. In order to standardize circuits, variability linked to the domain competency should be considered as well.

The influence of candidates' race on examiners' ratings in standardised assessments of clinical practice

PURPOSE

Delivering fair and reliable summative assessments in medical education assumes examiner decision making is devoid of bias. We investigated whether candidate racial appearances influenced examiner ratings in undergraduate clinical exams.

METHODS

We used an internet-based design. Examiners watched a randomised set of six videos of three different white candidates and three different non-white (Asian, black and Chinese) candidates taking a clinical history at either fail, borderline or pass grades. We compared the median and interquartile range (IQR) of the paired difference between scores for the white and non-white candidates at each performance grade and tested for statistical significance.

RESULTS

160 Examiners participated. At the fail grade, the black and Chinese candidates scored lower than the white candidate, with median paired differences of -2.5 and -1 respectively (both p < 0.001). At the borderline grade, the black and Chinese candidates scored higher than the white candidate, with median paired differences of +2 and +3, respectively (both p < 0.001). At the passing grade, the Asian candidate scored lower than the white candidate (median paired difference -1, p < 0.001).

CONCLUSION

The racial appearance of candidates appeared to influence the scores awarded by examiners, but not in a uniform manner.

Interrater reliability in neurology objective structured clinical examination across specialties

PURPOSE

To assess interrater reliability and examiners' characteristics, especially specialty, associated with scoring of neurology objective structured clinical examination (OSCE).

MATERIAL AND METHODS

During a neurology mock OSCE, five randomly chosen students volunteers were filmed while performing 1 of the 5 stations. Video recordings were scored by physicians from the Lyon and Clermont-Ferrand university teaching hospitals to assess students performance using both a checklist scoring and a global rating scale. Interrater reliability between examiners were assessed using intraclass coefficient correlation. Multivariable linear regression models including video recording as random effect dependent variable were performed to detect factors associated with scoring.

RESULTS

Thirty examiners including 15 (50%) neurologists participated. The intraclass correlation coefficient of checklist scores and global ratings between examiners were 0.71 (CI95% [0.45-0.95]) and 0.54 (CI95% [0.28-0.91]), respectively. In multivariable analyses, no factor was associated with checklist scores, while male gender of examiner was associated with lower global rating (β coefficient = -0.37; CI 95% [-0.62-0.11]).

CONCLUSIONS

Our study demonstrated through a video-based scoring method that agreement among examiners was good using checklist scoring while moderate using global rating scale in neurology OSCE. Examiner's specialty did not affect scoring whereas gender was associated with global rating scale.

Interactions between the sex of the clinician grader and the sex of the chiropractic student intern on spinal manipulation assessment grade

OBJECTIVE

The purpose of this project was to determine if there was any relationship between the sex of the clinician grader and the sex of the chiropractic student intern on student spinal manipulation assessment grades.

METHODS

Twelve thousand six hundred and thirty-one supervised patient adjustments by student interns were analyzed over a 3-year data collection window. Student interns were assessed by multiple male and female clinicians in a teaching clinic using a modified Dreyfus model scoring system on a 1-4 scale (1 = novice, 4 = proficient). A Mann-Whitney U test was used to compare the relationship between grader sex and student grade as well as student sex and student grade.

RESULTS

Sex of the grader had a statistically significant effect on spinal manipulation assessment grade, p < .001, with male clinician graders assigning average scores of 2.81 ± 0.39 (mean ± SD) and female clinician graders scores of 3.01 ± 0.52, r = .18. Sex of the student had a statistically significant but negligible (r = .08) effect on spinal manipulation assessment grade, p < .001, with male students averaging slightly higher scores (2.93 ± 0.47) than females (2.86 ± 0.44) on the modified Dreyfus scale.

CONCLUSION

Male clinicians tended to assign lower grades on spinal manipulation assessments than female clinicians. Male students on average received slightly higher scores than female students on spinal manipulation assessments.

The Role of Videotaped Clinical Skills Aggregated Peer Evaluations in the Enhancement of Evaluation Skills of Individual Medical School Faculty Members

The conventional methodology for appraising medical students has some limitations like inherent subjectivity, unstructured nature, and bias. Implementing the Objective Structured Clinical Examination (OSCE) can mitigate these shortcomings. However, the OSCE presents challenges, including substantial financial costs and time-intensive processes, particularly when assessing a large cohort of students. Consequently, an alternative assessment was needed to keep the advantages of OSCE and mask its limitations, especially in resource-constrained settings. In addition, many scholars have expressed concern over medical students' and interns' inadequate interviewing and physical examination competencies in recent years. Due to easy availability, videotaping is a convenient method for objectively observing students for aggregate review by faculty in order to ascertain what exactly faculty are assessing during medical student evaluations. This technique allows for aggregate faculty group improvement in the ability of educators to assess students' technical proficiency, data collection capabilities, standardized patient interaction demeanor, and strategies for fostering standardized patient comfort. Nonetheless, aggregate evaluation of videotape recordings for faculty assessment development or reliance on verbal feedback from medical students about faculty's ability to assess student skill is a matter of debate. Due to increased subspecialization, the subspecialist or specialist examiners face difficulties in assessing students' skills in specific and/or generalized domains. Despite these challenges, assessing the ability of faculty members' observation and subsequent evaluation of medical trainees remains a vital aspect of assessment throughout various specialties. This paper presents the concept of faculty members individually observing and rating premade recordings of standardized students performing clinical skills for evaluation so that when the individual faculty members' ratings are aggregated and summarized collectively for evaluation by the faculty members as a group, both the group and the individual faculty members will gain a greater understanding of what are appropriate ratings versus outlier ratings.

A resource efficient and reliable standard setting method for OSCEs: Borderline regression method using standardized patients as sole raters in clinical case encounters with medical students

Finding a reliable, practical and low-cost criterion-referenced standard setting method for performance-based assessments has proved challenging. The borderline regression method of standard setting for OSCEs has been shown to estimate reliable scores in studies using faculty as raters. Standardized patients (SPs) have been shown to be reliable OSCE raters but have not been evaluated as raters using this standard setting method. Our study sought to find whether SPs could be reliably used as sole raters in an OSCE of clinical encounters using the borderline regression standard setting method.SPs were trained for on a five-point global rating scale. In an OSCE for medical students, SPs completed skills checklists and the global rating scale. The borderline regression method was used to create case passing scores. We estimated the dependability of the final pass or fail decisions and the absolute dependability coefficients for global ratings, checklist scores, and case pass-score decisions using generalizability theory.The overall dependability estimate is 0.92 for pass or fail decisions for the complete OSCE. Dependability coefficients (0.70-0.86) of individual case passing scores range demonstrated high dependability.Based on our findings, the borderline regression method of standard setting can be used with SPs as sole raters in a medical student OSCE to produce a dependable passing score. For those already using SPs as raters, this can provide a practical criterion-referenced standard setting method for no additional cost or faculty time.

A serious game for online-based objective structured clinical examination in nursing: A qualitative study

BACKGROUND

The current pandemic has led to unprecedented scenarios worldwide. In this context, educational institutions had to move abruptly from in-person to online classes, having to be flexible and innovative, one of the main concerns being the semester-end examinations in practical modules.

OBJECTIVE

This study aimed to explore nursing students' perceptions of the use of a serious game-like model in their final online objective structured clinical examination (OSCE).

DESIGN

An exploratory phenomenological study was conducted during the COVID-19 pandemic, in particular during June 2020.

SETTINGS

This study took place at the University of Almeria with nursing students enrolled in a clinical placement module.

PARTICIPANTS

Eighty-nine fourth-year nursing students took part in this study.

METHODS

Following the educational principles of the OSCE, a virtual OSCE was designed, using the Genially platform, an online platform for creating interactive content, to set up an assessment as a story-telling game-like model. A total of 5 focus groups (FGs) were conducted afterwards via the GoogleMeet platform. In addition, 30 semi-structured interviews were performed via the same platform. The data was analysed using a content analysis approach and supported by the ATLAS.ti 8.4 software.

RESULTS

Our finding revealed 2 main themes and 4 sub-themes. The two main themes were (i) generating emotions and feelings in times of virtuality; including emotions and feelings experienced by students during their online assessment process, and (ii) online assessment: a potential alternative to educational barriers; describing the variability of traditional modality, its implications for learning and the acquisition of competences.

CONCLUSIONS

Serious game-like models, such as the story-telling game proposed, as part of their online OSCE assessment appear to be an appropriate alternative assessment method for face-to-face approaches. This paper adds new evidence on the use of innovative and state-of-art resources as part of nursing OSCE assessments in a new reality for most students and teachers. Serious game-like models in online OSCE may empower students and help them to remove perceived barriers in face-to-face assessments.

Factors affecting test anxiety: a qualitative analysis of medical students' views

BACKGROUND

Medical students are vulnerable to test anxiety (TA), which impacts their professional lives and jeopardizes the optimal health care of their patients. The qualitative exploration of TA among medical students is crucial to understanding the problem. Hence, this study examined medical students' insights into TA and their suggestions on how to reduce it.

METHODS

We conducted a phenomenological study on medical students at a public university. We utilized focus group discussions (FGDs) to investigate their experiences of TA. The FGDs were transcribed verbatim, and these transcripts were analyzed using Atlas.ti software. The thematic analysis followed the recommended guidelines.

RESULTS

Seven FGD sessions were conducted with 45 students. Three major themes emerged: the students, their academic resources, and the examiner. Each theme comprised mutually exclusive subthemes. The "students" theme was divided into negative vs. positive thoughts and self-negligence vs. self-care, "academic recources" into heavy curriculum vs. facilitative curricular aids, and "examiner" into criticism vs. feedback and strict vs. kind approaches.

CONCLUSION

This study provides a solid foundation for policymakers and decision makers in medical education to improve current assessment practices and student well-being. Medical students will be able to significantly alter and reduce TA if they are provided with additional psychological support and their examiners are trained on how to deal with examinees.