Examiner effect on the objective structured clinical exam - a study at five medical schools

Variance due to the examination conditions and factors associated with success in objective structured clinical examinations (OSCEs): first experiences at Paris-Saclay medical school

BACKGROUND

We aimed to measure the variance due to examination conditions during the first sessions of objective structured clinical examinations (OSCEs) performed at a French medical school and identify factors associated with student success.

METHODS

We conducted a retrospective, observational study using data from the first three OSCEs sessions performed at Paris-Saclay medical school in 2021 and 2022. For all sessions (each organized in 5 parallel circuits), we tested a circuit effect using a linear mixed-effects model adjusted for sex and the average academic level of students (according to written tests). Then, we studied the factors associated with student success at one station using a multivariate linear mixed-effects model, including the characteristics of students, assessors, and standardized patients.

RESULTS

The study included three OSCEs sessions, with 122, 175, and 197 students and a mean (± SD) session score of 13.7(± 1.5)/20, 12.7(± 1.7)/20 and 12.7(± 1.9)/20, respectively. The percentage of variance due to the circuit was 6.5%, 18.2% (statistically significant), and 3.8%, respectively. For all sessions, the student's average level and station scenario were significantly associated with the score obtained in a station. Still, specific characteristics of assessors or standardized patients were only associated with the student's score in April 2021 (first session).

CONCLUSION

The percentage of the variance of students' performance due to the examination conditions was significant in one out of three of the first OSCE sessions performed at Paris-Saclay medical school. This result seems more related to individual behaviors rather than specific characteristics of assessors or standardized patients, highlighting the need to continue training teaching teams.

NATIONAL CLINICAL TRIAL NUMBER

Not applicable.

[Success factors and obstacles in the implementation of competence-oriented teaching in surgery]

BACKGROUND AND OBJECTIVES

For successful competence-oriented teaching at the medical faculties it is important to identify the factors that influence its implementation in order to benefit from the strengths and balance out weaknesses. The present study examined the success factors and obstacles of the implementation of competence-oriented teaching in the surgical discipline from the point of view of students and lecturers.

METHODS

After implementation of competence-oriented teaching based on the teaching goals of the NKLM, in clinical examination courses (bedside teaching and block internship, BP) at two hospitals, a qualitative content analysis and quantification of the answers were performed using focus group interviews and questionnaires with students (S) and lecturers (D).

RESULTS

During the summer semester 2022 a total of 31 students and 14 lecturers were interviewed in focus groups and 143 questionnaires (123 S, 20 D) were analyzed. For the students the presence of concrete competences/teaching goals, guidelines for the lesson, transparent goals and ability to demand teaching goals as well as structured lessons and mentoring were the main success factors. Lecturers on the other hand reported the presence of concrete goals, assistance for the lesson preparation and the activity of the students as success factors. The results of the questionnaires showed that the majority (88% S, 75% D) were informed about the teaching goals and considered them to be followed (84%S, 95% D). Obstacles were the factors "time", "mentoring" and "information". Factors that were between negative and positive (indifferent factors) were "uncertainty about competence-orientation" and "uncertainty how to examine the teaching success".

DISCUSSION

Transparent structure and teaching goals as well as a mentoring system are the success factors for the implementation of competence-oriented lessons and should be used as strengths. Indifferent factors represent weaknesses and need to be addressed by training and instruction. Restricted time and personnel resources are the immanent problems that hamper the implementation and require fulminant structural changes.

[Systematic analysis of evaluations of Objective Structured Clinical Exam (OSCE) multiple circuits: Explanatory variables and inter-rater correlations]

INTRODUCTION

Objective Structured Clinical Examinations (OSCEs) assess professional performance in a simulated environment. Following their integration into the reform of the 2nd cycle of medical studies (R2C), this pedagogical modality was implemented in France. This study investigates the variability of students' OSCE scores, as well as their inter-rater reproducibility.

METHODS

This single-center retrospective study covered several sessions of evaluative OSCE circuits conducted between January 2022 and June 2023. Variables collected were: baseline situation family, competency domain, presence of a standardized participant for stations; gender and professional status for evaluators; scores (global, clinical and communication skills), number of previously completed OSCE circuits and faculty scores for students.

RESULTS

The variability of the overall score was explained mainly (79.7%, CI95% [77.4; 82.0]) by the station factor. The student factor and the circuit factor explained 7.5% [12.9; 20.2] and<0.01% [2.10-13; 2.10-9] respectively. The inter-rater intra-class correlation coefficient was 87.2% [86.4; 87.9] for the global score. Station characteristics (starting situation, domain) and evaluator characteristics (gender, status) were significantly associated with score variations.

CONCLUSION

This first study on the variability of OSCE circuit scores in France shows good reproducibility with influence of station characteristics. In order to standardize circuits, variability linked to the domain competency should be considered as well.

A Look at Demographics and Transition to Virtual Assessments: An Analysis of Bias in the American Board of Surgery General Surgery Certifying Exams

OBJECTIVE

The goals of this study were (1) to assess if examiner ratings in the American Board of Surgery (ABS) General Surgery Cetifying Exam (CE) are biased based on the gender, race, and ethnicity of the candidate or the examiners, and (2) if the format of delivering of the exams, in-person or virtual, affects how examiners rate candidates.

DESIGN

We included every candidate-examiner combination for first time takers of the general surgery oral exam. Total scores and pass/fail outcomes based on the 4 scores given by examiners to candidates were analyzed using multilevel models, with candidates as random effects. Explanatory variables included the gender, race, and ethnicity of candidates and examiners, and the format of the exam (in-person or virtual). Candidates' first attempt scores on the ABS General Surgery Qualifying Exam (QE) were also included in the models to control for the baseline knowledge of the candidate. Three sets of models were evaluated for each demographic variable (gender, race, ethnicity) due to missingness in data. p-values and coefficients of determination R2 were used to quantify the statistical and practical significance of the model coefficients (an existent relationship between the explored variables on CE scores was considered statistically and practically significant if the p-value was lower than 0.01 and R2 higher than 1%).

PARTICIPANTS

All first-time takers of the American Board of Surgery General Surgery Certifying Exam from 2016 to 2022 that had demographic data, and the examiners that participated in those exams.

RESULTS

The number of candidates/examiners for the 3 sets of models was 8665/514 (gender), 5906/465 (race), and 4678/295 (ethnicity). The demographic variables, format of the exam, or their interactions were not found to significantly relate to examiner-candidate ratings or pass/fail outcomes. The only variable that was significantly related to CE scores was candidates' QE scores, which was added to the models as a measure of candidates' initial knowledge; this held for all models for total scores (F[1,8659] = 1069.89, p-value < 0.01, R2 = 5% [gender models], F(1,5696.3) = 589.13, p-value < 0.01, R2 = 5% [race models], F(1,4459.5) = 278.33, p-value < 0.01, R2 = 5% [ethnicity models]), and pass/fail outcomes (CI = 1.61-1.73, p-value < 0.01, R2 = 3% [gender models], CI = 1.67-1.85, p-value < 0.01, R2 = 3% [race models], CI = 2.17-2.90, p-value < 0.01, R2 = 3% [ethnicity models]).

CONCLUSIONS

This study shows that there is not a relationship between candidate and examiner gender, race, or ethnicity, and exam outcomes based on statistical models looking at examiner-candidate ratings and pass/fail outcomes. In addition, the delivery of the certifying exam in a virtual format appears to have no statistical impact on outcomes compared to in-person delivery. This suggests that the ABS is performing well in both demographic bias and virtual space.

BLU-DAT: a new reliable and accurate arthrometer for measuring anterior knee laxity

PURPOSE

Although many arthrometers have been developed to assess anterior knee laxity, reliability and diagnostic accuracy of these devices are still debated. The aim of the present study was to evaluate the validity of a new arthrometer in the outpatient setting, with the hypothesis that it had good validity in terms of reliability and diagnostic accuracy.

METHODS

Seventy-eight subjects (39 with ACL injury and 39 with normal ACL) were tested. ATT was assessed by means of the Lachman test at 30° of flexion with a new testing device (BLU-DAT) under three different loading conditions: 7 kg (69 N), 9 kg (88 N) and maximum (MMT). The tests were performed on both knees to obtain SSD. In the ACL injury group, the tests were performed by two examiners and one of them repeated a second test series. Inter- and intra-observer reliability were assessed with the intraclass correlation coefficients (ICCs) for the average SSD measures. In the normal-ACL group, the analysis was performed with the same testing setup. Side-to-side difference measures of the two groups at every loading condition were compared by Student's t test. Data of test series were dichotomized based on the threshold value of 3-mm SSD as pathological ATT and 2 × 2 contingency tables were used to assess diagnostic accuracy.

RESULTS

The ICCs for intra-observer reliability at 7-kg (69 N), 9-kg (88 N) and MMT measurements were 0.781, 0.855 and 0.913, respectively. The ICC for inter-observer reliability at 7-kg (69 N), 9-kg (88 N) and MMT measurements were 0.701, 0.845 and 0.834, respectively. Comparison between the two groups showed a significant mean difference ranging from 3.4 mm for 7-kg (69 N) load to 4.6 mm for MMT. Overall accuracy ranged from 84.6% for 7-kg load to 98.7% for MMT.

CONCLUSION

The BLU-DAT has proven to be an instrument with good intra- and inter-observer reliability and very good accuracy in the diagnosis of ACL injuries in the outpatient setting. So, the BLU-DAT can be a new useful tool in everyday clinical practice to assist in the diagnosis of ACL injury.

LEVEL OF EVIDENCE

II.

Outcome Evaluation of a Transnational Postgraduate Capacity-Building Program Using the Objective Structured Clinical Examination

Previous studies have applied interview-oriented self-reported or peer-centered evaluation methods, rather than an objective and quantitative method, to evaluate outcome of a postgraduate capacity-building program and have mainly focused on the cognitive level. To investigate the feasibility of the objective structured clinical examination (OSCE) in an international setting and report the results of the outcome evaluation for the behavioral aspect. A case-control study. Ninety examinees divided into 2 case-control groups: 17 program-experienced doctors and 17 control doctors in the first group, and 28 mentees of the program-experienced doctors and 28 control doctors in the second group. A six-station OSCE was implemented. The OSCE scores were measured to evaluate (1) the direct educational effect regarding learning in the first group and (2) the indirect educational effects regarding transfer in the second group. Written questionnaire and interview data were collected for qualitative analysis. The quantitative results of the overall or subcomponent OSCE scores indicated no significant differences in the comparisons of the first and second case-control groups. The qualitative data indicated that the program improved participants' medical knowledge, skills, and self-confidence, however, it also revealed limited learning environment provided by the program. This transnational study has demonstrated the process for introducing and successfully completing the testing of an OSCE in Laos. Discrepancy in the goals of the OSCE and the education program limited the usefulness of OSCE as an assessment tool, leading to the lack of significant differences in its results.

Causal attributions of failure among post graduate medical residents in exit fellowship examination in Pakistan: A qualitative study

Background and Objective

Causal attributions are reasons given to certain events in life including failure and success. The objective of this qualitative study was to explore the attributions perceived as the cause of failure by post graduate residents, failing the final clinical exit examination of Fellowship of College of Physicians and Surgeons Pakistan (FCPS).

Methods

This exploratory study was conducted from July 2021 to July 2022, at Jinnah Hospital, Lahore. Study population was selected by purposeful maximal variation sampling. A total of ten post graduate residents from four specialties, failing in FCPS part two clinical examination were included. After written informed consent, semi structured face to face, in depth interviews were conducted. Data saturation was achieved after eight interviews after which two more interviews were carried out. All interviews were recorded and transcribed verbatim. Thematic analysis of the data was done by synthesizing codes into categories and themes.

Results

A total of 17 attributions noted were grouped into five categories namely 'Inadequate exam preparation', 'Personal factors,' 'Exam related factors,' 'Training related factors and 'Luck'. Two main themes of External and Internal factors emerged from these. Common attributions were; lack of effort (8/10), inadequate knowledge (8/10), family commitments (7/10), luck (8/10) and examiner's attitude (5/10).

Conclusions

Most of the residents attributed internal, unstable and controllable factors like inadequate knowledge and lack of effort. In addition, external uncontrollable factors of bad luck and harsh attitude of examiners were considered as contributory factors towards failure.

Are we ready yet for digital transformation? Virtual versus on-campus OSCE as assessment tools in pharmacy education. A randomized controlled head-to-head comparative assessment

Background

The global COVID-19 pandemic has influenced pharmacy education including learning, assessment, and exams. In the UAE, pharmacy instructors have adapted several innovative teaching methods to strive for quality learning outcomes. The current trial presented a head-to-head comparative assessment between on-campus versus virtual Objective Structured Clinical Examination (OSCE) with examiners' and students' perspectives.

Aim

The main aim was to compare fourth-year students' and examiners' perceptions of the feasibility (time and logistics), stress, performance, and satisfaction between on-campus versus virtual OSCE.

Method

A randomized controlled head-to-head comparative assessment between the On-campus and virtual OSCE was conducted to explore performance and satisfaction of pharmacy students and examiners towards the two OSCE settings. The virtual OSCE was carried out directly after the on-campus -OSCE and the setting was designed in a way that aligned with the on-campus OSCE but in a virtual way. Microsoft Teams® breakout room was used as a virtual stations. Respondus-lockdown-browse and Google Meet® were used for proctoring purposes.

Results

Students who sat for the on-campus assessment were more satisfied with the instructions, the orientation session, the time management, and the overall exam setting, the ability of the exam to assess their communication and clinical skills, professionalism and attitude, and the interactivity of the exam compared to the students who sat for the virtual assessment. Examiners' perceptions for both settings were the same with the exception of interaction with students (p less than 0.05) as the on-campus OSCE was more interactive.

Conclusion

Students still prefer the on-campus OSCE to the virtual OSCE format in many aspects. Nevertheless, virtual OSCE is still a feasible and satisfactory method of assessment when on-campus OSCE is not possible. There is a need of a specialized platform to conduct the virtual OSCE from A to Z rather than maximizing the use of options in the current digital platforms.

Quality Management of Objective Structured Clinical Examinations at a Multicampus Nursing College: A Pilot Study

BACKGROUND

Although objective structured clinical examinations (OSCEs) are deemed objective and bias-free, human error, inconsistency, nonuniformity in grading, and inter-rater variability have been reported. Quality management of OSCEs therefore is crucial.

METHOD

Semistructured individual interviews with 14 nurse educators and a qualitative document analysis of 15 external moderators' reports were conducted.

RESULTS

Participants identified measures in place that facilitated quality in management of OSCEs, including a peer review system, control measures ensuring confidentiality, pre-OSCE briefing, orientation, and validation of assessment tools. However, gaps were identified relating to inadequate OSCE assessment tools and documents, as well as a lack and maldistribution of resources, such as physical space, appropriate fidelity manikins, and sufficiently trained examiners.

CONCLUSION

To address gaps, developing robust policies, pilot testing OSCEs and assessment tools, efficiently budgeting for and using required resources, conducting in-depth examiner briefing and training, and setting a gold standard for assessment practices are recommended. [J Nurs Educ. 2023;62(3):155-161.].

Detection of grey zones in inter-rater agreement studies

BACKGROUND

In inter-rater agreement studies, the assessment behaviour of raters can be influenced by their experience, training levels, the degree of willingness to take risks, and the availability of clear guidelines for the assessment. When the assessment behaviour of raters differentiates for some levels of an ordinal classification, a grey zone occurs between the corresponding adjacent cells to these levels around the main diagonal of the table. A grey zone introduces a negative bias to the estimate of the agreement level between the raters. In that sense, it is crucial to detect the existence of a grey zone in an agreement table.

METHODS

In this study, a framework composed of a metric and the corresponding threshold is developed to identify grey zones in an agreement table. The symmetry model and Cohen's kappa are used to define the metric, and the threshold is based on a nonlinear regression model. A numerical study is conducted to assess the accuracy of the developed framework. Real data examples are provided to illustrate the use of the metric and the impact of identifying a grey zone.

RESULTS

The sensitivity and specificity of the proposed framework are shown to be very high under moderate, substantial, and near-perfect agreement levels for [Formula: see text] and [Formula: see text] tables and sample sizes greater than or equal to 100 and 50, respectively. Real data examples demonstrate that when a grey zone is detected in the table, it is possible to report a notably higher level of agreement in the studies.

CONCLUSIONS

The accuracy of the proposed framework is sufficiently high; hence, it provides practitioners with a precise way to detect the grey zones in agreement tables.

The influence of candidates' physical attributes on patient ratings in simulated assessments of clinical practice

BACKGROUND

We have previously shown that clinical examiners' scoring is not negatively impacted when a candidate has a tattoo, unnatural hair colour, or a regional accent. We investigated whether these physical attributes in exam candidates impact patient scoring.

METHODS

Simulated/real patients were randomly assigned to watch five videos of simulated candidate performances of a cranial nerve examination: clear fail, borderline, good, 'clear pass' without an attribute, and 'clear pass' with one of the attributes (tattoo, purple hair, accent). Participants scored domains of communication and professionalism. We compared scores for the clear pass candidates with and without attributes.

RESULTS

One hundred and eighty three patients participated. The total scores for the candidates with tattoos and purple hair were higher than the candidate with no physical attribute (p < 0.001). For the candidate with a Liverpool English accent no difference was identified (p = 0.120).

CONCLUSIONS

The presence of certain physical attributes (tattoos or purple hair) was associated with higher scores given by patients to candidates in a simulated physical examination station.

Objective structured clinical examination: Challenges and opportunities from students’ perspective

Background

Objective structured clinical examination (OSCE) has been used in evaluating clinical competence in health professions education around the world. Despite its implementation in Iraq for around a decade, limited studies investigated the challenges and opportunities to improve the standard and quality of this examination from student’s perspective.

Methods

This qualitative study was based on an online open-ended questionnaire survey that was carried out in the College of Medicine, Hawler Medical University, Iraq at the beginning of the 2018–2019 academic year. A convenience sample of 180 students in the clinical phase (4th, 5th, and 6th) year of study were invited to participate.

Results

A total of 141 students responded to the online questionnaire. The participants were generally happy with the OSCE, and they recognized many positive aspects, including the role of the OSCE in increasing confidence, engagement and motivating learning, the role of the OSCE in achieving a higher level of learning, the content validity of the OSCE, and the quality of the OSCE. The main weak points of the OSCE identified by the students included unfairness, gender discrimination, duration of the OSCE, and the behavior of the examiners. Suggestions to improve the OSCE examination included improving the examiners’ behavior, with the focus on the training of the examiners, and avoiding discrimination among students.

Conclusions

Most of the students were generally satisfied with the current OSCE examination. The main concern of the students was related to the organization of the OSCE. Valuable suggestions were raised to improve the OSCE quality including examiners’ and simulated patients’ training.

Pilot study of the DART tool - an objective healthcare simulation debriefing assessment instrument

BACKGROUND

Various rating tools aim to assess simulation debriefing quality, but their use may be limited by complexity and subjectivity. The Debriefing Assessment in Real Time (DART) tool represents an alternative debriefing aid that uses quantitative measures to estimate quality and requires minimal training to use. The DART is uses a cumulative tally of instructor questions (IQ), instructor statements (IS) and trainee responses (TR). Ratios for IQ:IS and TR:[IQ + IS] may estimate the level of debriefer inclusivity and participant engagement.

METHODS

Experienced faculty from four geographically disparate university-affiliated simulation centers rated video-based debriefings and a transcript using the DART. The primary endpoint was an assessment of the estimated reliability of the tool. The small sample size confined analysis to descriptive statistics and coefficient of variations (CV%) as an estimate of reliability.

RESULTS

Ratings for Video A (n = 7), Video B (n = 6), and Transcript A (n = 6) demonstrated mean CV% for IQ (27.8%), IS (39.5%), TR (34.8%), IQ:IS (40.8%), and TR:[IQ + IS] (28.0%). Higher CV% observed in IS and TR may be attributable to rater characterizations of longer contributions as either lumped or split. Lower variances in IQ and TR:[IQ + IS] suggest overall consistency regardless of scores being lumped or split.

CONCLUSION

The DART tool appears to be reliable for the recording of data which may be useful for informing feedback to debriefers. Future studies should assess reliability in a wider pool of debriefings and examine potential uses in faculty development.

Excellent Agreement of In-Person Scoring versus Scoring of Digitally Recorded Simulated Suture Skills Examination

This study's objective was to evaluate the agreement between in-person performance scores and digitally recorded assessment scores by the same examiner using a simulated suturing skill examination. With ethics approval, veterinary students underwent a simulated skills examination proctored by an in-person examiner and simultaneously digitally recorded using two wide-angle cameras mounted overtop and to the side of the surgical field. Performance scores were based on a nine-item rubric. In-person examination scores were compared for agreement with those obtained by blind review of the digital recording of the same session, by the same examiner, 6-18 months following the in-person examination. Thirty-nine students were enrolled. All rubric categories could be assessed during digital assessment of the recording from the camera mounted above the surgical area. In two instances, the side digital recording had to be reviewed to confirm correct needle holder grip. Concordance correlation between performance scores from in-person and post hoc digital assessment was excellent (r = .93). The excellent agreement between in-person and digital assessment suggests that digitally recording skills examinations can provide adequate suturing skills assessment, presenting several benefits. Digitally recorded assessment allows for anonymity, which can reduce assessor bias/favoritism, provide a record of performance that students can review and critically self-reflect upon, and reduce the number of in-person examiners required to complete surgical skills examinations. Additionally, digitally recorded assessment could become an option for students who experience anxiety performing a skills exam in the presence of an examiner.

“Could You Work in My Team?”: Exploring How Professional Clinical Role Expectations Influence Decision-Making of Assessors During Exit-Level Medical School OSCEs

Decision-making in clinical assessment, such as exit-level medical school Objective Structured Clinical Examinations (OSCEs), is complex. This study utilized an empirical phenomenological qualitative approach with thematic analysis to explore OSCE assessors' perceptions of the concept of a “prototypical intern” expressed during focus group discussions. Topics discussed included the concept of a prototypical intern, qualities to be assessed, and approaches to clinical assessment decision-making. The thematic analysis was then applied to a theoretical framework (Cultural Historical Activity Theory—CHAT) that explored the complexity of making assessment decisions amidst potentially contradicting pressures from academic and clinical perspectives. Ten Australasian medical schools were involved with 15 experienced and five less experienced assessors participating. Thematic analysis of the data revealed four major themes in relation to how the prototypical intern concept influences clinical assessors' judgements: (a) Suitability of marking rubric based on assessor characteristics and expectations; (b) Competence as final year student vs. performance as a prototypical intern; (c) Safety, trustworthiness and reliability as constructs requiring assessment and (d) Contradictions in decision making process due to assessor differences. These themes mapped well within the interaction between two proposed activity systems in the CHAT model: academic and clinical. More clinically engaged and more experienced assessors tend to fall back on a heuristic, mental construct of a “prototypical intern,” to calibrate judgements, particularly, in difficult situations. Further research is needed to explore whether consensus on desirable intern qualities and their inclusion into OSCE marksheets decreases the cognitive load and increases the validity of assessor decision making.

Minimally Trained Analysts Can Perform Fast, Objective Assessment of Orthopedic Technical Skill from Fluoroscopic Images

Skill assessment in orthopedics has traditionally relied on subjective impressions from a supervising surgeon. The feedback derived from these tools may be limited by bias and other practical issues. Objective analysis of intraoperative fluoroscopic images offers an inexpensive, repeatable, and precise assessment strategy without bias. Assessors generally refrain from using the scores of images obtained throughout the operation to evaluate skill for practical reasons. A new system was designed to facilitate rapid analysis of this fluoroscopy via minimally trained analysts. Four expert and four novice analysts independently measured one objective metric for skill using both a custom analysis software and a commercial alternative. Analysts were able to measure the objective metric three times faster when using the custom software, and without a practical difference in accuracy in comparison to the expert analysts using the commercial software. These results suggest that a well-designed fluoroscopy analysis system can facilitate inexpensive, reliable, and objective assessment of surgical skills.

Tutor-Student Partnership in Practice OSCE to Enhance Medical Education

BACKGROUND

Training of examiners is essential to ensure the quality of objective structured clinical examination (OSCE). We aimed to study a perceived effectiveness of tutor-student partnership in a practice OSCE module by novice OSCE tutors and medical students.

METHOD

We implemented a practice OSCE at a medical faculty in France with novice tutors and third year medical students as partners. Each tutor (n = 44) served as a partner for the group of 5 students in the conception of the scenario and as an evaluator of the tutored station. Students (n = 303) were involved in the conception of a case and the roles of a physician, evaluator and a simulated patient. Data were obtained through self-assessment questionnaires. Descriptive statistics were used to analyze items of the questionnaires. Free-form answers were coded and analyzed thematically.

RESULTS

A total of 36 tutors (82%) and 185 students (61%) responded to the questionnaires. The intervention was well perceived. Thirty-two percent of the tutors reported some difficulties in the assessment of student performance and were disposed to receive further training. Fifty-five percent of the students considered the participation in the OSCE case development appropriate to their level of knowledge, and 70% perceived it as beneficial allowing them to set their learning goals.

CONCLUSION

This initiative provides a relevant method beneficial to OSCE tutors, medical students, and the faculty. Tutors learn how to assess student performance according to expected achievement levels. It allows students to be engaged as partners in co-creation of learning and teaching.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s40670-021-01421-9.

Influence of a Survival Swimming Training Programme on Water Safety Knowledge, Attitudes and Skills: A Randomized Controlled Trial among Young Adults in Sri Lanka

Drowning among young adults is high in Sri Lanka. Water safety education is a recommended strategy for drowning prevention but is often overlooked for young adults. This study aimed to evaluate the effectiveness of an adapted educational intervention, "Swim for Safety" on improving water safety knowledge, attitudes and survival swimming skills among undergraduates (19-28 years) in Sri Lanka. This study employed a parallel-group, two-arm randomized controlled trial design. The intervention group (n = 78) received a face-to-face, 12-lesson education programme, and the control group (n = 78) received a brochure and weekly mobile phone messages for six consecutive weeks. Baseline, post-intervention and three-month follow-up knowledge, attitudes and skills were evaluated. Knowledge and attitudes were assessed using a self-administered questionnaire and skills were evaluated following a skills assessment protocol. In total 116 participants, 60 intervention group and 56 control group, completed the study. At baseline there were no differences between groups in median scores of water safety knowledge, attitudes and survival swimming skills. The intervention group demonstrated statistically significant increases in median water safety knowledge, attitudes and survival swimming skill scores compared with the control group, following the intervention and maintained at three-month follow-up (p < 0.05). The adapted Swim for Safety programme significantly improved water safety knowledge, attitudes, and survival swimming skills among young adults in Sri Lanka. Therefore, it is recommended that the SfS programme be implemented widely to prevent drowning in young adults.

Designing and running an online Objective Structured Clinical Examination (OSCE) on Zoom: A peer-led example

INTRODUCTION

The COVID-19 pandemic threatened medical teaching and progression secondary to the 'lockdown' measures which limit physical interactions. Consequently, there was a necessity to shift medical teaching and assessment online. Our challenge was to logistically organise and conduct an Objective Structured Clinical Examination (OSCE) using a virtual, socially distanced approach.

METHODS

We found that Zoom offered the most suitable functionality in mimicking a physical OSCE online and its breakout rooms were used to represent different stations. One organiser manually transferred candidates between stations, allowing for the OSCE to operate punctually. Examiners screen shared to display an automated slideshow with station vignettes, investigation results, and viva questions.

RESULTS

Participants found the use of Zoom's features to be extremely beneficial with one candidate describing the experience as 'brilliant, very efficient and smooth'. Numerous aspects of the OSCE from decision making, to verbal communication, to medical knowledge were assessed, equivalent to a physical OSCE.

DISCUSSION

As the transformation of medical education is catalysed by the pandemic, it is important to ensure that peer to peer teaching and assessments are revolutionised with a quality similar to that of physical teaching. We have displayed how this may be achieved in an OSCE setting using Zoom teleconferencing.

The impact of grey zones on the accuracy of agreement measures for ordinal tables

Background

In an inter-rater agreement study, if two raters tend to rate considering different aspects of the subject of interest or have different experience levels, a grey zone occurs among the levels of a square contingency table showing the inter-rater agreement. These grey zones distort the degree of agreement between raters and negatively impact the decisions based on the inter-rater agreement tables. In this sense, it is important to know how the existence of a grey zone impacts the inter-rater agreement coefficients to choose the most reliable agreement coefficient against the grey zones to reach out with more reliable decisions.

Methods

In this article, we propose two approaches to create grey zones in simulations setting and conduct an extensive Monte Carlo simulation study to figure out the impact of having grey zones on the weighted inter-rater agreement measures for ordinal tables over a comprehensive simulation space.

Results

The weighted inter-rater agreement coefficients are not reliable against the existence of grey zones. Increasing sample size and the number of categories in the agreement table decreases the accuracy of weighted inter-rater agreement measures when there is a grey zone. When the degree of agreement between the raters is high, the agreement measures are not significantly impacted by the existence of grey zones. However, if there is a medium to low degree of inter-rater agreement, all the weighted coefficients are more or less impacted.

Conclusions

It is observed in this study that the existence of grey zones has a significant negative impact on the accuracy of agreement measures especially for a low degree of true agreement and high sample and tables sizes. In general, Gwet’s AC2 and Brennan-Prediger’s κ with quadratic or ordinal weights are reliable against the grey zones.

Supplementary Information

The online version contains supplementary material available at (10.1186/s12874-021-01248-3).

Examiners' decision-making processes in observation-based clinical examinations

BACKGROUND

Objective structured clinical examinations (OSCEs) are commonly used to assess the clinical skills of health professional students. Examiner judgement is one acknowledged source of variation in candidate marks. This paper reports an exploration of examiner decision making to better characterise the cognitive processes and workload associated with making judgements of clinical performance in exit-level OSCEs.

METHODS

Fifty-five examiners for exit-level OSCEs at five Australian medical schools completed a NASA Task Load Index (TLX) measure of cognitive load and participated in focus group interviews immediately after the OSCE session. Discussions focused on how decisions were made for borderline and clear pass candidates. Interviews were transcribed, coded and thematically analysed. NASA TLX results were quantitatively analysed.

RESULTS

Examiners self-reported higher cognitive workload levels when assessing a borderline candidate in comparison with a clear pass candidate. Further analysis revealed five major themes considered by examiners when marking candidate performance in an OSCE: (a) use of marking criteria as a source of reassurance; (b) difficulty adhering to the marking sheet under certain conditions; (c) demeanour of candidates; (d) patient safety, and (e) calibration using a mental construct of the 'mythical [prototypical] intern'. Examiners demonstrated particularly higher mental demand when assessing borderline compared to clear pass candidates.

CONCLUSIONS

Examiners demonstrate that judging candidate performance is a complex, cognitively difficult task, particularly when performance is of borderline or lower standard. At programme exit level, examiners intuitively want to rate candidates against a construct of a prototypical graduate when marking criteria appear not to describe both what and how a passing candidate should demonstrate when completing clinical tasks. This construct should be shared, agreed upon and aligned with marking criteria to best guide examiner training and calibration. Achieving this integration may improve the accuracy and consistency of examiner judgements and reduce cognitive workload.

Standardized examinees: development of a new tool to evaluate factors influencing OSCE scores and to train examiners

Introduction: The Objective Structured Clinical Examination (OSCE) is an established format for practical clinical assessments at most medical schools and discussion is underway in Germany to make it part of future state medical exams. Examiner behavior that influences assessment results is described. Erroneous assessments of student performance can result, for instance, from systematic leniency, inconsistent grading, halo effects, and even a lack of differentiation between the tasks to be performed over the entire grading scale. The aim of this study was to develop a quality assurance tool that can monitor factors influencing grading in a real OSCE and enable targeted training of examiners. Material, Methods and Students: Twelve students at the Medical Faculty of the University of Heidelberg were each trained to perform a defined task for a particular surgical OSCE station. Definitions were set and operationalized for an excellent and a borderline performance. In a simulated OSCE during the first part of the study, the standardized student performances were assessed and graded by different examiners three times in succession; video recordings were made. Quantitative and qualitative analysis of the videos was also undertaken by the study coordinator. In the second part of the study, the videos were used to investigate the examiners' acceptance of standardized examinees and to analyze potential influences on scoring that stemmed from the examiners' experience. Results: In the first part of the study, the OSCE scores and subsequent video analysis showed that standardization for defined performance levels at different OSCE stations is generally possible. Individual deviations from the prescribed examinee responses were observed and occurred primarily with increased complexity of OSCE station content. In the second part of the study, inexperienced examiners assessed a borderline performance significantly lower than their experienced colleagues (13.50 vs. 15.15, p=0.035). No difference was seen in the evaluation of the excellent examinees. Both groups of examiners graded the item "ocial competence" - despite identical standardization - significantly lower for examinees with borderline performances than for excellent examinees (4.13 vs. 4.80, p<0.001). Conclusion: Standardization of examinees for previously defined performance levels is possible, making a new tool available in future not only for OSCE quality assurance, but also for training examiners. Detailed preparation of the OSCE checklists and intensive training of the examinees are essential. This new tool takes on a special importance if standardized OSCEs are integrated into state medical exams and, as such, become high-stakes assessments.

"Evaluation of a best practice approach to assess undergraduate clinical skills in Paediatrics"

BACKGROUND

The Objective Structured Clinical Examination (OSCE) has been used in pediatrics since the 1980s. Its main drawback is that large numbers of children are needed to make up for the fatigue factor inherent in prolonged testing periods. Also, examinations mainly include children between 7 and 16 years old. We describe the summative examination used in our institution to evaluate medical students' clinical competencies in pediatrics with realistic available resources and for a wider age-range. We also evaluated different factors known to influence medical students' performances.

METHODS

This retrospective, descriptive, observational study evaluated the 740 distinct pediatric examination results of fourth-year medical students over 5 years. Their summative examination combined two different assessment methods: a structured real-patient examination (SRPE) using standardized assessment grids for the most frequent pediatric diagnoses, and a computer-based written examination (CBWE).

RESULTS

Our approach defined an appropriate setting for some key elements of the educational objectives of pediatrics training, such as balancing the child-parent-pediatrician triangle and the ability to interact with pediatric patients, from newborns to 16-year-old adolescents, in a child-friendly fashion in realistic scenarios. SRPE scores showed no associations with students' degrees of exposure to specific lecture topics, vignettes, or bedside teaching. The impacts of clinical setting, topic, and individual examiners on SRPE scores was quite limited. Setting explained 1.6%, topic explained 4.5%, and examiner explained 4.7% of the overall variability in SRPE scores.

CONCLUSIONS

By combining two different assessment methods, we were able to provide a best-practice approach for assessing clinical skills in Pediatrics over a wide range of real patients.

Rater training for standardised assessment of Objective Structured Clinical Examinations in rural Tanzania

OBJECTIVES

To describe a simulation-based rater training curriculum for Objective Structured Clinical Examinations (OSCEs) for clinician-based training for frontline staff caring for mothers and babies in rural Tanzania.

BACKGROUND

Rater training for OSCE evaluation is widely embraced in high-income countries but not well described in low-income and middle-income countries. Helping Babies Breathe, Essential Care for Every Baby and Bleeding after Birth are standardised training programmes that encourage OSCE evaluations. Studies examining the reliability of assessments are rare.

METHODS

Training of raters occurred over 3 days. Raters scored selected OSCEs role-played using standardised learners and low-fidelity mannikins, assigning proficiency levels a priori. Researchers used Zabar's criteria to critique rater agreement and mitigate measurement error during score review. Descriptive statistics, Fleiss' kappa and field notes were used to describe results.

RESULTS

Six healthcare providers scored 42 training scenarios. There was moderate rater agreement across all OSCEs (κ=0.508). Kappa values increased with Helping Babies Breathe (κ=0.28-0.48) and Essential Care for Every Baby (κ=0.42-0.77) by day 3 of training, but not with Bleeding after Birth (κ=0.58-0.33). Raters identified average proficiency 50% of the time.

CONCLUSION

Our study shows that the in-country raters in this study had a hard time identifying average performance despite moderate rater agreement. Rater training is critical to ensure that the potential of training programmes translates to improved outcomes for mothers and babies; more research into the concepts and training for discernment of competence in this setting is necessary.

DRIFT happens, sometimes: Examining time based rater variance in a high-stakes OSCE

Introduction: Examiner based variance can affect test taker outcomes. The aim of this study was to investigate the examiner-based effect of DRIFT or differential rater functioning over time. Methods: Average station level scores from five administrations of the same version of a highstakes 12-station OSCE were analyzed for the presence of DRIFT. Results: Test-takers who were scored earlier appeared to receive a score advantage, while those who were scored later, appeared to receive neither a score advantage nor disadvantage due to the DRIFT behavior. A specific form of DRIFT, primacy (the assignment of progressively harsher scores), was present in one out of the 228 examiner scoring opportunities investigated in this study. In other words, less than 1% of the examiner scoring that took place displayed significant levels of DRIFT scoring behavior. Discussion and Conclusions: The noted score advantage influenced the test outcomes of only one examinee who performed close to the cut-score on all other stations. Prior publications report broader effects of DRIFT, but the current assessment context, particularly access to examiner training, may have had a modulating effect in the present study.

Effects and Consequences of Being an OSCE Examiner in Surgery-A Qualitative Study

OBJECTIVE

Even though objective structured clinical examination (OSCE) is a well-investigated format for competency-based practical examination, only a few studies have explored the motivations of OSCE examiners and their opinions, both positive and negative, toward being an examiner. The aim of this study was to gain insights into the views of OSCE examiners using semi-structured interviews.

DESIGN

Surgical OSCE examiners were queried at two medical faculties in Germany via semi-structured interviews. The interviews were transcribed verbatim and analyzed using the techniques of structured qualitative content analysis.

SETTING

This study was conducted at the medical faculties of the Goethe University, Frankfurt, Germany and of the Otto-von-Guericke University, Magdeburg, Germany.

PARTICIPANTS

All of the study participants were surgeons working at the university hospital of one of the faculties.

RESULTS

A total of 29 examiners were queried until a saturation of content was achieved. A critical reflection of one's own teaching was described as a major benefit by most participants. Furthermore, they noted that the standards and competences examined during the OSCE boosted the detail of their teaching sessions in the wards. However, the examiners criticized missed operations due the examination and were not appreciated by superiors for being an examiner. Most of the examiners (22/29) preferred to be an examiner themselves rather than appointing student peer examiners. If they had appointed someone else, that would mean they would miss valuable experiences useful for their own teaching.

CONCLUSIONS

Being an OSCE examiner confers several advantages, notably the reflection of one's own teaching, which was described as highly valuable by the examiners.

Fairness and objectivity of a multiple scenario objective structured clinical examination

Introduction: The aim of the Objective Structured Clinical Examination (OSCE) is a standardized and fair assessment of clinical skills. Observing second clinical year medical students during a summative OSCE assessing a General Practice clerkship, we noticed that information exchange with peers led to a progressively faster and overly focused management of simulations. Therefore, we established a Multiple Scenario-OSCE (MS-OSCE) where all students had to manage the same chief complaint at a station but it's underlying scenarios being randomly changed during students' rotation through their parcours. We wanted to ensure they fully explore differential diagnosis instead of managing their task influenced by shared information. We wanted to assess if a MS-OSCE violates the assumption of objectivity and fairness given that students are not tested with the same scenarios. Methods: We developed and piloted five OSCE stations (chest pain, abdominal pain, back pain, fatigue and acute cough) with two or three different underlying scenarios each. At each station these scenarios randomly changed from student to student. Performance was assessed with a checklist and global rating. The effect of scenarios and raters on students' grades was assessed calculating the intraclass correlation coefficient with a fixed effect two level linear model. Results: A total of 169 students and 23 raters participated in the MS-OSCE. The internal consistency over all stations was 0.65 by Cronbach's alpha. The difference of the mean grades between the scenarios of a given chief complaint ranged from 0.03 to 0.4 on a 1 to 5 grading scale. The effect of scenarios on the variance of the final grades at each station ranged from 4% to 9% and of raters from 20% to 50% when adjusted for students' skills. Conclusions: The effect of different scenarios on the grades was relevant but small compared to the effect of raters on grades. Improving rater training is more important to ensure objectivity and fairness of MS-OSCE than providing the same scenario to all students.

Emotion recognition and extraversion of medical students interact to predict their empathic communication perceived by simulated patients

BACKGROUND

This study assessed the impact of medical students' emotion recognition ability and extraversion on their empathic communication, as perceived by simulated patients in a training context.

METHODS

This study used a crossed-effect data structure and examined 245 students in their fourth year of medical school. The students' personality traits were assessed based on a self-assessment questionnaire of the short form of the Big Five Inventory; their emotion recognition ability was measured using a performance test (Diagnostic Analysis of Nonverbal Accuracy-2, Adult Facial Expressions). Simulated patients evaluated the medical students' empathic communication.

RESULTS

Students with a combination of high emotion recognition ability and extraversion received more positive ratings from simulated patients than their fellow students with a combination of emotion recognition ability and low extraversion. The main effects of emotion recognition or extraversion were not sufficient to yield similar effects. There were no other effects related to the remaining Big Five variables.

CONCLUSIONS

The results support the hypothesis that to build rapport with patients, medical staff need to combine emotional capabilities with a dispositional interest in interpersonal encounters.

[Examinations while studying medicine - more than simply grades]

Assessment drives learning. Examinations need to be aligned primarily with learning objectives, as well as teaching and assessment methods of the courses on offer. In doing so, various examination instruments are required to measure on levels of competency that build on one another. An appropriate mix is essential to reflect the variety of learning outcomes of a chosen curriculum. Furthermore, examinations also possess the characteristics of evaluation: They reflect the knowledge and abilities of students and assess the teaching at a defined location. Digital examinations in the form of multiple-choice-question (MCQ) testing enable a higher degree of automation and accelerate the processes of creation, implementation, and evaluation of the examination results. Thus, they enjoy increasing popularity, provided that the technical requirements for large semester cohorts are met. Shifting examination processes to computers or tablets entails not only a wealth of new challenges but also opportunities.

Examiner seniority and experience are associated with bias when scoring communication, but not examination, skills in objective structured clinical examinations in Australia

PURPOSE

The biases that may influence objective structured clinical examination (OSCE) scoring are well understood, and recent research has attempted to establish the magnitude of their impact. However, the influence of examiner experience, clinical seniority, and occupation on communication and physical examination scores in OSCEs has not yet been clearly established.

METHODS

We compared the mean scores awarded for generic and clinical communication and physical examination skills in 2 undergraduate medicine OSCEs in relation to examiner characteristics (gender, examining experience, occupation, seniority, and speciality). The statistical significance of the differences was calculated using the 2-tailed independent t-test and analysis of variance.

RESULTS

Five hundred and seventeen students were examined by 237 examiners at the University of New South Wales in 2014 and 2016. Examiner gender, occupation (academic, clinician, or clinical tutor), and job type (specialist or generalist) did not significantly impact scores. Junior doctors gave consistently higher scores than senior doctors in all domains, and this difference was statistically significant for generic and clinical communication scores. Examiner experience was significantly inversely correlated with generic communication scores.

CONCLUSION

We suggest that the assessment of examination skills may be less susceptible to bias because this process is fairly prescriptive, affording greater scoring objectivity. We recommend training to define the marking criteria, teaching curriculum, and expected level of performance in communication skills to reduce bias in OSCE assessment.

The sights and insights of examiners in objective structured clinical examinations

PURPOSE

The objective structured clinical examination (OSCE) is considered to be one of the most robust methods of clinical assessment. One of its strengths lies in its ability to minimise the effects of examiner bias due to the standardisation of items and tasks for each candidate. However, OSCE examiners' assessment scores are influenced by several factors that may jeopardise the assumed objectivity of OSCEs. To better understand this phenomenon, the current review aims to determine and describe important sources of examiner bias and the factors affecting examiners' assessments.

METHODS

We performed a narrative review of the medical literature using Medline. All articles meeting the selection criteria were reviewed, with salient points extracted and synthesised into a clear and comprehensive summary of the knowledge in this area.

RESULTS

OSCE examiners' assessment scores are influenced by factors belonging to 4 different domains: examination context, examinee characteristics, examinee-examiner interactions, and examiner characteristics. These domains are composed of several factors including halo, hawk/dove and OSCE contrast effects; the examiner's gender and ethnicity; training; lifetime experience in assessing; leadership and familiarity with students; station type; and site effects.

CONCLUSION

Several factors may influence the presumed objectivity of examiners' assessments, and these factors need to be addressed to ensure the objectivity of OSCEs. We offer insights into directions for future research to better understand and address the phenomenon of examiner bias.