Does quantity ensure quality? Standardized OSCE-stations for outcome-oriented evaluation of practical skills at different medical faculties

[Success factors and obstacles in the implementation of competence-oriented teaching in surgery]

BACKGROUND AND OBJECTIVES

For successful competence-oriented teaching at the medical faculties it is important to identify the factors that influence its implementation in order to benefit from the strengths and balance out weaknesses. The present study examined the success factors and obstacles of the implementation of competence-oriented teaching in the surgical discipline from the point of view of students and lecturers.

METHODS

After implementation of competence-oriented teaching based on the teaching goals of the NKLM, in clinical examination courses (bedside teaching and block internship, BP) at two hospitals, a qualitative content analysis and quantification of the answers were performed using focus group interviews and questionnaires with students (S) and lecturers (D).

RESULTS

During the summer semester 2022 a total of 31 students and 14 lecturers were interviewed in focus groups and 143 questionnaires (123 S, 20 D) were analyzed. For the students the presence of concrete competences/teaching goals, guidelines for the lesson, transparent goals and ability to demand teaching goals as well as structured lessons and mentoring were the main success factors. Lecturers on the other hand reported the presence of concrete goals, assistance for the lesson preparation and the activity of the students as success factors. The results of the questionnaires showed that the majority (88% S, 75% D) were informed about the teaching goals and considered them to be followed (84%S, 95% D). Obstacles were the factors "time", "mentoring" and "information". Factors that were between negative and positive (indifferent factors) were "uncertainty about competence-orientation" and "uncertainty how to examine the teaching success".

DISCUSSION

Transparent structure and teaching goals as well as a mentoring system are the success factors for the implementation of competence-oriented lessons and should be used as strengths. Indifferent factors represent weaknesses and need to be addressed by training and instruction. Restricted time and personnel resources are the immanent problems that hamper the implementation and require fulminant structural changes.

Comparability of Objective Structured Clinical Examinations (OSCEs) and Written Tests for Assessing Medical School Students' Competencies: A Scoping Review

Objective Structured Clinical Examinations (OSCEs) and written tests are commonly used to assess health professional students, but it remains unclear whether the additional human resources and expenses required for OSCEs, both in-person and online, are worthwhile for assessing competencies. This scoping review summarized literature identified by searching MEDLINE and EMBASE comparing 1) OSCEs and written tests and 2) in-person and online OSCEs, for assessing health professional trainees' competencies. For Q1, 21 studies satisfied inclusion criteria. The most examined health profession was medical trainees (19, 90.5%), the comparison was most frequently OSCEs versus multiple-choice questions (MCQs) (18, 85.7%), and 18 (87.5%) examined the same competency domain. Most (77.5%) total score correlation coefficients between testing methods were weak (r < 0.40). For Q2, 13 articles were included. In-person and online OSCEs were most used for medical trainees (9, 69.2%), checklists were the most prevalent evaluation scheme (7, 63.6%), and 14/17 overall score comparisons were not statistically significantly different. Generally low correlations exist between MCQ and OSCE scores, providing insufficient evidence as to whether OSCEs provide sufficient value to be worth their additional cost. Online OSCEs may be a viable alternative to in-person OSCEs for certain competencies where technical challenges can be met.

[Testing of practical surgical teaching at a distance-Experiences with a hybrid OSCE in surgery]

BACKGROUND

The COVID-19 pandemic has changed medical teaching worldwide. Digital teaching and examinations are successful for cognitive learning objectives, whereas practical skills had to be carried out predominantly in person under strict hygiene standards.

AIM

This study presents the opportunities and challenges of using a presence objective structured clinical examination (OSCE) at a distance with digital support.

METHOD

Following surgical practical teaching an OSCE was conducted in presence, where students demonstrated practical skills in one room while the examiners were connected via videoconference from another room. Students were surveyed about the OSCE and sustained learning via a standardized online questionnaire after completion of the surgical teaching. Additionally, examiners were surveyed on their experiences.

RESULTS

In the online survey 40 students participated (25% of n = 157 students contacted) and 36 completed questionnaires were analyzed. Overall, the implementation of an OSCE even under pandemic conditions was perceived as very positive by the students (92% of students n = 33). In particular, the acquisition of practical skills was rated as very high. For 78% (n = 28) of the students, the acquisition of competencies through the practical examination was particularly sustainable. The vast majority of students and examiners felt safe regarding infection control because of the hygiene concept (92%, n = 33). Overall, 80 students achieved grade 1 (51%), 71 students grade 2 (45.2%) and 6 students grade 3 (3.8%) (grade 1 = very good, grade 6 = very bad).

CONCLUSION

Practical examinations are essential for checking practical learning objectives and can be implemented at a distance with a well-developed hygiene concept and digital support.

Tutor-Student Partnership in Practice OSCE to Enhance Medical Education

BACKGROUND

Training of examiners is essential to ensure the quality of objective structured clinical examination (OSCE). We aimed to study a perceived effectiveness of tutor-student partnership in a practice OSCE module by novice OSCE tutors and medical students.

METHOD

We implemented a practice OSCE at a medical faculty in France with novice tutors and third year medical students as partners. Each tutor (n = 44) served as a partner for the group of 5 students in the conception of the scenario and as an evaluator of the tutored station. Students (n = 303) were involved in the conception of a case and the roles of a physician, evaluator and a simulated patient. Data were obtained through self-assessment questionnaires. Descriptive statistics were used to analyze items of the questionnaires. Free-form answers were coded and analyzed thematically.

RESULTS

A total of 36 tutors (82%) and 185 students (61%) responded to the questionnaires. The intervention was well perceived. Thirty-two percent of the tutors reported some difficulties in the assessment of student performance and were disposed to receive further training. Fifty-five percent of the students considered the participation in the OSCE case development appropriate to their level of knowledge, and 70% perceived it as beneficial allowing them to set their learning goals.

CONCLUSION

This initiative provides a relevant method beneficial to OSCE tutors, medical students, and the faculty. Tutors learn how to assess student performance according to expected achievement levels. It allows students to be engaged as partners in co-creation of learning and teaching.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s40670-021-01421-9.

Video Recording in Veterinary Medicine OSCEs: Feasibility and Inter-rater Agreement between Live Performance Examiners and Video Recording Reviewing Examiners

The Objective Structured Clinical Examination (OSCE) is a valid, reliable assessment of veterinary students' clinical skills that requires significant examiner training and scoring time. This article seeks to investigate the utility of implementing video recording by scoring OSCEs in real-time using live examiners, and afterwards using video examiners from within and outside the learners' home institution. Using checklists, learners (n=33) were assessed by one live examiner and five video examiners on three OSCE stations: suturing, arthrocentesis, and thoracocentesis. When stations were considered collectively, there was no difference between pass/fail outcome between live and video examiners (χ² = 0.37, p = .55). However, when considered individually, stations (χ² = 16.64, p < .001) and interaction between station and type of examiner (χ² = 7.13, p = .03) demonstrated a significant effect on pass/fail outcome. Specifically, learners being assessed on suturing with a video examiner had increased odds of passing the station as compared with their arthrocentesis or thoracocentesis stations. Internal consistency was fair to moderate (0.34-0.45). Inter-rater reliability measures varied but were mostly moderate to strong (0.56-0.82). Video examiners spent longer assessing learners than live raters (mean of 21 min/learner vs. 13 min/learner). Station-specific differences among video examiners may be due to intermittent visibility issues during video capture. Overall, video recording learner performances appears reliable and feasible, although there were time, cost, and technical issues that may limit its routine use.

Improving competence and safety in pain medicine: a practical clinical teaching strategy for students combining simulation and bedside teaching

BACKGROUND

Pain is a devastating sensation and has to be treated immediately. Therefore, we developed a training program to improve the knowledge of medical students in the field of pain medicine. In the present study, the applicability and efficacy of this training program was tested.

METHODS

Half of the students attended first a training with simulated patients (SP) followed by bedside teaching (Group 1). Group 2 performed the training programs in reverse order. The evaluation based on standardized questionnaires completed by students (self-assessment) and all students took part in two practical examinations after the learning interventions.

RESULTS

This study included 35 students. The quality of the simulation was evaluated by the students with average grade 1.1 (1 = very good, 6 = very bad). The practical work on the ward with patients was rated with grade 1.4 of 6, the whole course with 1.1. Students of Group A were significantly better in the final examination (grade 1.7 vs. grade 2.2, p < 0.05). To rate the improvement of skills (self-assessment) we used a Likert Scale (1 = very certain, 5 = very uncertain). The following skills were similar in both groups and significantly better after the course: taking responsibility, expert knowledge, empathy, relationship building and communication.

CONCLUSIONS

Training with simulated patients in combination with small-group teaching at the bedside with real patients achieves a dramatic increase in student competence. Students prefer learning from the simulation before bedside teaching and propose to include simulation into the curricular teaching of pain medicine.

Standardized examinees: development of a new tool to evaluate factors influencing OSCE scores and to train examiners

Introduction: The Objective Structured Clinical Examination (OSCE) is an established format for practical clinical assessments at most medical schools and discussion is underway in Germany to make it part of future state medical exams. Examiner behavior that influences assessment results is described. Erroneous assessments of student performance can result, for instance, from systematic leniency, inconsistent grading, halo effects, and even a lack of differentiation between the tasks to be performed over the entire grading scale. The aim of this study was to develop a quality assurance tool that can monitor factors influencing grading in a real OSCE and enable targeted training of examiners. Material, Methods and Students: Twelve students at the Medical Faculty of the University of Heidelberg were each trained to perform a defined task for a particular surgical OSCE station. Definitions were set and operationalized for an excellent and a borderline performance. In a simulated OSCE during the first part of the study, the standardized student performances were assessed and graded by different examiners three times in succession; video recordings were made. Quantitative and qualitative analysis of the videos was also undertaken by the study coordinator. In the second part of the study, the videos were used to investigate the examiners' acceptance of standardized examinees and to analyze potential influences on scoring that stemmed from the examiners' experience. Results: In the first part of the study, the OSCE scores and subsequent video analysis showed that standardization for defined performance levels at different OSCE stations is generally possible. Individual deviations from the prescribed examinee responses were observed and occurred primarily with increased complexity of OSCE station content. In the second part of the study, inexperienced examiners assessed a borderline performance significantly lower than their experienced colleagues (13.50 vs. 15.15, p=0.035). No difference was seen in the evaluation of the excellent examinees. Both groups of examiners graded the item "ocial competence" - despite identical standardization - significantly lower for examinees with borderline performances than for excellent examinees (4.13 vs. 4.80, p<0.001). Conclusion: Standardization of examinees for previously defined performance levels is possible, making a new tool available in future not only for OSCE quality assurance, but also for training examiners. Detailed preparation of the OSCE checklists and intensive training of the examinees are essential. This new tool takes on a special importance if standardized OSCEs are integrated into state medical exams and, as such, become high-stakes assessments.

Validation of a competence-based assessment of medical students' performance in the physician's role

BACKGROUND

Assessing competence of advanced undergraduate medical students based on performance in the clinical context is the ultimate, yet challenging goal for medical educators to provide constructive alignment between undergraduate medical training and professional work of physicians. Therefore, we designed and validated a performance-based 360-degree assessment for competences of advanced undergraduate medical students.

METHODS

This study was conducted in three steps: 1) Ten facets of competence considered to be most important for beginning residents were determined by a ranking study with 102 internists and 100 surgeons. 2) Based on these facets of competence we developed a 360-degree assessment simulating a first day of residency. Advanced undergraduate medical students (year 5 and 6) participated in the physician's role. Additionally knowledge was assessed by a multiple-choice test. The assessment was performed twice (t1 and t2) and included three phases: a consultation hour, a patient management phase, and a patient handover. Sixty-seven (t1) and eighty-nine (t2) undergraduate medical students participated. 3) The participants completed the Group Assessment of Performance (GAP)-test for flight school applicants to assess medical students' facets of competence in a non-medical context for validation purposes. We aimed to provide a validity argument for our newly designed assessment based on Messick's six aspects of validation: (1) content validity, (2) substantive/cognitive validity, (3) structural validity, (4) generalizability, (5) external validity, and (6) consequential validity.

RESULTS

Our assessment proved to be well operationalised to enable undergraduate medical students to show their competences in performance on the higher levels of Bloom's taxonomy. Its generalisability was underscored by its authenticity in respect of workplace reality and its underlying facets of competence relevant for beginning residents. The moderate concordance with facets of competence of the validated GAP-test provides arguments of convergent validity for our assessment. Since five aspects of Messick's validation approach could be defended, our competence-based 360-degree assessment format shows good arguments for its validity.

CONCLUSION

According to these validation arguments, our assessment instrument seems to be a good option to assess competence in advanced undergraduate medical students in a summative or formative way. Developments towards assessment of postgraduate medical trainees should be explored.

Effects and Consequences of Being an OSCE Examiner in Surgery-A Qualitative Study

OBJECTIVE

Even though objective structured clinical examination (OSCE) is a well-investigated format for competency-based practical examination, only a few studies have explored the motivations of OSCE examiners and their opinions, both positive and negative, toward being an examiner. The aim of this study was to gain insights into the views of OSCE examiners using semi-structured interviews.

DESIGN

Surgical OSCE examiners were queried at two medical faculties in Germany via semi-structured interviews. The interviews were transcribed verbatim and analyzed using the techniques of structured qualitative content analysis.

SETTING

This study was conducted at the medical faculties of the Goethe University, Frankfurt, Germany and of the Otto-von-Guericke University, Magdeburg, Germany.

PARTICIPANTS

All of the study participants were surgeons working at the university hospital of one of the faculties.

RESULTS

A total of 29 examiners were queried until a saturation of content was achieved. A critical reflection of one's own teaching was described as a major benefit by most participants. Furthermore, they noted that the standards and competences examined during the OSCE boosted the detail of their teaching sessions in the wards. However, the examiners criticized missed operations due the examination and were not appreciated by superiors for being an examiner. Most of the examiners (22/29) preferred to be an examiner themselves rather than appointing student peer examiners. If they had appointed someone else, that would mean they would miss valuable experiences useful for their own teaching.

CONCLUSIONS

Being an OSCE examiner confers several advantages, notably the reflection of one's own teaching, which was described as highly valuable by the examiners.

[Examinations while studying medicine - more than simply grades]

Assessment drives learning. Examinations need to be aligned primarily with learning objectives, as well as teaching and assessment methods of the courses on offer. In doing so, various examination instruments are required to measure on levels of competency that build on one another. An appropriate mix is essential to reflect the variety of learning outcomes of a chosen curriculum. Furthermore, examinations also possess the characteristics of evaluation: They reflect the knowledge and abilities of students and assess the teaching at a defined location. Digital examinations in the form of multiple-choice-question (MCQ) testing enable a higher degree of automation and accelerate the processes of creation, implementation, and evaluation of the examination results. Thus, they enjoy increasing popularity, provided that the technical requirements for large semester cohorts are met. Shifting examination processes to computers or tablets entails not only a wealth of new challenges but also opportunities.