Peer assessment in the objective structured clinical examination: A scoping review

Perceived and actual value of Student-led Objective Structured Clinical Examinations

INTRODUCTION

Student-led Objective Structured Clinical Examinations (OSCEs) provide formative learning opportunities prior to Faculty-led OSCEs. It is important to undertake quality assurance measurements of peer-led assessments because, if they are found to be unreliable and invalid, they may have detrimental impacts. The objectives of this study were to explore and evaluate Student-led OSCEs hosted by fifth-year medical students.

METHODS

Student-led OSCE results were analysed to examine reliability (Cronbach's alpha). The relationship between Student-led and Faculty-led OSCEs was evaluated using linear regression. Qualitative data were acquired by survey and semi-structured interviews and were analysed using an inductive content analysis approach.

RESULTS

In total, 85 (94%) of 91 eligible students consented to study participation. Student-led OSCEs had a low-moderate reliability [Cronbach alpha = 0.47 (primary care) and 0.61 (human reproduction/paediatrics) (HRH)]. A statistically significant, positive relationship between Student-led and Faculty-led OSCE results was observed. Faculty-led OSCE grades increased by 0.49 (95% CI: 0.18, 0.80) to 1.09 (95% CI: 0.67, 1.52), for each percentage increase in Student-led OSCE result. Student-led OSCE participants highly valued the authentic peer-assessed experience. Reported benefits included a reduction of perceived stress and anxiety prior to Faculty-led OSCEs, recognition of learning gaps, contribution to overall clinical competency and facilitation of collaboration between peers.

DISCUSSION

Student-led OSCEs are moderately reliable and can predict Faculty-led OSCE performance. This form of near-peer assessment encourages the metacognitive process of reflective practice and can be effectively implemented to direct further study. Faculties should collaborate with their student bodies to facilitate Student-led OSCEs and offer assistance to improve the quality, and benefits, of these endeavours.

Comparison of peer, self, and faculty objective structured clinical examination evaluations in a PharmD nonprescription therapeutics course

PURPOSE

Objective structured clinical examinations (OSCE) are a valuable assessment within healthcare education, as they provide the opportunity for students to demonstrate clinical competency, but can be resource intensive to provide faculty graders. The purpose of this study was to determine how overall OSCE scores compared between faculty, peer, and self-evaluations within a Doctor of Pharmacy (PharmD) curriculum.

METHODS

This study was conducted during the required nonprescription therapeutics course. Seventy-seven first-year PharmD students were included in the study, with 6 faculty members grading 10-15 students each. Students were evaluated by 3 graders: self, peer, and faculty. All evaluators utilized the same rubric. The primary endpoint of the study was to compare the overall scores between groups. Secondary endpoints included interrater reliability and quantification of feedback type based on the evaluator group.

RESULTS

The maximum possible score for the OSCE was 50 points; the mean scores for self, peer, and faculty evaluations were 43.3, 43.5, and 41.7 points, respectively. No statistically significant difference was found between the self and peer raters. However, statistical significance was found in the comparison of self versus faculty (p = 0.005) and in peer versus faculty (p < 0.001). When these scores were correlated to a letter grade (A, B, C or less), higher grades had greater similarity among raters compared to lower scores. Despite differences in scoring, the interrater reliability, or W score, on overall letter grade was 0.79, which is considered strong agreement.

CONCLUSIONS

This study successfully demonstrated how peer and self-evaluation of an OSCE provides a comparable alternative to traditional faculty grading, especially in higher performing students. However, due to differences in overall grades, this strategy should be reserved for low-stakes assessments and basic skill evaluations.

What do we know about Objective Structured Clinical Examination in Sport and Exercise Medicine? A scoping review

Background and objectives

Despite the importance of the Objective Structured Clinical Examination (OSCE) in Sport and Exercise Medicine, the literature on the topic is fragmented and has been poorly developed. The goal of this review was to map current knowledge about how the OSCE is used in Sport and Exercise Medicine, and to identify knowledge gaps for future research.

Method

The authors conducted a scoping review. They searched PubMed and Scopus for articles using key terms related to 'OSCE' and 'sport medicine' with no limit on search start date and up to July 2022. Retrieved records were imported, abstracts were screened, and full-text articles were reviewed. A forward and backward citation tracking was conducted. Data was extracted and a qualitative meta-summary of the studies was conducted.

Results

A total of 469 records were screened, and 22 studies were included. The objectives of the studies included using OSCEs to assess knowledge/skills after a training program (n = 11), to assess an intervention (n = 8), and to assess and improve the OSCE itself (n = 3). Thirteen studies reported validity and/or reliability of the OSCE.

Conclusion

Despite the widespread use of OSCEs in the examination of Sport and Exercise Medicine trainees, only a handful of scholarly works have been published. More research is needed to support the use of OSCE in Sport and Exercise Medicine for its initial purpose. We highlight avenues for future research such as assessing the need for a deeper exploration of the relationship between candidate characteristics and OSCE scores.

Objective structured clinical examination to teach competency in planetary health care and management - a prospective observational study

BACKGROUND

Health professionals are increasingly called upon and willing to engage in planetary health care and management. However, so far, this topic is rarely covered in medical curricula. As the need for professional communication is particularly high in this subject area, this study aimed to evaluate whether the objective structured clinical examination (OSCE) could be used as an accompanying teaching tool.

METHODS

During the winter semester 2022/2023, 20 third- and fifth-year medical students voluntarily participated in a self-directed online course, three workshops, and a formal eight-station OSCE on planetary health care and management. Each examinee was also charged alternatingly as a shadower with the role of providing feedback. Experienced examiners rated students' performance using a scoring system supported by tablet computers. Examiners and shadowers provided timely feedback on candidates` performance in the OSCE. Immediately after the OSCE, students were asked about their experience using a nine-point Likert-scale survey and a videotaped group interview. Quantitative analysis included the presentation of the proportional distribution of student responses to the survey and of box plots showing percentages of maximum scores for the OSCE performance. The student group interview was analyzed qualitatively.

RESULTS

Depending on the sub-theme, 60% -100% of students rated the subject of planetary health as likely to be useful in their professional lives. Similar proportions (57%-100%) were in favour of integrating planetary health into required courses. Students perceived learning success from OSCE experience and feedback as higher compared to that from online courses and workshops. Even shadowers learned from observation and feedback discussions. Examiners assessed students' OSCE performance at a median of 80% (interquartile range: 83%-77%) of the maximum score.

CONCLUSIONS

OSCE can be used as an accompanying teaching tool for advanced students on the topic of planetary health care and management. It supports learning outcomes, particularly in terms of communication skills to sensitise and empower dialogue partners, and to initiate adaptation steps at the level of individual patients and local communities.

Peer assessment: Development and delivery of the OSCE

INTRODUCTION

There is an expectation that healthcare professionals display competence in teaching, assessment and providing feedback. Development begins with formative peer-assisted learning and teaching in the undergraduate environment. Using peers or near-peers (in this case having 1 year more experience than the examination cohort) to provide assessment in summative exams remains unexplored. This study investigates how the use of near-peers compares to marking by academic staff in a summative OSCE.

MATERIALS AND METHODS

BDS4 Peer assessors (PAs) developed an OSCE question and marking schedule. Each PA (n = 3) was paired with an academic staff assessor (ASA) (n = 3). Peer and academic marked the candidates independently. Two years later, the process was repeated on the same cohort of candidates with the PA now 1-year post qualification. Statistical analysis compared the scores awarded by PA during each timeframe and against the marks awarded by the ASA.

RESULTS

During round 1, 28 students (62.2%) were awarded the same score by PA and ASA. On 17 occasions, there was a discrepancy (37.8%). Bias was skewed in favour of PA scoring higher (mean difference of differences -0.0667). During round 2, 27 students (55.1%) were awarded the same score by PA and ASA. On 22 occasions (44.9%), there was a discrepancy. Bias was skewed in favour of ASA scoring higher (mean difference of differences 0.0612).

DISCUSSION

Levels of agreement between PA and ASA are strong. Our results show PA mark more leniently as undergraduates and less leniently at 1-year post graduation.

CONCLUSIONS

Peer assessors are able to write OSCE stations, produce marking schemes and effectively assess their near-peers.

The SPEER: An interprofessional team behavior rubric to optimize geriatric clinical care

Geriatric patients with complex health care needs can benefit from interprofessional (IP) care; however, a major gap in health professional education is determining how to prepare future providers for IP collaboration. Effective IP team behavior assessment tools are needed to teach, implement, and evaluate IP practice skills. After review of IP evaluation tools, the Standardized Patient Encounter Evaluation Rubric (SPEER) was created to evaluate team dynamics in IP practice sites.Independent sample t-tests between faculty and learner SPEER scores showed learners scored themselves 15 points higher than their faculty scores (p < .001). Cronbach's α showed high internal consistency (α = 0.91). Paired t-tests found that learners identified improvements in the team's ability to address the patient's education needs and to allow the patients to voice their expectations. Faculty identified improvements in the teams' ability to make recommendations. Faculty evaluations of learner teams showed improvements in raw ratings on all but two items. Qualitative data analysis for emergent themes showed learners desired team functioning feedback and how teamwork could improve to provide optimal IP care.In conclusion, the SPEER can help faculty and learners identify growth in their teams' ability to perform key IP skills in clinical sites.

The Objective Structured Clinical Examination (OSCE) in Periodontology with Simulated Patient: The Most Realistic Approach to Clinical Practice in Dentistry

The objective structured clinical examination (OSCE) is becoming an increasingly established assessment test in dental schools. The use of simulated patients in the OSCE makes the stations more similar to clinical practice. Therefore, the student can show their technical and clinical knowledge, and certainly, their ability to manage the patient. These sorts of tests, in which simulated patients can be included, would be used before the student started clinical practice with patients and/or at the end of the degree. The objective of this work was to describe how the periodontology station was developed using a simulated patient for students of a fifth year dentistry degree taking an OSCE test. Furthermore, a questionnaire was created to learn the perception of the students about this station and its characteristics. The fifth year students at the European University of Madrid positively evaluated this station in their examination. In addition, it was recorded that they preferred a simulated patient in their tests, rather than stations with clinical cases, images, X-rays, and presentations. It is essential that once the OSCE has been completed, the student receives a feedback to learn where they have failed and, therefore, be able to improve any of the aspects evaluated in the station.

Simulation-based summative assessment in healthcare: an overview of key principles for practice

BACKGROUND

Healthcare curricula need summative assessments relevant to and representative of clinical situations to best select and train learners. Simulation provides multiple benefits with a growing literature base proving its utility for training in a formative context. Advancing to the next step, "the use of simulation for summative assessment" requires rigorous and evidence-based development because any summative assessment is high stakes for participants, trainers, and programs. The first step of this process is to identify the baseline from which we can start.

METHODS

First, using a modified nominal group technique, a task force of 34 panelists defined topics to clarify the why, how, what, when, and who for using simulation-based summative assessment (SBSA). Second, each topic was explored by a group of panelists based on state-of-the-art literature reviews technique with a snowball method to identify further references. Our goal was to identify current knowledge and potential recommendations for future directions. Results were cross-checked among groups and reviewed by an independent expert committee.

RESULTS

Seven topics were selected by the task force: "What can be assessed in simulation?", "Assessment tools for SBSA", "Consequences of undergoing the SBSA process", "Scenarios for SBSA", "Debriefing, video, and research for SBSA", "Trainers for SBSA", and "Implementation of SBSA in healthcare". Together, these seven explorations provide an overview of what is known and can be done with relative certainty, and what is unknown and probably needs further investigation. Based on this work, we highlighted the trustworthiness of different summative assessment-related conclusions, the remaining important problems and questions, and their consequences for participants and institutions of how SBSA is conducted.

CONCLUSION

Our results identified among the seven topics one area with robust evidence in the literature ("What can be assessed in simulation?"), three areas with evidence that require guidance by expert opinion ("Assessment tools for SBSA", "Scenarios for SBSA", "Implementation of SBSA in healthcare"), and three areas with weak or emerging evidence ("Consequences of undergoing the SBSA process", "Debriefing for SBSA", "Trainers for SBSA"). Using SBSA holds much promise, with increasing demand for this application. Due to the important stakes involved, it must be rigorously conducted and supervised. Guidelines for good practice should be formalized to help with conduct and implementation. We believe this baseline can direct future investigation and the development of guidelines.

Evaluation of the Experience of Peer-led Mock Objective Structured Practical Examination for First- and Second-year Medical Students

BACKGROUND

The objective structured practical examination (OSPE) is used as an assessment tool of laboratory practical sessions. This study described the design and implementation of peer-led mock OSPE for first- and second-year medical students, investigated the perception of the students of the peer-led mock OSPE and the impact of attending the mock OSPE on the performance.

METHODS

This is a cross-sectional study. Two mock OSPEs were designed and conducted by third-, fourth- and fifth- year medical students for year one and two. Each mock OSPE involved six stations. Thirty-three medical students facilitated the OSPE. The OSPEs were conducted prior to the summative end of block exams. Following the mock OSPEs, an online survey was sent to the participants to assess their satisfaction, quality and benefits of the mock OSPE. The study also evaluated the impact of the mock OSPE on students' performance.

RESULTS

Out of 313 first-year students, 279 (89.1%) attended the mock OSPE and out of 298 second-year students, 213 (71.5%) attended. A total of 192 (68.8%) first-year medical students and 102 (47.9%) second-year medical students completed the questionnaire. There was no significant difference between attending and non-attending the mock OSPE in the students' performance in the summative OSPE. The majority of students felt more confident, less anxious, and lowered the levels of stress after attending the mock OSPE. More than half of the students felt that attending the mock OSPE helped in easing the steps, better preparation, provided sufficient orientation, well explained the materials and helped them to learn the concept of the final OSPE. The majority of students found the mock OSPE stimulating.

CONCLUSION

Attending the mock OSPE did not affect the students' performance in the summative OSPE. However, the peer-assessed mock OSPE improved the medical students' confidence and lowered the anxiety associated with OSPE.

The pedagogical value of near-peer feedback in online OSCEs

PURPOSE OF THE ARTICLE

During the Covid-19 pandemic, formative OSCE were transformed into online OSCE, and senior students (near peers) substituted experienced clinical teachers. The aims of the study were to evaluate quality of the feedbacks given by near peers during online OSCEs and explore the experience of near-peer feedback from both learner's and near peer's perspectives.

MATERIALS AND METHODS

All 2nd year medical students (n = 158) attended an online OSCE under the supervision of twelve senior medical students. Outcome measures were 1) students' perception of the quality of the feedback through an online survey (Likert 1-5); 2) objective assessment of the quality of the feedback focusing on both the process and the content using a feedback scale (Likert 1-5); 3) experience of near peer feedback in two different focus groups.

RESULTS

One hundred six medical students answered the questionnaire and had their feedback session videotaped. The mean perceived overall quality of senior students' overall feedback was 4.75 SD 0.52. They especially valued self-evaluation (mean 4.80 SD 0.67), balanced feedback (mean 4.93 SD 0.29) and provision of simulated patient's feedback (mean 4.97 SD 0.17). The overall objective assessment of the feedback quality was 3.73 SD 0.38: highly scored skills were subjectivity (mean 3.95 SD 1.12) and taking into account student's self-evaluation (mean 3.71 (SD 0.87). Senior students mainly addressed history taking issues (mean items 3.53 SD 2.37) and communication skills (mean items 4.89 SD 2.43) during feedback. Participants reported that near peer feedback was less stressful and more tailored to learning needs- challenges for senior students included to remain objective and to provide negative feedback.

CONCLUSION

Increased involvement of near peers in teaching activities is strongly supported for formative OSCE and should be implemented in parallel even if experience teachers are again involved in such teaching activities. However, it requires training not only on feedback skills but also on the specific content of the formative OSCE.

Professionalism and clinical short answer question marking with machine learning

Machine learning may assist in medical student evaluation. This study involved scoring short answer questions administered at three centres. Bidirectional encoder representations from transformers were particularly effective for professionalism question scoring (accuracy ranging from 41.6% to 92.5%). In the scoring of 3-mark professionalism questions, as compared with clinical questions, machine learning had a lower classification accuracy (P < 0.05). The role of machine learning in medical professionalism evaluation warrants further investigation.

Evaluation of Interprofessional Education on Effective Communication Between Pharmacy and Physician Assistant Students

PURPOSE

The purpose of this study was to assess the effect of Team Strategies and Tools to Enhance Performance and Patient Safety (TeamSTEPPS) on student self-perceived competencies and perceptions of interprofessional (IP) communication and teamwork in a clinical case review activity. TeamSTEPPS is an evidence-based curriculum that is used to enhance and support IP healthcare communication.

METHODS

A repeated-measures, pretest/posttest study evaluated physician assistant students' and student pharmacists' perceptions of TeamSTEPPS. Students completed Performance Assessment for Communication and Teamwork (PACT) surveys, evaluating teamwork, knowledge, attitudes, and skills perceptions before and after a TeamSTEPPS lecture and associated activity with peer feedback.

RESULTS

Overall, 87.4% (n = 429) completed pre- and post-PACT surveys. Apart from the Mutual Support domain (p = .898), all changes were significantly positive (p < .004), with the greatest improvements occurring in the Attitudes and Perceived Skills domains.

CONCLUSION

TeamSTEPPS IP education, application, and peer feedback improved students' perceptions of multiple domains, including effective communication. Using TeamSTEPPS tools in IP formats enabled the students to safely practice and collaborate in preparation for clinical practice.

Perceptions of Physical Therapy Students on their Psychomotor Examinations: a Qualitative Study

INTRODUCTION

Practical examinations are necessary to demonstrate learning in the psychomotor, cognitive, and affective domains. Student perceptions of the organization and execution of practical examinations are an important consideration in the development of practical examinations.

REVIEW OF THE LITERATURE

Multiple other health professions have investigated students' perceptions of objective structured clinical examinations (OSCE). There is little in the physical therapy literature with respect to student perception regarding proctor presence during practical examinations or OSCEs.

SUBJECTS

The participants were members of the classes of 2019-2021 in a Doctor of Physical Therapy (DPT) program at a New England University.

METHODS

A qualitative thematic approach was applied to de-identified transcripts of student focus group interviews. Independently coded themes were identified, discussed, and refined iteratively.

RESULTS AND DISCUSSION

Four themes emerged with multiple subthemes: impact of proctor being present; realistic, patient-focused experience; preparation for the practical; and stress. Students valued preparation that included clear expectations, utilization of formative assessments, and peer feedback prior to the practical. They also noted that a distractive-free testing space, having no proctor present in the room, recording the practical, and the format of OSCE's decreased stress and improved performance.

CONCLUSIONS

These findings add to the body of knowledge in physical therapy and provide guidance to faculty as they plan and organize practical examinations.

Assessing the utility of virtual OSCE sessions as an educational tool: a national pilot study

BACKGROUND

Objective Structured Clinical Examinations (OSCEs) are a common form of assessment used across medical schools in the UK to assess clinical competence and practical skills and are traditionally held in an in-person format. In the past, medical students have often prepared for such exams through in-person peer-assisted learning (PAL), however, due to the recent Covid-19 pandemic, many in-person teaching sessions transitioned to online-based formats. There is currently a paucity of research on the utility of virtual PAL OSCE sessions and thus, we carried out a national pilot study to determine the feasibility of virtual OSCE teaching via feedback from participants and examiners.

METHODS

A total of 85 students from 19 UK-based medical schools with eight students based internationally attended the series of online OSCE workshops delivered via Zoom®. All students and examiners completed a feedback questionnaire at the end of each session regarding parameters, which included questions on pre-and post-workshop confidence in three OSCE domains: history-taking, communication and data interpretation. A Likert scale using 5 Likert items was used to self-report confidence, and the results were analysed using the Mann-Whitney U test after assessing for normality using the Shapiro-Wilk test.

RESULTS

Results from student feedback showed an increase in confidence for all three OSCE domains after each event (p < 0.001) with 69.4% agreeing or strongly agreeing that online OSCE sessions could sufficiently prepare them for in-person exams. Questionnaire feedback revealed that 97.6% of students and 86.7% of examiners agreed that virtual OSCE teaching would be useful for preparing for in-person OSCE examinations after the pandemic.

CONCLUSION

Most participants in the virtual OSCE sessions reported an improvement in their confidence in history-taking, communication and data interpretation skills. Of the participants and examiners that had also experienced in-person OSCE examinations, the majority also reported that they found virtual OSCE sessions to be as engaging and as interactive as in-person teaching. This study has demonstrated that virtual OSCE workshops are a feasible option with the potential to be beneficial beyond the pandemic. However, more studies are required to assess the overall impact on student learning and to determine the value of virtual OSCE workshops on exam performance.

Observed structured clinical examination as a means of assessing clinical skills competencies of ANPs

Observed structured clinical examinations (OSCEs) are a common method of assessment within higher education to prepare for the advanced nurse practitioner (ANP) role. This article reviews a wide range of literature relating to OSCE assessment in the healthcare arena, from an ANP, interprofessional and advanced clinical practice perspective. Theories underpinning OSCE and advanced nursing roles are explored, with relevant supporting literature indicating how established OSCEs can become integrated with other methods to improve outcomes for this level of practice. Alternative assessments are explored with specific reference to the context of the education delivered, and the suitability for higher education today.

Tutor-Student Partnership in Practice OSCE to Enhance Medical Education

BACKGROUND

Training of examiners is essential to ensure the quality of objective structured clinical examination (OSCE). We aimed to study a perceived effectiveness of tutor-student partnership in a practice OSCE module by novice OSCE tutors and medical students.

METHOD

We implemented a practice OSCE at a medical faculty in France with novice tutors and third year medical students as partners. Each tutor (n = 44) served as a partner for the group of 5 students in the conception of the scenario and as an evaluator of the tutored station. Students (n = 303) were involved in the conception of a case and the roles of a physician, evaluator and a simulated patient. Data were obtained through self-assessment questionnaires. Descriptive statistics were used to analyze items of the questionnaires. Free-form answers were coded and analyzed thematically.

RESULTS

A total of 36 tutors (82%) and 185 students (61%) responded to the questionnaires. The intervention was well perceived. Thirty-two percent of the tutors reported some difficulties in the assessment of student performance and were disposed to receive further training. Fifty-five percent of the students considered the participation in the OSCE case development appropriate to their level of knowledge, and 70% perceived it as beneficial allowing them to set their learning goals.

CONCLUSION

This initiative provides a relevant method beneficial to OSCE tutors, medical students, and the faculty. Tutors learn how to assess student performance according to expected achievement levels. It allows students to be engaged as partners in co-creation of learning and teaching.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s40670-021-01421-9.

The predictors of nursing students' clinical competency in pre-internship objective structured clinical examination: The roles of exam anxiety and academic success

BACKGROUND

Identifying the predictors of nursing students' clinical competency in objective structured clinical examination (OSCE) is of utmost importance. Therefore, the present study was to investigate the predictive roles of exam anxiety and academic success in nursing students' clinical competency in the pre-internship OSCE.

METHODS

In this descriptive-analytical study, a total number of 102 nursing students, meeting the inclusion criteria and enrolled in the sixth semester (third year) of the graduate nursing program in Iran, were firstly selected by the census method. The pre-internship OSCE was then implemented at eight stations based on a pre-designed schedule template. The required data were also collected through a demographic-academic characteristics questionnaire and the State-Trait Anxiety Inventory to determine the exam anxiety score, the nursing program grade point average (GPA) to reflect on the levels of academic success, and the OSCE score to control clinical competency. The data analysis was also performed at the significance level of 0.05.

RESULTS

The linear regression model, in which the exam anxiety, the nursing program GPA, and the demographic-academic characteristics variables had been imported, could explain 33.52% of the variance of the nursing students' clinical competency in the pre-internship OSCE (R² = 0.616). Of the variables concerned, only the nursing program GPA could be a significant predictor of the nursing students' clinical competency scores in the OSCE, so that 0.8 points were added to the clinical competency scores in the OSCE as the nursing program GPA increased by one unit (p = 0.000, β = 0.717), but no significant relationship was observed between exam anxiety and clinical competency in the pre-internship OSCE among the nursing students.

CONCLUSIONS

The results of this study endorsed the use of the OSCE in assessing the nursing students' clinical competency and implementing learning strategies to strengthen the levels of academic success in such individuals.

The role of training in student examiner rating performance in a student-led mock OSCE

INTRODUCTION

Peer assessments are increasingly prevalent in medical education, including student-led mock Objective Structured Clinical Examinations (OSCE). While there is some evidence to suggest that examiner training may improve OSCE assessments, few students undergo training before becoming examiners. We sought to evaluate an examiner training programme in the setting of a student-led mock OSCE.

METHODS

A year‑2 mock OSCE comprised of history taking (Hx) and physical examination (PE) stations was conducted involving 35 year‑3 (Y3) student examiners and 21 year‑5 (Y5) student examiners who acted as reference examiners. Twelve Y3 student-examiners attended an OSCE examiner training programme conducted by senior faculty. During the OSCE, Y3 and Y5 student examiners were randomly paired to grade the same candidates and scores were compared. Scores for checklist rating (CR) and global rating (GR) domains were assigned for both Hx and PE stations.

RESULTS

There was moderate to excellent correlation between Y3 and Y5 student examiners for both Hx (ICC 0.71-0.96) and PE stations (ICC 0.71-0.88) across all domains. For both Hx and PE stations, GR domain had poorer correlation than CR domains. Examiner training resulted in better correlations for PE but not Hx stations. Effect sizes were lower than the minimum detectible effect (MDE) sizes for all comparisons made.

DISCUSSION

Y3 student examiners are effective substitutes for Y5 student examiners in a Y2 mock OSCE. Our findings suggest that examiner training may further improve marking behaviour especially for PE stations. Further studies with larger sample sizes are required to further evaluate the effects of dedicated examiner training.

Assessment of medical students' clinical performance using high-fidelity simulation: comparison of peer and instructor assessment

BACKGROUND

High-fidelity simulators are highly useful in assessing clinical competency; they enable reliable and valid evaluation. Recently, the importance of peer assessment has been highlighted in healthcare education, and studies using peer assessment in healthcare, such as medicine, nursing, dentistry, and pharmacy, have examined the value of peer assessment. This study aimed to analyze inter-rater reliability between peers and instructors and examine differences in scores between peers and instructors in the assessment of high-fidelity-simulation-based clinical performance by medical students.

METHODS

This study analyzed the results of two clinical performance assessments of 34 groups of fifth-year students at Ajou University School of Medicine in 2020. This study utilized a modified Queen's Simulation Assessment Tool to measure four categories: primary assessment, diagnostic actions, therapeutic actions, and communication. In order to estimate inter-rater reliability, this study calculated the intraclass correlation coefficient and used the Bland and Altman method to analyze agreement between raters. A t-test was conducted to analyze the differences in evaluation scores between colleagues and faculty members. Group differences in assessment scores between peers and instructors were analyzed using the independent t-test.

RESULTS

Overall inter-rater reliability of clinical performance assessments was high. In addition, there were no significant differences in overall assessment scores between peers and instructors in the areas of primary assessment, diagnostic actions, therapeutic actions, and communication.

CONCLUSIONS

The results indicated that peer assessment can be used as a reliable assessment method compared to instructor assessment when evaluating clinical competency using high-fidelity simulators. Efforts should be made to enable medical students to actively participate in the evaluation process as fellow assessors in high-fidelity-simulation-based assessment of clinical performance in situations similar to real clinical settings.

Scoping reviews in medical education: A scoping review

OBJECTIVES

Over the last two decades, the number of scoping reviews in core medical education journals has increased by 4200%. Despite this growth, research on scoping reviews provides limited information about their nature, including how they are conducted or why medical educators undertake this knowledge synthesis type. This gap makes it difficult to know where the field stands and may hamper attempts to improve the conduct, reporting and utility of scoping reviews. Thus, this review characterises the nature of medical education scoping reviews to identify areas for improvement and highlight future research opportunities.

METHOD

The authors searched PubMed for scoping reviews published between 1/1999 and 4/2020 in 14 medical education journals. The authors extracted and summarised key bibliometric data, the rationales given for conducting a scoping review, the research questions and key reporting elements as described in the PRISMA-ScR. Rationales and research questions were mapped to Arksey and O'Malley's reasons for conducting a scoping review.

RESULTS

One hundred and one scoping reviews were included. On average, 10.1 scoping reviews (SD = 13.1, median = 4) were published annually with the most reviews published in 2019 (n = 42). Authors described multiple reasons for undertaking scoping reviews; the most prevalent being to summarise and disseminate research findings (n = 77). In 11 reviews, the rationales for the scoping review and the research questions aligned. No review addressed all elements of the PRISMA-ScR, with few authors publishing a protocol (n = 2) or including stakeholders (n = 20). Authors identified shortcomings of scoping reviews, including lack of critical appraisal.

CONCLUSIONS

Scoping reviews are increasingly conducted in medical education and published by most core journals. Scoping reviews aim to map the depth and breadth of emerging topics; as such, they have the potential to play a critical role in the practice, policy and research of medical education. However, these results suggest improvements are needed for this role to be fully realised.

An Advanced Communication Skills Workshop Using Standardized Patients for Senior Medical Students

INTRODUCTION

Medical students often lack training in advanced communication skills encompassing emotionally fraught situations and those in which an intense emotional response is expected. Such skills are required for clinical situations encountered during residency. We created and evaluated an advanced communication skills workshop (ACSW) using standardized patients for senior medical students. The workshop emphasized communication skills for four scenarios-strong emotion, goals of care, medical error, and palliative care assessment-and utilized formative peer assessment and feedback.

METHODS

We created the four ACSW cases with case-specific communication behavior checklists and a common modified Master Interview Rating Scale in a Capstone Course for senior medical students. In groups of three, students rotated through three of four stations. Each student conducted one of the interviews while the other two completed the checklists and provided verbal feedback. We performed one-way analyses of variance on Likert responses and content analysis on open responses on a post-ACSW survey.

RESULTS

Ninety-one students completed the ACSW and survey. Students assigned high value to all four ACSW student roles: interviewer, observer, feedback recipient, and feedback provider. Students rated the experience above average to excellent on nearly all survey items. Open-response themes included "liked the opportunity to give or receive peer feedback" (46%) and "found the checklists helpful" (45%).

DISCUSSION

Feasible and well received by senior medical students, our ACSW offers an opportunity to practice and observe advanced communication skills and peer feedback. A peer-assisted, formative learning model, the ACSW efficiently addresses a key aspect of residency preparation.

Developing Miller's Pyramid to Support Students' Assessment Literacy

Assessment literacy is increasingly recognized as an important concept to consider when developing assessment strategies for courses and programs. Assessment literacy approaches support students in their understanding of assessment expectations and help them both understand and optimize their performance in assessment. In this teaching tip, a model for assessment literacy that builds on the well-known Miller's Pyramid model for assessment in clinical disciplines is proposed and contextualized. The model progresses thinking from assessment methods themselves to consideration of the activities that need to be built into curricula to ensure that assessment literacy is addressed at each level of the pyramid. The teaching tip provides specific examples at each of the levels. Finally, the relevance of this work to overall curriculum design is emphasized.

Agreement between medical students' peer assessments and faculty assessments in advanced resuscitation skills examinations in South Korea

PURPOSE

In medical education, peer assessment is considered to be an effective learning strategy. Although several studies have examined agreement between peer and faculty assessments regarding basic life support (BLS), few studies have done so for advanced resuscitation skills (ARS) such as intubation and defibrillation. Therefore, this study aimed to determine the degree of agreement between medical students' and faculty assessments of ARS examinations.

METHODS

This retrospective explorative study was conducted during the emergency medicine (EM) clinical clerkship of fourth-year medical students from April to July 2020. A faculty assessor (FA) and a peer assessor (PA) assessed each examinee's resuscitation skills (including BLS, intubation, and defibrillation) using a checklist that consisted of 20 binary items (performed or not performed) and 1 global proficiency rating using a 5-point Likert scale. The prior examinee assessed the next examinee after feedback and training as a PA. All 54 students participated in peer assessment. The assessments of 44 FA/PA pairs were analyzed using the intraclass correlation coefficient (ICC) and Gwet's first-order agreement coefficient.

RESULTS

The PA scores were higher than the FA scores (mean±standard deviation, 20.2±2.5 [FA] vs. 22.3±2.4 [PA]; P<0.001). The agreement was poor to moderate for the overall checklist (ICC, 0.55; 95% confidence interval [CI], 0.31 to 0.73; P<0.01), BLS (ICC, 0.19; 95% CI, -0.11 to 0.46; P<0.10), intubation (ICC, 0.51; 95% CI, 0.26 to 0.70; P<0.01), and defibrillation (ICC, 0.49; 95% CI, 0.23 to 0.68; P<0.01).

CONCLUSION

Senior medical students showed unreliable agreement in ARS assessments compared to faculty assessments. If a peer assessment is planned in skills education, comprehensive preparation and sufficient assessor training should be provided in advance.

Formative peer assessment in higher healthcare education programmes: a scoping review

OBJECTIVES

Formative peer assessment focuses on learning and development of the student learning process. This implies that students are taking responsibility for assessing the work of their peers by giving and receiving feedback to each other. The aim was to compile research about formative peer assessment presented in higher healthcare education, focusing on the rationale, the interventions, the experiences of students and teachers and the outcomes of formative assessment interventions.

DESIGN

A scoping review.

DATA SOURCES

Searches were conducted until May 2019 in PubMed, Cumulative Index to Nursing and Allied Health Literature, Education Research Complete and Education Research Centre. Grey literature was searched in Library Search, Google Scholar and Science Direct.

ELIGIBILITY CRITERIA

Studies addressing formative peer assessment in higher education, focusing on medicine, nursing, midwifery, dentistry, physical or occupational therapy and radiology published in peer-reviewed articles or in grey literature.

DATA EXTRACTIONS AND SYNTHESIS

Out of 1452 studies, 37 met the inclusion criteria and were critically appraised using relevant Critical Appraisal Skills Programme, Joanna Briggs Institute and Mixed Methods Appraisal Tool tools. The pertinent data were analysed using thematic analysis.

RESULT

The critical appraisal resulted in 18 included studies with high and moderate quality. The rationale for using formative peer assessment relates to giving and receiving constructive feedback as a means to promote learning. The experience and outcome of formative peer assessment interventions from the perspective of students and teachers are presented within three themes: (1) organisation and structure of the formative peer assessment activities, (2) personal attributes and consequences for oneself and relationships and (3) experience and outcome of feedback and learning.

CONCLUSION

Healthcare education must consider preparing and introducing students to collaborative learning, and thus develop well-designed learning activities aligned with the learning outcomes. Since peer collaboration seems to affect students' and teachers' experiences of formative peer assessment, empirical investigations exploring collaboration between students are of utmost importance.

Formative assessment of practical skills with peer-assessors: quality features of an OSCE in general medicine at the Heidelberg Medical Faculty

Background: Objective Structured Clinical Examinations (OSCEs) have become an established examination format at German medical faculties. Medical experts routinely use a summative assessment to evaluate practical and communicative skills, while the use of the OSCE format by student examiners, as a formative examination, remains rather limited. Objective: The formative OSCE program of the Department of General Practice and Implementation Research at the Heidelberg Medical Faculty, which is conducted and evaluated by peer tutors, is examined with regard to its quality criteria and compared with summative OSCEs from other departments. Methods: Difficulties and discriminatory power of individual testing stations were determined for the summative, as well as the formative OSCE, and compared with each other. To assess the reliability of the measurements, an analysis of the data was carried out using the Generalizability theory. In addition, a comparison is made between the assessments of student examiners and second assessments by medical experts. Results: The stations of the formative OSCE show similar difficulties as those of the summative comparison OSCEs (Pform=0.882; Psum=0.845 - 0.902). With respect to measurement reliability, there are no differences between the OSCE in General Medicine and the other subjects. The assessments of student examiners and medical experts correlate highly (r=0.888). Conclusion: The formative OSCE in General Medicine is comparable to the summative comparison formats in terms of its quality criteria. The use of student examiners can be a reliable alternative to medical experts in formative OSCEs.

Assessing Advanced Communication Skills via Objective Structured Clinical Examination: A Comparison of Faculty Versus Self, Peer, and Standardized Patient Assessors

Construct: The construct addressed in this study is assessment of advanced communication skills among senior medical students. Background: The question of who should assess participants during objective structured clinical examinations (OSCEs) has been debated, and options discussed in the literature have included peer, self, standardized patient, and faculty assessment models. What is not known is whether same-level peer assisted learning can be utilized for formative assessment of advanced communication skills when no faculty, standardized patients, or other trained assessors are involved in providing feedback. If successful, such an educational model would optimize resource utilization and broaden the scope of topics that could be covered in formative OSCEs. Approach: The investigators developed a 4-station formative OSCE focused on advanced communication skills for senior medical students, and evaluated the concordance of assessment done by same-level peers, self, standardized patients, and faculty for 45 students. After each station, examinees completed a self-assessment checklist and received checklist-based assessment and verbal feedback from same-level peers only. Standardized patients completed checklist-based assessments outside the room, and faculty did so after the OSCE via video review; neither group provided direct feedback to examinees. The investigators assessed inter-rater agreement and mean difference scores on the checklists using faculty score as the gold standard. Findings: There was fair to good overall agreement among self, same-level peer, standardized patient, and faculty-assessment of advanced communication skills. Relative to faculty, peer and standardized patient assessors overestimated advanced communication skills, while self-assessments underestimated skills. Conclusions: Self and same-level peer-assessment may be a viable alternative to faculty assessment for a formative OSCE on advanced communication skills for senior medical students.

Impressions on Reliability and Students' Perceptions of Learning in a Peer-Based OSCE

BACKGROUND

Peer assessment of performance in the objective structured clinical examination (OSCE) is emerging as a learning instrument. While peers can provide reliable scores, there may be a trade-off with students' learning. The purpose of this study is to evaluate a peer-based OSCE as a viable assessment instrument and its potential to promote learning and explore the interplay between these two roles.

METHODS

A total of 334 medical students completed an 11-station OSCE from 2015 to 2016. Each station had 1-2 peer examiners (PE) and one faculty examiner (FE). Examinees were rated on a 7-point scale across 5 dimensions: Look, Feel, Move, Special Tests and Global Impression. Students participated in voluntary focus groups in 2016 to provide qualitative feedback on the OSCE. Authors analysed assessment data and transcripts of focus group discussions.

RESULTS

Overall, PE awarded higher ratings compared with FE, sources of variance were similar across 2 years with unique variance consistently being the largest source, and reliability (r _φ ) was generally low. Focus group analysis revealed four themes: Conferring with Faculty Examiners, Difficulty Rating Peers, Insider Knowledge, and Observing and Scoring.

CONCLUSIONS

While peer assessment was not reliable for evaluating OSCE performance, PE's perceived that it was beneficial for their learning. Insight gained into exam technique and self-appraisal of skills allows students to understand expectations in clinical situations and plan approaches to self-assessment of competence.

Peers as OSCE assessors for junior medical students - a review of routine use: a mixed methods study

BACKGROUND

Peer-assisted learning is well established in medical education; however, peer tutors rarely act as assessors for the OSCE. In the compulsory, near-peer teaching programme covering basic medical skills at the University of Heidelberg, peer tutors serve as assessors on a formative OSCE. This study aimed to investigate the feasibility and acceptance of peer assessors and to survey the perceived advantages and disadvantages of their use.

METHODS

In 2016 and 2017 all OSCE peer assessors (third to sixth-year medical students) and all of the peer-assessed students in 2017 (second-year-medical students) were invited to participate in a survey. Both groups were asked to complete a tablet-based questionnaire immediately after the OSCE. Peer assessors were asked to rate eight statements and the peer-assessed students to rate seven statements on a five-point Likert scale. Both were asked to comment on the advantages and disadvantages of peer-assessors.

RESULTS

Overall, 74 of 76 peer assessors and 307 of 308 peer-assessed students participated in the study. 94% (67/74) of peer assessors and 90% (276/307) of the peer-assessed group thought that it is important to have peer tutors as assessors. Of the peer assessors, 92% (68/74) felt confident in giving structured feedback during the OSCE and 66% (49/74) felt they had improved their teaching skills. Of the peer-assessed students, 99% (306/307) were satisfied with their peers as OSCE assessors and 96% (292/307) considered the peer feedback during the OSCE as helpful. The participants mentioned structural benefits, such as lower costs, and suggested the quality of the OSCE was higher due to the use of peer assessors. The use of peer assessors was found to be beneficial for the learners in the form of high-quality feedback and an overall reduction in stress. Furthermore, the use of peer assessors was found to be beneficial for the peer assessors (improved teaching and clinical skills).

CONCLUSION

From a learner's perspective, the use of peer assessors for a formative OSCE that is part of a near-peer teaching program aimed at junior medical students is favourable for all. A broad implementation of peer assessment in the formative OSCE should be encouraged to investigate effects on quality and stress-reduction.

Comparison of peer assessment and faculty assessment in an interprofessional simulation-based team training program

Challenges related to limited clinical sites and shortage of clinical instructors may reduce the quality of clinical experiences, leading to increased demand for the establishment of simulation-based training programs in the curricula of educational institutions. However, simulation-based training programs in health education place great demands on faculty resources. It is interesting, therefore, to investigate peers contributions in formal assessment, and how this compares to faculty assessment. This paper report the results from the comparison of direct observation by peer observers who had received short rater training, and post-hoc video-based assessment by trained facilitators. An observation form with six learning outcomes was used to rate team performance. Altogether 262 postgraduate nursing students, bachelor of nursing students and medical students participated, organized into 44 interprofessional teams. A total of 84 peers and two facilitators rated team performance. The sum score of all six learning outcomes showed that facilitators were more lenient than peer observers (p = .014). The inter-rater reliability varied considerably when comparing scores from peer observers from the three different professions with those of the facilitators. The results indicate that peer assessment may support, but not replace, faculty assessment.

No observed effect from a student-led mock objective structured clinical examination on subsequent performance scores in medical students in Canada

Student-led peer-assisted mock objective structured clinical examinations (MOSCEs) have been used in different settings to help students prepare for subsequent higher-stakes, faculty-run OSCEs. MOSCE participants generally valued feedback from peers and report benefits to learning. Our study investigated whether participation in a peer-assisted MOSCE affects subsequent OSCE performance. To determine whether mean OSCE scores differed depending on whether medical students participated in the MOSCE, we conducted a between-subjects analysis of variance (ANOVA), with cohort (2016 vs. 2017) and MOSCE participation (MOSCE vs. No MOSCE) as independent variables and mean OSCE score as the dependent variable. Participation in the MOSCE had no influence on mean OSCE scores (P=0.19). There was a significant correlation between mean MOSCE scores and mean OSCE scores (Pearson's r = 0.52, P<0.001). Whereas previous studies report self-reported benefits from participation in student-lead MOSCEs, it was not associated with objective benefits in this study.

Experiences of using an OSCE protocol in clinical examinations of nursing students - A comparison of student and faculty assessments

Peer-assessment in nursing education using an OSCE protocol is an increasing educational activity that complements that of teachers. However, little is known about how students' and teachers' assessments correspond. The study aimed to compare OSCE assessments made by student examiners and faculty examiners during examinations of clinical skills in undergraduate nursing education. Four cohorts of third-year nursing students participated between 2014 and 2016. The students underwent a clinical examination of the management of central venous catheters and totally implantable venous access devices. Students who performed the examinations were observed both by a faculty examiner and student examiner. Both observers used the same OSCE protocol for the assessment but independently. The OSCE protocols from both faculty and student examiners were reviewed and compared. Total agreement between the student and faculty examiner was reached in 127 of 135 (94%) paired protocols. The level of agreement was substantial with a kappa value of 0.79 (95% CI 0.65-0.93). The conclusion was that the level of agreement between student and faculty examiners was high when using an OSCE protocol in clinical examinations of two different clinical skill tasks. The structured checklist (OSCE protocol) was easy to use for the student examiners despite the lack of experience or training in advance.

Rating the quality of teamwork-a comparison of novice and expert ratings using the Team Emergency Assessment Measure (TEAM) in simulated emergencies

Background

Training in teamwork behaviour improves technical resuscitation performance. However, its effect on patient outcome is less clear, partly because teamwork behaviour is difficult to measure. Furthermore, it is unknown who should evaluate it. In clinical practice, experts are obliged to participate in resuscitation efforts and are thus unavailable to assess teamwork quality. Consequently, we sought to determine if raters with little clinical experience and experts provide comparable evaluations of teamwork behaviour.

Methods

Novice and expert raters judged teamwork behaviour during 6 emergency medicine simulations using the Teamwork Emergency Assessment Measure (TEAM). Ratings of both groups were analysed descriptively and compared with U and t tests. We used a mixed effects model to identify the proportion of variance in TEAM scores attributable to rater status and other sources.

Results

Twelve raters evaluated 7 teams rotating through 6 cases, for a total of 84 observations. We found no significant difference between expert and novice ratings for 7 of the 11 items of the TEAM or in the sums of all item scores. Novices rated teamwork behaviour higher on 4 items and overall. Rater status accounted for 11.1% of the total variance in scores.

Conclusions

Experts’ and novices’ ratings were similarly distributed, implying that raters with limited experience can provide reliable data on teamwork behaviour. Novices show a consistent, but slightly more lenient rating behaviour. Clinical studies and real-life teams may thus employ novices using a structured observational tool such as TEAM to inform their performance review and improvement.

Electronic supplementary material

The online version of this article (10.1186/s13049-019-0591-9) contains supplementary material, which is available to authorized users.

A student-initiated objective structured clinical examination as a sustainable cost-effective learning experience

BACKGROUND

The objective structured clinical examination (OSCE) has gained widespread use as a form of performance assessment. However, opportunities for students to participate in practice OSCEs are limited by the financial, faculty and administrative investments required.

OBJECTIVES

To determine the feasibility and acceptability of a student-run mock OSCE (MOSCE) as a learning experience for medical students of all 4 years.

DESIGN

We conducted a five-station MOSCE for third-year students. This involved fourth-year students as examiners and first-/second-year students as standardized patients (SPs). Each examiner scored examinees using a checklist and global rating scale while providing written and verbal feedback. MOSCE stations and checklists were designed by students and reviewed by a faculty supervisor. Following the MOSCE, participants completed surveys which elucidated their perceptions on the roles they took during the MOSCE.

RESULTS

Fifty examinees participated in the MOSCE. Of these, 42 (84%) consented to participate in the study and submitted completed questionnaires. Twenty-four examiners participated in the OSCE and consented to participate in the study, with 22 (92%) submitting completed questionnaires. Fifty-three of 60 SPs (88%) agreed to take part in this study, and 51 (85%) completed questionnaires. The internal consistency of the five-station OSCE was calculated as a Cronbach's alpha of 0.443. Students commented positively on having the opportunity to network and engage in mentorship activities and reinforce clinical concepts.

CONCLUSIONS

Examinees, examiners, and SPs all perceived the MOSCE to be a beneficial learning experience. We found the MOSCE to be a feasible and acceptable means of providing additional OSCE practice to students prior to higher-stakes evaluations.