Do students' self-reflections of performance align with their graded performance in objective structured clinical exams?

INTRODUCTION

Self-awareness of strengths and weaknesses through self-reflection are important for life-long learning and development. The aim of this study was to assess the alignment in third-year undergraduate pharmacy students' self-reflections of their objective structured clinical exam (OSCE) performance to their actual OSCE scores and explore the most common aspects students reflected on as markers of perceived performance.

METHODS

Students completed a three-station OSCE and a written self-reflection about their performance. These reflections were coded using a latent pattern content analysis, with categories defined as "doing well (≥ 50% on exam)" and "not doing well (< 50% on exam)" and compared to their actual OSCE exam scores, to determine the degree of alignment.

RESULTS

Two hundred sixty-nine students completed the OSCE and reflection. Students had a low degree of alignment between their self-reflections and actual OSCE performance. Low alignment was overwhelmingly prevalent and significant in high-achieving students with OSCE scores of ≥90%. Most common aspects students reflected on as indicators of performance were finishing on time and communicating effectively. High-achieving students reflected on aspects such as empathy, systematic questioning, and patient teach-back as aspects of their performance.

CONCLUSIONS

Student reflections on exam performance do not align with their actual performance, particularly amongst the high-achieving students. High-achieving students were more aware of the different aspects that affected their performance. To ensure that high-achieving students are aware of their strengths, educators should provide more targeted feedback mechanisms and positive reassurances to help these students become more confident in their decision-making skills.

Comparison of peer, self, and faculty objective structured clinical examination evaluations in a PharmD nonprescription therapeutics course

PURPOSE

Objective structured clinical examinations (OSCE) are a valuable assessment within healthcare education, as they provide the opportunity for students to demonstrate clinical competency, but can be resource intensive to provide faculty graders. The purpose of this study was to determine how overall OSCE scores compared between faculty, peer, and self-evaluations within a Doctor of Pharmacy (PharmD) curriculum.

METHODS

This study was conducted during the required nonprescription therapeutics course. Seventy-seven first-year PharmD students were included in the study, with 6 faculty members grading 10-15 students each. Students were evaluated by 3 graders: self, peer, and faculty. All evaluators utilized the same rubric. The primary endpoint of the study was to compare the overall scores between groups. Secondary endpoints included interrater reliability and quantification of feedback type based on the evaluator group.

RESULTS

The maximum possible score for the OSCE was 50 points; the mean scores for self, peer, and faculty evaluations were 43.3, 43.5, and 41.7 points, respectively. No statistically significant difference was found between the self and peer raters. However, statistical significance was found in the comparison of self versus faculty (p = 0.005) and in peer versus faculty (p < 0.001). When these scores were correlated to a letter grade (A, B, C or less), higher grades had greater similarity among raters compared to lower scores. Despite differences in scoring, the interrater reliability, or W score, on overall letter grade was 0.79, which is considered strong agreement.

CONCLUSIONS

This study successfully demonstrated how peer and self-evaluation of an OSCE provides a comparable alternative to traditional faculty grading, especially in higher performing students. However, due to differences in overall grades, this strategy should be reserved for low-stakes assessments and basic skill evaluations.

Exploring in-person self-led debriefings for groups of learners in simulation-based education: an integrative review

BACKGROUND

Facilitator-led debriefings are well-established for debriefing groups of learners in immersive simulation-based education. However, there has been emerging interest in self-led debriefings whereby individuals or groups of learners conduct a debriefing themselves, without the presence of a facilitator. How and why self-led debriefings influence debriefing outcomes remains undetermined.

RESEARCH AIM

The aim of this study was to explore how and why in-person self-led debriefings influence debriefing outcomes for groups of learners in immersive simulation-based education.

METHODS

An integrative review was conducted, searching seven electronic databases (PubMed, Cochrane, Embase, ERIC, SCOPUS, CINAHL Plus, PsychINFO) for peer-reviewed empirical studies investigating in-person self-led debriefings for groups of learners. Data were extracted, synthesised, and underwent reflexive thematic analysis.

RESULTS

Eighteen empirical studies identified through the search strategy were included in this review. There was significant heterogeneity in respect to study designs, aims, contexts, debriefing formats, learner characteristics, and data collection instruments. The synthesised findings of this review suggest that, across a range of debriefing outcome measures, in-person self-led debriefings for groups of learners following immersive simulation-based education are preferable to conducting no debriefing at all. In certain cultural and professional contexts, such as postgraduate learners and those with previous debriefing experience, self-led debriefings can support effective learning and may provide equivalent educational outcomes to facilitator-led debriefings or self-led and facilitator-led combination strategies. Furthermore, there is some evidence to suggest that self-led and facilitator-led combination approaches may optimise participant learning, with this approach warranting further research. Reflexive thematic analysis of the data revealed four themes, promoting self-reflective practice, experience and background of learners, challenges of conducting self-led debriefings and facilitation and leadership. Similar to facilitator-led debriefings, promoting self-reflective practice within groups of learners is fundamental to how and why self-led debriefings influence debriefing outcomes.

CONCLUSIONS

In circumstances where simulation resources for facilitator-led debriefings are limited, self-led debriefings can provide an alternative opportunity to safeguard effective learning. However, their true value within the scope of immersive simulation-based education may lie as an adjunctive method alongside facilitator-led debriefings. Further research is needed to explore how to best enable the process of reflective practice within self-led debriefings to understand how, and in which contexts, self-led debriefings are best employed and thus maximise their potential use.

Peer-assessment ability of trainees in clinical restorative dentistry: can it be fostered?

Objectives

The study was conducted to investigate whether peer-assessment among dental students at the clinical stage can be fostered and become closer to that of experienced faculty assessors.

Methods

A prospective pilot study was conducted in 2021 at the Faculty of Dentistry, Damascus University. Sixteen 5th year clinical students volunteered to participate in the study. A modified version of the validated Peer Direct Observation of Procedural Skills (Peer-DOPS) assessment form was used together with a grading rubric. Participants undertook peer-assessment on their colleagues across three encounters. The difference between peers and faculty assessment was the main variable.

Results

The mean difference between peers and faculty assessment decreased after each encounter with a significant difference and a medium effect size between the first and third encounters (p = 0.016, d = 0.67). Peer-assessment was significantly higher than faculty, however, the overestimation declined with each encounter reducing the difference between peer- and faculty assessment. Peers’ perception of the educational benefit of conducting assessment was overwhelmingly positive, reporting improvements in their own performance.

Conclusion

This pilot study provides preliminary evidence that dental students assessment ability of their peers can be fostered and become closer to that of experienced faculty assessment with practice and assessment-specific instruction.

Dental students' experience of conscious sedation: A qualitative review of student reflections

INTRODUCTION

The UK General Dental Council (GDC) requires dental graduates to competently identify, manage and refer patients with dental pain and anxiety. This study aimed to explore sedation training experience quantitatively and qualitatively through individual reflective logs.

MATERIALS & METHODS

A single-centred mixed-methods evaluation of teaching, within a UK university conscious sedation department. Fourth-year dental students undertook lectures and supervised clinical sessions following the undergraduate curriculum. Patient attendance patterns, individual experience and group experience were analysed descriptively. Reflective log sheets were analysed by Thematic Framework Analysis.

RESULTS

Seventy-two students participated. Of 153 booked patients, 79 (51.6%) attended of which 74 (48.4%) were treated by undergraduates. The mean performed inhalation sedation and intravenous sedation cases per student were 1 and 0.8, respectively. Three students (4%) assisted only. Group experience varied. Three themes arose from reflections: consolidation of theory and learning; confidence through experience; and responding to challenges.

DISCUSSION

Whilst experience quotas were not met, GDC requirements for increasing students' knowledge and confidence within CS were met. Practical experience enhanced learning. "Hands-on" experience was most valuable for self-reported confidence but learning by proxy also aided development. Students recognised challenges, but not the implications for themselves or their career. Strategies to reduce barriers to experience require research.

CONCLUSION

Whilst variable, all students were provided learning opportunities. Physical experience gave the greatest confidence. Opportunities afforded by the undergraduate curriculum allowed students to learn and develop through consolidation of theory, response to challenges and ultimately the gaining of confidence.

Implementing Standardized Patient Caregivers to Practice Difficult Conversations in a Pediatric Dentistry Course

INTRODUCTION

Standardized patient (SP) methodology has been used in health professional education to help students develop communication, deeper diagnostic reasoning, and critical thinking skills. Few examples demonstrate the use of SPs to practice difficult conversations with pediatric caregivers in the pediatric dentistry literature. The objective of this educational activity was to describe the implementation of three SPs in a pediatric dentistry course for second-year dental students.

METHODS

We developed three SP encounters covering interactions with caregivers of an infant with severe early childhood caries, an adolescent on the path to gender affirmation, and a child with autism and dental caries whose caregiver was resistant to fluoride- and silver-containing dental materials. We describe the case design process, rubric construction and calibration, student debriefing, and pandemic modifications. We evaluated the effectiveness of the implementation by thematic analysis of student reflections following each encounter using a qualitative descriptive framework.

RESULTS

Eighty-three students completed each encounter. Qualitative analysis showed that students preferred a more realistic encounter by having a child or other distraction present. Students relied on different elements of motivational interviewing depending on the objective of each encounter and the age of the patient. Overall, the SP encounters were well received by students and faculty as an alternative or supplement to traditional student evaluation methods.

DISCUSSION

We noted a number of lessons learned about implementing SP methodology in pediatric dental education. With these experiences now in place, future evaluations should measure student performance in the SP encounters against student performance during clinical care.

Development of an Online Asynchronous Clinical Learning Resource ("Ask the Expert") in Dental Education to Promote Personalized Learning

This article describes the development and testing of an online asynchronous clinical learning resource named "Ask the Expert" to enhance clinical learning in dentistry. After the resource development, dental students from years 3 and 4 were randomly allocated to two groups (Group A-"Ask the Expert" and L-"lecturer-led"). All the students attempted a pre-test related to replacement of teeth in the anterior aesthetic zone. Group A (33 students) underwent an online case-based learning session of 60 minutes' duration without a facilitator, while Group L (27 students) concurrently underwent a case-based learning session of 60 minutes' duration with a lecturer facilitating the session. An immediate post-test was conducted followed by a retention test after one week. Student feedback was obtained. There was a significant increase in the test scores (maximum score 10) for both groups when comparing the pre-test (Group A-5.61 ± 1.34, Group L-5.22 ± 1.57) and immediate post-test scores (Group A-7.42 ± 1.34, Group L-8.04 ± 1.22; paired t-test, p < 0.001). However, no significant difference was observed in the test scores when comparing Group A to Group L for both the immediate post-test as well as the retention test (Group A-5.36 ± 1.29, Group L-5.33 ± 1.39 (independent sample t-test, p > 0.05). To conclude, adequately structured online asynchronous learning resources are comparable in their effectiveness to online synchronous learning in the undergraduate dental curriculum.

Effects of active engagement and spaced retrieval practice on knowledge and application of a self-assessment rubric

PURPOSE/OBJECTIVES

This study examined effects of active engagement (ENG) and spaced retrieval practice (SRP) on D1 dental students' self-assessment knowledge acquisition.

METHODS

D1 dental students (N = 120) were randomly assigned to one of four conditions in a 2 (ENG or no ENG) X 2 (SRP or no SRP) design. Students were tested on their knowledge of a self-assessment grading rubric (on a 30-item multiple-choice exam) and their ability to apply the rubric (on a simulated dentoform assessment).

RESULTS

There were significant main effects of both ENG and SRP on both outcome measures. Both ENG and SRP increased students' knowledge of, and ability to apply, the self-assessment rubric. Effects of ENG were larger than those of SRP. There was also a significant ENG X SRP interaction on the knowledge measure. The effect of SRP was larger in the no-ENG conditions than the ENG conditions, which may reflect a ceiling effect in the ENG conditions.

CONCLUSIONS

ENG and SRP have the potential to substantially enhance D1 dental students' learning outcomes. These results should motivate dental educators to reject outdated instructional techniques and thereby unleash students' full learning potential. Additional research is needed with a wider range of dental students and learner topics.

Self-confidence as a predictor of senior dental student academic success

PURPOSE

Noncognitive indicators, such as personality type and self-confidence, have been used as predictors of achievement. Accordingly, we investigated whether self-confidence in clinical skills predicts academic success among predoctoral senior dental students.

METHODS

Two consecutive cohorts (N = 336) of predoctoral senior dental students were invited to rate their self-confidence in clinical procedures twice: at the beginning and at the end of the academic year. Academic performance in clinical assessments was retrieved from scholastic records. Multiple linear regression models were constructed to predict clinical assessment grades using self-confidence scores as a predictor after controlling for possible confounders.

RESULTS

Self-confidence scores in clinical procedures were positively correlated with clinical and academic performance. Self-confidence in clinical procedures and grade point averages were consistent predictors of student performance. Additionally, self-confidence had improved at the end of the senior year (p = 0.003).

CONCLUSIONS

Self-confidence in clinical procedures was a predictor of future clinical success. Therefore, fostering and nurturing the confidence of students and integrating self-confidence assessment into the curricula at relevant timepoints should be a goal in dental education. Moreover, self-assessment of confidence in clinical skills could be used to guide students and graduates in developing their own personal development plan.

Student Potential for Self-assessment in a Clinical Dentistry Practical Training Course on Communication Skills

This study aimed to investigate student potential for self-assessment in a clinical dentistry practical training course focused on communication skills. Participants were 124 fourth-year students (70 males, 54 females; all Japanese) in 2017 and 2018 at the Nippon Dental University, School of Life Dentistry at Niigata. Participating students belonged to different cohorts in 2017 and 2018. Participants were asked to complete a self-evaluation sheet at the end of each unit of the course. Their self-evaluation scores and the faculty evaluation scores for each student for Units 1-1, 1-2, and 1-3 were statistically analyzed. The results showed that females tended to rate themselves significantly higher than males. Furthermore, there were significant differences in evaluation scores between students and faculty for nine of 11 evaluation items for male students and 10 of 11 items for female students in Unit 1-3. Faculty expectations increased from Unit 1-1 to Unit 1-3, although students were satisfied with their performance and had a sense of achievement. However, students' actual performance was below faculty expectations, suggesting faculty evaluations were stricter than students' self-evaluation. Self-assessment may enhance students' ability for self-directed learning and may also inform how faculty can effectively educate dental students. Dental educators should support students to increase their levels of self-efficacy, which will enhance their self-evaluation skills.

Pursue today and assess tomorrow - how students' subjective perceptions influence their preference for self- and peer assessments

BACKGROUND

Alternative assessments engage students in the assessment process to improve both short- and long-term outcomes by developing their judgments and responsibility about their own learning, and that of their peers. In this study, we investigated students' perception towards self- and peer-assessment, their objectivity and impact on students' learning.

METHODS

The study was conducted at the Medical University of Vienna. Attitudes of second year undergraduate medical students towards self- and peer-assessment, and their objectivity, appropriateness, and the impact of these assessments on students' learning activities, was inquired using a self-developed questionnaire.

RESULTS

Four hundred twenty-three students participated in this study. Self-assessment was found more appropriate method to assess students' knowledge. Most of students agreed that peer-assessment is not objective (M = - 0.07). Majority of students evaluated that peer assessment has no or little impact on their active and passive learning (M = - 0.23, - 0.35), on the other hand self-assessment was reported as a helpful tool for gaining long-term knowledge (M = 0.13) and following the content of courses (M = 0.16).

CONCLUSION

Based on our results, students' perspective on peer assessment were negative, on the whole, students had positive attitudes towards self-assessment and negative attitudes towards peer-assessment. This study also determined that self-assessment leads to the promotion of students' learning.

Assessment of student performance on a mock new prescription counseling session and an objective structured clinical examination across five years

INTRODUCTION

New prescription counseling (NPC) provides vital information to patients regarding newly prescribed medications to improve treatment outcomes. This evaluation's goals were to describe the implementation of teaching and assessment methods incorporated into an NPC module across two courses at Auburn University and evaluate student performance on assessments of NPC skills over five years.

METHODS

Assessments included self and peer evaluations of a recorded mock NPC session, NPC objective structured clinical examination (OSCE), and overall course grade. Scores were analyzed over a five-year period (2012 to 2016) to evaluate mean student performance in each of four domains (gathering information, communication, management strategies, and monitoring and follow-up) and overall for each assessment. Pearson's correlation coefficients between these scores were calculated by combining all five years of data.

RESULTS

Seven-hundred thirty-three students were included in the analysis. No trends in mean domain and total assessment scores were noted across years. Self and peer evaluation domain and total scores were significantly, though moderately, correlated across assessments (r = 0.43-0.51), except in the communication domain (r = 0.12). Overall, NPC OSCE total scores were not correlated with NPC recording self or peer evaluation total scores (r = 0.06 and r = 0.11, respectively). Assessment total scores were all moderately correlated with overall course grades.

CONCLUSIONS

Utilization of a scaffolding approach with class discussion, demonstration, role-play, self and peer evaluation, and OSCEs, is an effective means of building student competence in patient counseling for the NPC context.

Assessing Advanced Communication Skills via Objective Structured Clinical Examination: A Comparison of Faculty Versus Self, Peer, and Standardized Patient Assessors

Construct: The construct addressed in this study is assessment of advanced communication skills among senior medical students. Background: The question of who should assess participants during objective structured clinical examinations (OSCEs) has been debated, and options discussed in the literature have included peer, self, standardized patient, and faculty assessment models. What is not known is whether same-level peer assisted learning can be utilized for formative assessment of advanced communication skills when no faculty, standardized patients, or other trained assessors are involved in providing feedback. If successful, such an educational model would optimize resource utilization and broaden the scope of topics that could be covered in formative OSCEs. Approach: The investigators developed a 4-station formative OSCE focused on advanced communication skills for senior medical students, and evaluated the concordance of assessment done by same-level peers, self, standardized patients, and faculty for 45 students. After each station, examinees completed a self-assessment checklist and received checklist-based assessment and verbal feedback from same-level peers only. Standardized patients completed checklist-based assessments outside the room, and faculty did so after the OSCE via video review; neither group provided direct feedback to examinees. The investigators assessed inter-rater agreement and mean difference scores on the checklists using faculty score as the gold standard. Findings: There was fair to good overall agreement among self, same-level peer, standardized patient, and faculty-assessment of advanced communication skills. Relative to faculty, peer and standardized patient assessors overestimated advanced communication skills, while self-assessments underestimated skills. Conclusions: Self and same-level peer-assessment may be a viable alternative to faculty assessment for a formative OSCE on advanced communication skills for senior medical students.

Impact of simulated patients on students' self-assessment of competency in practice of geriatric dentistry

PURPOSE

The evaluation of the effectiveness of simulation training in dentistry has previously been focused primarily on psychomotor hand skills. This study explored the impact of simulated patient (SP) encounters in simulation on students' self-assessment of competency in their clinical and communication abilities with geriatric patients.

METHODS

Students from 2 cohorts were recruited for this study. Cohort 1 (n = 30) participated in the standard curriculum with no simulation training and served as the control group. Cohort 2 (n = 34) participated in a SP experience, simulating the initial stages of a care visit for 2 nursing home patients. Students' perceptions of competency to perform these clinical and communication tasks were assessed. A group debriefing session was held 5 weeks post-simulation where Cohort 2 completed a student feedback form.

RESULTS

A statistically significant change (P < 0.00001) was noted for both cohorts in their self-reported competence to perform clinical tasks following exposure to an independent clinical experience. In addition to this gain, individuals in Cohort 2 demonstrated improvements following simulation and expressed different responses of impact to questions related to treatment, pharmacology, and managing a complex medical history.

CONCLUSIONS

This study suggests that simulation of patient interactions using SPs can strengthen students' self-assessment of competency in their abilities, leading to more genuine interactions with actual patients. These findings will help inform the design of future SP encounters as a component of an evolving humanistic curriculum.

Graduating dentists' perceptions about their professional competence in Finland and Lithuania

AIMS

Efforts to harmonise dental education in Europe have been put into action by the Association for Dental Education in Europe (ADEE). The aim of the study was to explore graduating dentists' perceptions about their professional readiness for clinical work in Finland and Lithuania.

MATERIALS AND METHODS

The survey targeted fifth-year dental students at the University of Oulu and the University of Turku in Finland, and at the Lithuanian University of Health Sciences, Kaunas, Lithuania (Lithuanian and international students) in 2016-2017. The competences were evaluated in the questionnaire in 21 dental procedures based on ADEE competences by options: "I'd manage well," "I'd need more training" or "It would not quite succeed." The option "I'd manage well" was chosen in the analyses.

RESULTS

Students felt that they were most competent in producing and maintaining accurate patient records (91.9%), implementing sterilisation and hygiene in dental practice (91.3%) and working with other members of dental team and health profession (90.0%). The largest differences between Finnish and Lithuanian students were in designing and adjusting occlusal splints (87.0% vs. 14.3%) and in undertaking subgingival scaling (95.7% vs. 57.1%). The biggest differences between Lithuanian and international students were in identifying (and treating) abnormal and anxiety-related patient (73.1% vs. 25.5%) and implementing tobacco cessation (65.3% vs. 31.9%).

CONCLUSIONS

The graduating dental students in Finland and Lithuania manage well in most of the clinical procedures based on the ADEE competences. The students were most confident when dealing with tasks that are common in dental practice.

Comparison of Dental Students' Self-Evaluation and Faculty Evaluation of Communication Skills During a Standardized Patient Exercise

The aims of this retrospective study, conducted in 2017, were to explore dental students' perceptions of their first standardized patient encounters and to assess the relationship between students' self-evaluation and faculty members' evaluation of students' communication skills in those encounters. Data from a simulation training laboratory at one U.S. dental school were obtained for all 46 second-year students, who had a standardized patient communication learning session. The students had completed self-evaluations before and after a debriefing with a clinical psychology and/or social work faculty member and three or four student peers. The faculty members had also completed evaluations of the students' communications skills. The results showed that the students found the standardized patient encounter helpful. The students rated "making an introduction" most positively. Students had a weighted Kappa agreement of 0.22 (p=0.024) with the faculty evaluations on their post-debriefing evaluation of overall communication skills, which was within the 0.21-0.40 range of fair agreement. This study found that, during their first standardized patient simulation exercise, the second-year students rated their overall communication skills in fair agreement with the faculty member after debriefing.

Quality assurance processes for standardized patient programs

OUR PROBLEM

As the pharmacy profession evolves to include non-dispensing services and collaborative care, greater emphasis is placed on communication skills building through standardized patient programs. Best practices for assuring the quality of standardized patient (SP) programs, however, remains unclear. The objective of this manuscript is to summarize quality assurance processes for standardized patient programs from health professions education literature.

METHODOLOGICAL LITERATURE REVIEW

A search of PubMed and Scopus between 2011 and 2016 was conducted and 22 articles were retained for thematic analysis. Articles were screened for relevance to quality assurance.

OUR RECOMMENDATIONS AND THEIR APPLICATIONS

The thematic analysis revealed four themes: (1) enhanced SP training programs, (2) structured feedback to students, (3) statistical measurements to ensure inter-rater reliability, and (4) observation and evaluation of the SP to improve SP performance. Specific methods to assure the quality of an SP program were identified, including training program content and feedback techniques.

POTENTIAL IMPACT

Although SP programs varied widely in their implementation, there were several common strategies used to evaluate the consistency of performance, effectiveness of feedback to students, and reliability of grading. Additional research is necessary to establish standards for SP programs across professional healthcare disciplines.

Peer assessment in the objective structured clinical examination: A scoping review

BACKGROUND

The objective structured clinical examination (OSCE), originally designed with experts assessing trainees' competence, is more frequently employed with an element of peer assessment and feedback. Although peer assessment in higher education has been studied, its role in OSCEs has not reviewed.

AIMS

The aim of this study is to conduct a scoping review and explore the role of peer assessment and feedback in the OSCE.

METHODS

Electronic database and hand searching yielded 507 articles. Twenty-one full records were screened, of which 13 were included in the review. Two independent reviewers completed each step of the review.

RESULTS

Peer-based OSCEs are used to assess students' accuracy in assessing OSCE performance and to promote learning. Peer examiners (PE) tend to award better global ratings and variable checklist ratings compared to faculty and provide high-quality feedback. Participating in these OSCEs is perceived as beneficial for learning.

CONCLUSIONS

Peer assessment and feedback can be used to gauge PE reliability and promote learning. Teachers using these OSCEs must use methodology which fits their purpose. Competency-based education calls for diversification of assessment practices and asks how assessment impacts learning; the peer-based OSCE responds to these demands and will become an important practice in health professions education.