Faculty Evaluations Correlate Poorly with Medical Student Examination Performance in a Fourth-Year Emergency Medicine Clerkship

Medical Faculty and Medical Student Opinions on the Utility of Questions to Teach and Evaluate in the Clinical Environment

Objectives

We sought to report medical student and faculty perceptions of the purpose and utility of questions on clinical rounds.

Methods

We developed and administered a survey to third and fourth-year medical students and teaching physicians. The survey elicited attitudes about using questions to teach on rounds in both benign and malignant learning environments.

Results

Ninety-seven percent of faculty and 85% of students predicted they will use questions to teach. Nine percent of students described learning-impairing stress during benign bedside teaching. Fifty-nine percent of faculty felt questions were mostly for teaching; 74% of students felt questions were mostly for evaluation. Forty-six percent of students felt questions underestimated their knowledge. Students felt questions were more effective for classroom teaching than bedside teaching. Faculty and students agreed that a malignant environment detrimentally affected learning and performance.

Conclusions

Students and faculty supported the use of questions to teach and evaluate, especially in benign teaching environments. Many students described stress severe enough to affect their learning and performance, even when questioned in benign teaching environments. Faculty underestimated the degree to which students experience stress-related learning impairment and the degree to which students see questions as evaluation rather than teaching. Nearly half of students felt that questions underestimated their own knowledge. Students feel more stress and less learning when questioned with a patient present. Faculty must realize that even in the best learning environment some students experience stress-impaired learning and performance, perhaps because of the conflict between learning and evaluation.

The Effect of Pass/Fail Exam Grading on Exam Performance in a Pediatric Clerkship

OBJECTIVE

Clerkship grades are a component of determining a residency candidate's competitiveness. In 2017, the University of Minnesota Medical School's pediatric clerkship transitioned its standardized multiple-choice exam, the Aquifer Pediatrics Examination, to pass/fail with eligibility for honors being determined by clinical performance, not exam performance. We assessed the effect this change had on Aquifer exam performance and evaluated for correlation between Aquifer exam performance and clinical evaluation scores in order to gather insight into the validity of each type of assessment with respect to one another.

METHODS

We analyzed de-identified data from 750 medical students between the academic years of 2016 to 2017 and 2019 to 2020. Individual Aquifer exam scores were compared to individual clinical performance scores. Differences in exam performance before and after the transition to pass/fail were investigated with a two-sample t-test and Cohen's d for effect size.

RESULTS

No correlation was found between Aquifer exam scores and clinical performance scores. The mean Aquifer exam score prior to the transition to pass/fail was 80.02 ± 7.51 while the mean after the exam was made pass/fail was 77.8 ± 7.42. This difference was statistically significant (P < .001) with a Cohen's d (effect size) of 0.297.

CONCLUSIONS

A lack of correlation between the Aquifer exam scores and clinical performance scores was found. There was a small yet statistically significant decrease in Aquifer exam scores after the change to pass/fail; it is not clear if this represents a meaningful decrease in learning by students.

Impact of Students' Scheduling Choice on Clerkship Examination Score Performance in a Time-Varying Competency-Based Curriculum

PURPOSE

Standardized end-of-clerkship examinations typically occur on the last day of the clerkship. However, recent trends toward time-varying competency-based medical education have offered students more test scheduling flexibility, creating an opportunity to study the impact of student-selected examination timing.

METHOD

Starting with the graduating class of 2018, students took the required standardized end-of-core clerkship examinations at any available time they chose during their clinical years. Before this change, these examinations were administered to all students on the last day of the clerkship. Students' examination dates relative to clerkship completion were analyzed between 2017 and 2020 (inclusive of before and after flexible exam timing) to assess the impact that student-selected exam timing had on test performance on National Board of Medical Examiners shelf clinical science examinations for required core clerkships.

RESULTS

Data on 146 medical students in 2017 (fixed exam timing) and 466 medical students between 2018 and 2020 (flexible exam timing) were included. Among students offered flexible exam timing, between 2.7% (internal medicine) and 14.6% (psychiatry) took their exam before actually taking clerkship, while between 22.7% (psychiatry) and 40.0% (surgery) took their exam more than 90 days after the clerkship ended. Exam scores were statistically higher for those who took the exam at a time of their choosing compared with those who were required to take it at the end of individual rotations and when the exam scores were combined (fixed exam timing mean = 73.9, standard deviation [SD] = 7.8; flexible exam timing mean = 77.4, SD = 6.0, P < .001). The percent of students with passing scores was statistically higher in internal medicine, pediatrics, and psychiatry.

CONCLUSIONS

Self-selection of shelf exam timing appears to increase shelf exam scores. As more medical schools transition to competency-based medical education, providing scheduling flexibility appears not to negatively affect student achievement.

Exploring the Impact of Pre-course High-Fidelity Simulation on Professional Socialization of Medical Students in Emergency Medicine Internship Rotation—A Qualitative Approach

Background

Medical students in block clerkships constantly adapt to new environments and learn to interact with new people as they rotate between specialties. This frequent change potentially limited interns' opportunities for participation in real clinical practice. The aims of this study were to explore interns' conceptualization of their learning opportunities and experiences in the workplace during an emergency medicine (EM) block internship. In addition, the study also explored how participating in the pre-rotation high-fidelity simulation (HFS) orientation influenced interns' perception of their transition, participation and learning experiences in the real EM setting.

Methods

We implemented a newly developed pre-EM rotation orientation curriculum for interns. This orientation took place on the first day of the 2-week EM internship rotation. Two focus group discussions were held after each simulation training, one immediately after simulation to understand the students' perception and the educational impact of this activity, the other at the end of EM rotation to explore and compare their roles and perception in both simulation activity and the real clinical practice. A total of 151 seventh-year medical students enrolled in the pre-course HFS and post-hoc focus group discussions between 2017 and 2019. We applied thematic analysis to systemically identify, examine, and construct themes.

Results

Four major themes were constructed from the data; 1. Challenges in finding authentic learning experiences within the context of emergency medicine; 2. Effectiveness of the pre-course HFS 3. Limitations of EM internship rotation curriculum and pre-course simulation. 4. Suggestions for EM block-internship curriculum reforms. Our study's key findings indicate that pre-rotation orientation HFS activity, which offered a psychologically safe space for students to explore facets of EM and gain a contextualized understanding of the emergency work culture and environment, was essential for enhancing students' ability to identify and maximize practice affordances in real workplace.

Conclusion

Simulation, facilitates interns' negotiation of legitimate peripheral participation opportunities as they transition into the EM community of practice during their block internship rotation; putting students at the center of the learning process.

Effect of student-directed solicitation of evaluation forms on the timeliness of completion by preceptors in the United States

PURPOSE

Summative evaluation forms assessing a student's clinical performance are often completed at the end of a clinical experience from a faculty preceptor. In our institution, despite the use of an electronic system, completion timeliness has been suboptimal potentially limiting our ability to monitor student progress. The aim of the present study was to determine if a student-centered approach to summative evaluation form collection at the end of a pediatrics clinical experience will enhance timeliness of completion for third year medical students at Pennsylvania State College of Medicine.

METHODS

This was a pre- and post-intervention educational quality improvement project focused on 156 (82 pre-intervention, 74 post-intervention) third year medical students at Pennsylvania State College of Medicinecompleting their four-week pediatric clerkship. Utilizing Research Electronic Data Capture (REDCap) informatics support, student-directed evaluation form solicitation was encouraged. Wilcoxon rank sum test was applied to compare pre-intervention (May 1, 2017 to March 2, 2018) and post-intervention (April 2, 2018 to December 21, 2018) percentages of forms completed before rotation midpoint.

RESULTS

Seven hundred and forty evaluation forms were submitted during the pre-intervention phase and five-hundred and seventeen were submitted during the post-intervention phase. Form completion percentage before rotation midpoint increased after implementation of student-directed solicitation (9.6% versus 39.7%, P<0.05).

CONCLUSION

Our clerkship relies on subjective summative evaluations to track student progress, employ improvement strategies, and determine criteria for advancement; however, our preceptors struggled with timely submission. Allowing students to direct the solicitation of evaluation forms enhanced the timeliness of completion at our institution and should be considered in clerkships with similar difficulties.