Validity of faculty and resident global assessment of medical students' clinical knowledge during their pediatrics clerkship

Effect of student-directed solicitation of evaluation forms on the timeliness of completion by preceptors in the United States

PURPOSE

Summative evaluation forms assessing a student's clinical performance are often completed at the end of a clinical experience from a faculty preceptor. In our institution, despite the use of an electronic system, completion timeliness has been suboptimal potentially limiting our ability to monitor student progress. The aim of the present study was to determine if a student-centered approach to summative evaluation form collection at the end of a pediatrics clinical experience will enhance timeliness of completion for third year medical students at Pennsylvania State College of Medicine.

METHODS

This was a pre- and post-intervention educational quality improvement project focused on 156 (82 pre-intervention, 74 post-intervention) third year medical students at Pennsylvania State College of Medicinecompleting their four-week pediatric clerkship. Utilizing Research Electronic Data Capture (REDCap) informatics support, student-directed evaluation form solicitation was encouraged. Wilcoxon rank sum test was applied to compare pre-intervention (May 1, 2017 to March 2, 2018) and post-intervention (April 2, 2018 to December 21, 2018) percentages of forms completed before rotation midpoint.

RESULTS

Seven hundred and forty evaluation forms were submitted during the pre-intervention phase and five-hundred and seventeen were submitted during the post-intervention phase. Form completion percentage before rotation midpoint increased after implementation of student-directed solicitation (9.6% versus 39.7%, P<0.05).

CONCLUSION

Our clerkship relies on subjective summative evaluations to track student progress, employ improvement strategies, and determine criteria for advancement; however, our preceptors struggled with timely submission. Allowing students to direct the solicitation of evaluation forms enhanced the timeliness of completion at our institution and should be considered in clerkships with similar difficulties.

Using the Clerkship Shelf Exam Score as a Qualification for an Overall Clerkship Grade of Honors: A Valid Practice or Unfair to Students?

Most clerkships require medical students to pass the National Board of Medical Examiners (NBME) subject (shelf) exam to pass the clerkship. Many use the NBME's recommended honors cut score on the shelf exam to determine medical student eligibility for an overall clerkship grade of honors. This use of a conjunctive scoring model for determining honors is inconsistent with the logic behind the intended use of this model for making pass-fail determinations. Further, many clerkships use grading systems that employ both this conjunctive model for honors eligibility and a compensatory scoring model for determining the overall clerkship grade. For students who fall short of the shelf exam honors cut score, such a grading system effectively increases the weighting of shelf exam performance and decreases the clerkship's transparency about the weighting of performance on other assessments toward the clerkship composite score and overall grade. It may also lead to contradictory grading results in which student B has a higher composite score than student A, yet student B receives a lower overall grade. The author illustrates how to calculate a weight for shelf exam performance that would be fairer to students and would help create a more transparent weighting scheme for the grading system. The author recommends that clerkships restructure their grading systems so that shelf exam honors-eligibility cut scores are not used as conjunctive criteria for determining overall clerkship grades of honors. A reexamination of the NBME's practice of suggesting honors-eligibility cut scores for shelf exams is also recommended.

Optimizing physician skill development for medical students: The four-part assessment

BACKGROUND

Medical student performance has been poorly correlated with residency performance and warrants further investigation. We propose a novel surgical assessment tool to determine correlations with clinical aptitude.

METHODS

Retrospective review of medical student assessments from 2013 to 2015. Faculty rating of student performance was evaluated by: 1) case presentation, 2) problem definition, 3) question response and 4) use of literature and correlated to final exam assessment. A Likert scale interrater reliability was evaluated.

RESULTS

Sixty student presentations were scored (4.8 assessors/presentation). A student's case presentation, problem definition, and question response was correlated with performance (r = 0.49 to 0.61, p ≤ 0.003). Moderate correlations for either question response or use of literature was demonstrated (0.3 and 0.26, p < 0.05).

CONCLUSION

Our four-part assessment tool identified correlations with course and examination grades for medical students. As surgical education evolves, validated performance and reliable testing measures are required.

Faculty Evaluations Correlate Poorly with Medical Student Examination Performance in a Fourth-Year Emergency Medicine Clerkship

BACKGROUND

Clerkship directors routinely evaluate medical students using multiple modalities, including faculty assessment of clinical performance and written examinations. Both forms of evaluation often play a prominent role in final clerkship grade. The degree to which these modalities correlate in an emergency medicine (EM) clerkship is unclear.

OBJECTIVE

We sought to correlate faculty clinical evaluations with medical student performance on a written, standardized EM examination of medical knowledge.

METHODS

This is a retrospective study of fourth-year medical students in a 4-week EM elective at one academic medical center. EM faculty performed end of shift evaluations of students via a blinded online system using a 5-point Likert scale for 8 domains: data acquisition, data interpretation, medical knowledge base, professionalism, patient care and communication, initiative/reliability/dependability, procedural skills, and overall evaluation. All students completed the National EM M4 Examination in EM. Means, medians, and standard deviations for end of shift evaluation scores were calculated, and correlations with examination scores were assessed using a Spearman's rank correlation coefficient.

RESULTS

Thirty-nine medical students with 224 discrete faculty evaluations were included. The median number of evaluations completed per student was 6. The mean score (±SD) on the examination was 78.6% ± 6.1%. The examination score correlated poorly with faculty evaluations across all 8 domains (ρ 0.074-0.316).

CONCLUSION

Faculty evaluations of medical students across multiple domains of competency correlate poorly with written examination performance during an EM clerkship. Educators need to consider the limitations of examination score in assessing students' ability to provide quality patient clinical care.

The Use of Rubrics in the Clinical Evaluation of Podiatric Medical Students: Objectification of the Subjective Experience

BACKGROUND

We assessed the differences in podiatric medical students' clinical professionalism objective scores (CPOSs) by comparing a previous nonrubric evaluation tool with a more recently implemented objective-centered rubric evaluation tool. This type of study has never been performed or reported on in the podiatric medical education literature.

METHODS

We conducted a retrospective analysis of 89 third-year podiatric medical students between academic years 2010-2011 and 2011-2012. A Pearson correlation coefficient analysis was performed to compare CPOSs from the students' first (CPOS1) and second (CPOS2) rotations. A correlation analysis was performed comparing students' grade point averages (GPAs) with each of the individual CPOSs to verify the validity of the rubric evaluation tool.

RESULTS

The Pearson correlation coefficients for the relationship between 2012 CPOS1 and CPOS2 and GPA were r = 0.233 (P ≤ .093) and r = 0.290 (P < .035) and for the relationship between 2013 CPOS1 and CPOS2 and GPA were r = 0.525 (P = .001) and r = 0.730 (P < .001).

CONCLUSIONS

These findings suggest that the use of a rubric in the evaluation of podiatric medical students' CPOSs is correlated with their GPAs, and CPOS2 demonstrated a higher correlation than CPOS1. We believe that implementation of the rubric evaluation tool has increased the accuracy of the evaluation of podiatric medical students with respect to CPOSs.

Evaluation of a simulation-based pediatric clinical skills curriculum for medical students

INTRODUCTION

Simulation-based education is expensive and requires greater resources than traditional methods, yet there is limited evidence to justify such expenditures for medical student education.

METHODS

We describe the implementation and evaluation of a simulation-based curriculum delivered to medical students during a pediatric clerkship. This prospective mixed-methods study evaluated a 5-day long simulation-based clinical skills curriculum (PRE-Clerkship EDucational Exercises [PRECEDE]) at the Johns Hopkins University School of Medicine. Two hundred medical students participated in PRECEDE during a 2-year period and were compared with 236 students who had not. Outcomes were assessed across 3 levels of Kirkpatrick's framework for evaluation. The 4-level model consists of reaction, learning, behavior, and results criteria. Secondary outcomes measured changes in assessment scores across 16 student performance domains during clerkship, changes in performance on the National Board of Medical Examiners subject examination in pediatrics, and student assessments of the curriculum.

RESULTS

Improvements were noted across 3 levels of the Kirkpatrick's model. Student performance evaluations were significantly higher across all 16 evaluation components, with effect sizes ranging from small to medium (Cohen's d, 0.23-0.44). Students scored significantly higher on the National Board of Medical Examiners pediatric shelf examination (80 vs. 77, P<0.001). Ninety-seven percent of the medical students agreed that their skills increased and that the time lost to real clinical experiences was a worthwhile trade-off for this curriculum

CONCLUSIONS

The implementation of a simulation-based curriculum within a pediatrics clerkship resulted in higher knowledge scores and led to improvements in medical student clinical performance during the clerkship.

Setting standards to determine core clerkship grades in pediatrics

OBJECTIVE

One of the greatest challenges for clerkship directors is assigning a final grade and determining the precise point at which a student either passes or fails a clinical clerkship. The process of incorporating both subjective and objective assessment data to provide a final summative grade can be challenging. We describe our experience conducting a standard-setting exercise to set defensible cut points in a 4-tiered grading system in our pediatric clerkship.

METHODS

Using the Hofstee standard-setting approach, 8 faculty members participated in an exercise to establish grade cut points. These faculty members were subsequently surveyed to assess their attitudes toward the standard-setting process as well as their reactions to these newly proposed standards. We applied the new cut points to a historic cohort of 116 Johns Hopkins University School of Medicine students from the academic year 2012-2013 to assess the potential impact on grade distributions.

RESULTS

The resultant grading schema would lead to a significant increase in the number of students receiving a failing grade and a decrease in the number of students receiving a grade of honors in a historical cohort. Faculty reported that the Hofstee method was easy to understand and fair. All faculty members thought that grade inflation presently exists within the pediatric clerkship.

CONCLUSIONS

This study demonstrates that practical standards using the Hofstee method can be set for medical students in a pediatric clerkship in which multiple performance measures are used.