Hawks and Doves: Perceptions and Reality of Faculty Evaluations

OBJECTIVES

Internal medicine clerkship grades are important for residency selection, but inconsistencies between evaluator ratings threaten their ability to accurately represent student performance and perceived fairness. Clerkship grading committees are recommended as best practice, but the mechanisms by which they promote accuracy and fairness are not certain. The ability of a committee to reliably assess and account for grading stringency of individual evaluators has not been previously studied.

METHODS

This is a retrospective analysis of evaluations completed by faculty considered to be stringent, lenient, or neutral graders by members of a grading committee of a single medical college. Faculty evaluations were assessed for differences in ratings on individual skills and recommendations for final grade between perceived stringency categories. Logistic regression was used to determine if actual assigned ratings varied based on perceived faculty's grading stringency category.

RESULTS

"Easy graders" consistently had the highest probability of awarding an above-average rating, and "hard graders" consistently had the lowest probability of awarding an above-average rating, though this finding only reached statistical significance only for 2 of 8 questions on the evaluation form (P = .033 and P = .001). Odds ratios of assigning a higher final suggested grade followed the expected pattern (higher for "easy" and "neutral" compared to "hard," higher for "easy" compared to "neutral") but did not reach statistical significance.

CONCLUSIONS

Perceived differences in faculty grading stringency have basis in reality for clerkship evaluation elements. However, final grades recommended by faculty perceived as "stringent" or "lenient" did not differ. Perceptions of "hawks" and "doves" are not just lore but may not have implications for students' final grades. Continued research to describe the "hawk and dove effect" will be crucial to enable assessment of local grading variation and empower local educational leadership to correct, but not overcorrect, for this effect to maintain fairness in student evaluations.

Use of a structured approach and virtual simulation practice to improve diagnostic reasoning

OBJECTIVES

In 2015, the National Academy of Medicine IOM estimated that 12 million patients were misdiagnosed annually. This suggests that despite prolonged training in medical school and residency there remains a need to improve diagnostic reasoning education. This study evaluates a new approach.

METHODS

A total of 285 medical students were enrolled in this 8 center, IRB approved trial. Students were randomized to receive training in either abdominal pain (AP) or loss of consciousness (LOC). Baseline diagnostic accuracy of the two different symptoms was assessed by completing a multiple-choice question (MCQ) examination and virtual patient encounters. Following a structured educational intervention, including a lecture on the diagnostic approach to that symptom and three virtual patient practice cases, each student was re-assessed.

RESULTS

The change in diagnostic accuracy on virtual patient encounters was compared between (1) baseline and post intervention and (2) post intervention students trained in the prescribed symptom vs. the alternate symptom (controls). The completeness of the student's differential diagnosis was also compared. Comparison of proportions were conducted using χ ²-tests. Mixed-effects regressions were used to examine differences accounting for case and repeated measures. Compared with baseline, both the AP and LOC groups had marked post-intervention improvements in obtaining a correct final diagnosis; a 27% absolute improvement in the AP group (p<0.001) and a 32% absolute improvement in the LOC group (p<0.001). Compared with controls (the groups trained in the alternate symptoms), the rate of correct diagnoses increased by 13% but was not statistically significant (p=0.132). The completeness and efficiency of the differential diagnoses increased by 16% (β=0.37, p<0.001) and 17% respectively (β=0.45, p<0.001).

CONCLUSIONS

The study showed that a virtual patient platform combined with a diagnostic reasoning framework could be used for education and diagnostic assessment and improved correct diagnosis compared with baseline performance in a simulated platform.

Gender Disparity in Evaluation of Internal Medicine Clerkship Performance

IMPORTANCE

Women studying medicine currently equal men in number, but evidence suggests that men and women might not be evaluated equally throughout their education.

OBJECTIVE

To examine whether there are differences associated with gender in either objective or subjective evaluations of medical students in an internal medicine clerkship.

DESIGN, SETTING, AND PARTICIPANTS

This single-center retrospective cohort study evaluated data from 277 third-year medical students completing internal medicine clerkships in the 2017 to 2018 academic year at an academic hospital and its affiliates in Pennsylvania. Data were analyzed from September to November 2020.

EXPOSURE

Gender, presumed based on pronouns used in evaluations.

MAIN OUTCOMES AND MEASURES

Likert scale evaluations of clinical skills, standardized examination scores, and written evaluations were analyzed. Univariate and multivariate linear regression were used to observe trends in measures. Word embeddings were analyzed for narrative evaluations.

RESULTS

Analyses of 277 third-year medical students completing an internal medicine clerkship (140 women [51%] with a mean [SD] age of 25.5 [2.3] years and 137 [49%] presumed men with a mean [SD] age of 25.9 [2.7] years) detected no difference in final grade distribution. However, women outperformed men in 5 of 8 domains of clinical performance, including patient interaction (difference, 0.07 [95% CI, 0.04-0.13]), growth mindset (difference, 0.08 [95% CI, 0.01-0.11]), communication (difference, 0.05 [95% CI, 0-0.12]), compassion (difference, 0.125 [95% CI, 0.03-0.11]), and professionalism (difference, 0.07 [95% CI, 0-0.11]). With no difference in examination scores or subjective knowledge evaluation, there was a positive correlation between these variables for both genders (women: r = 0.35; men: r = 0.26) but different elevations for the line of best fit (P < .001). Multivariate regression analyses revealed associations between final grade and patient interaction (women: coefficient, 6.64 [95% CI, 2.16-11.12]; P = .004; men: coefficient, 7.11 [95% CI, 2.94-11.28]; P < .001), subjective knowledge evaluation (women: coefficient, 6.66 [95% CI, 3.87-9.45]; P < .001; men: coefficient, 5.45 [95% CI, 2.43-8.43]; P < .001), reported time spent with the student (women: coefficient, 5.35 [95% CI, 2.62-8.08]; P < .001; men: coefficient, 3.65 [95% CI, 0.83-6.47]; P = .01), and communication (women: coefficient, 6.32 [95% CI, 3.12-9.51]; P < .001; men: coefficient, 4.21 [95% CI, 0.92-7.49]; P = .01). The model based on the men's data also included growth mindset as a significant variable (coefficient, 4.09 [95% CI, 0.67-7.50]; P = .02). For narrative evaluations, words in context with "he or him" and "she or her" differed, with agentic terms used in descriptions of men and personality descriptors used more often for women.

CONCLUSIONS AND RELEVANCE

Despite no difference in final grade, women scored higher than men on various domains of clinical performance, and performance in these domains was associated with evaluators' suggested final grade. The content of narrative evaluations significantly differed by student gender. This work supports the hypothesis that how students are evaluated in clinical clerkships is associated with gender.

A Primary Care Provider's Guide to Bone Health in Spinal Cord-Related Paralysis

Individuals with spinal cord injury/disorder (SCI/D) are at high risk for developing secondary osteoporosis. Bone loss after neurologic injury is multifactorial and is dependent on the time from and extent of neurologic injury. Most bone loss occurs in the first year after complete motor paralysis, and fractures occur most commonly in the distal femur and proximal tibia (paraplegic fracture). The 2019 International Society for Clinical Densitometry Position Statement in SCI establishes that dual-energy X-ray absorptiometry (DXA) can be used to both diagnose osteoporosis and predict lower extremity fracture risk in individuals with SCI/D. Pharmacologic treatments used in primary osteoporosis have mixed results when used for SCI/D-related osteoporosis. Ambulation, standing, and electrical stimulation may be helpful at increasing bone mineral density (BMD) in individuals with SCI/D but do not necessarily correlate with fracture risk reduction. Clinicians caring for individuals with spinal cord-related paralysis must maintain a high index of suspicion for fragility fractures and consider referral for surgical evaluation and management.

Which Student Characteristics Are Most Important in Determining Clinical Honors in Clerkships? A Teaching Ward Attending Perspective

PURPOSE

To explore faculty perspectives on which characteristics of high-performing clerkship students are most important when determining an honors or top grade designation for clinical performance.

METHOD

In 2016-2017, the authors surveyed faculty (teaching ward attendings) for internal medicine clerkships and 1 pediatrics clerkship in inpatient settings at 5 U.S. academic medical centers. Survey items were framed around competencies, 24 student characteristics, and attitudes toward evaluation. Factor analysis examined constructs defining high-performing students.

RESULTS

Of 516 faculty invited, 319 (62%) responded. The top 5 characteristics as rated by respondents were taking ownership, clinical reasoning, curiosity, dependability, and high ethical standards (in descending order). Twenty-one characteristics fit into 3 factors (Cronbach alpha, 0.81-0.87). Clinical reasoning did not fit into a factor. Factor 1 was the most important (mean rating, 8.7/10 [95% confidence interval (CI), 8.6-8.8]). It included professionalism components (ownership, curiosity, dependability, high ethical standards), presentation and interviewing skills, seeking feedback, and documentation. Factor 2 (mean, 7.9 [95% CI, 7.7-8.0]) included aspects of teamwork and communication, such as positive attitude and comments from others. Factor 3 (mean, 7.6 [95% CI, 7.4-7.7]) addressed systems-based thinking, including patient safety and care transitions.

CONCLUSIONS

Professionalism components, clinical reasoning, and curiosity were among the most important characteristics distinguishing high-performing clerkship students. These may represent behaviors that are highly valued, observable, and relevant to training stage. Improved definition of the characteristics associated with clinical honors would assist students, faculty, and residency program directors when interpreting clinical performance within core clerkships.