Five ways to get a grip on grouped self-assessments of competence for program evaluation

Examining the accuracy of residents' self-assessments and faculty assessment behaviours in anesthesiology

BACKGROUND

Residents' accurate self-assessment and clinical judgment are essential for optimizing their clinical skills development. Evidence from the medical literature suggests that residents generally do poorly at self-assessing their performance, often due to factors relating to learners' personal backgrounds, cultures, the specific contexts of the learning environment and rater bias or inaccuracies. We evaluated the accuracy of anesthesiology residents' self-assessed Global Entrustment scores and determined whether differences between faculty and resident scores varied by resident seniority, faculty leniency, and/or year of assessment.

METHODS

We employed variance components modeling techniques and analyzed 329 pairs of faculty and self-assessed entrustment scores among 43 faculty assessors and 15 residents. Using faculty scores as the gold standard, we compared faculty scores with residents' scores (x_i(faculty)-x_i(resident)), and determined residents' accuracy, including over- and under-confidence.

RESULTS

The results indicate that residents were respectively over- and under-confident in 10.9% and 54.4% of the assessments but more consistent in their individual self-assessments (rho = 0.70) than faculty assessors. Faculty scores were significantly higher (α = 0.396; z = 4.39; p < 0.001) than residents' self-assessed scores. Being a lenient/dovish (β = 0.121, z = 3.16, p < 0.01) and a neutral (β = 0.137, z = 3.57, p < 0.001) faculty assessor predicted a higher likelihood of resident under-confidence. Senior residents were significantly less likely to be under-confident compared to junior residents (β = -0.182, z =-2.45, p < 0.05). The accuracy of self-assessments did not significantly vary during the two years of the study period.

CONCLUSIONS

The majority of residents' self-assessments were inaccurate. Our findings may help identify the sources of such inaccuracies.

Medical students' personal experiences, religion, and spirituality explain their (dis)comfort with a patient's religious needs

BACKGROUND

Physicians often avoid discussing patients' religious and spiritual concerns, even though most patients (i.e., 50-94%) want integrated care. To address this gap, medical students interviewed a Standardized Patient (SP) who was upset because the daughter did not confront her fiancée about converting to Orthodox Judaism. Students reflected on how their own religion and spirituality affected engaging with their patient.

METHODS

With a 97% response rate, 231 first-year medical students responded to open-ended questions about their patient encounter. For this quantitative content analysis, we used inductive reasoning, identifying three themes: (1) impact of students' own religion on their comfort, (2) change in comfort, and (3) their learning. We used deductive reasoning to compare qualitative results from half of the students who began the curriculum with a questionnaire about their own spirituality with the other students completing afterwards.

RESULTS

Most students said being religious positively influenced their comfort, whether they were also Orthodox Jewish or from a different religion. Among uncomfortable students (6.5%), some attributed this to not being religious. Some students (4.8%) grew more comfortable discussing the religious issue, and 18.2% became uncomfortable due to lacking knowledge of Orthodox Judaism and the awkwardness of the topic. Students who had completed the questionnaire beforehand gave more comments about connecting with their patients than students who completed the questionnaire afterwards (X2=11.047, p<.001).

CONCLUSIONS

Students' own religion influenced their comfort with discussing religious concerns, with some feeling more connected and others becoming uncomfortable. This finding helps inform medical educators about teaching mind-body-spirit care.Résumé.