Utility of factor analysis in optimization of resident assessment and faculty evaluation

Academic Faculty Demonstrate Weak Agreement in Evaluating Orthopaedic Surgery Residents

Background

As objective metrics fade, subjective elements of orthopaedic surgery applicants carry increasing importance during recruitment. Academic orthopaedic surgeons believe they can select for high-performing orthopaedic residents. However, can they agree? The purpose of this study was (1) to analyze an academic orthopaedic surgery department to determine whether they agree on which residents performed best during residency and; (2) to correlate preresidency and intraresidency factors with postresidency evaluations of resident performance.

Methods

With Institutional Review Board [IRB] approval, an orthopaedic surgery department completed surveys to evaluate residency performance for 10 years of graduated residents (2012-2022). Faculty determined (1) Post-Residency Class Rank (PRCR)-ranked from the highest (1) to lowest performing resident (5) relative to their class based on faculty perspective of performance-and (2) Rank List Score (RLS)-ranked based off the 5-point AOA SLOR used during recruitment interviews. RLS assessed how likely the department would have graduates in the program again based on residency performance. Free marginal Cohen's kappa statistics assessed faculty inter-rater agreement. Preresidency metrics (United States Medical Licensing Exam [USMLE] 1 and 2 scores, research publications, etc) were correlated with Orthopaedic In-Training Exam (OITE) scores, research productivity, American Board of Orthopaedic Surgery (ABOS) scores, and faculty-derived rankings. Linear regressions with forward variable entry (p < 0.05) were used to determine factors associated with excellent resident performance.

Results

Eighteen of 25 faculty members (72%) provided survey responses evaluating 46 residents. Faculty agreed 37% and 38% of the time for PRCR (kappa 0.26) and RLS (kappa 0.23), respectively. Step 2 score was the only preresidency factor significantly associated with PRCR (p = 0.03, r2 = 0.15) and RLS (p = 0.02, r2 = 0.3). The only intraresidency factor significantly correlated with PRCR (p = 0.002, r2 = 0.50) and RLS (p = 0.01, r2 = 0.39) was PGY-4 OITE score.

Conclusions

An academic orthopaedic surgery department is able to come to a consensus on evaluations of residency performance relative to peers in the same year of training (PRCR) and an objective standard (RLS). Step 2 and Post-Graduate Year (PGY)-4 OITE scores were the only preresidency and intraresidency factors with significant association to higher postresidency, faculty-derived performance scores.

Level of Evidence

III.

Challenges in measuring ACGME competencies: considerations for milestones

Background

Measuring milestones, competencies, and sub-competencies as residents progress through a training program is an essential strategy in Accreditation Council for Graduate Medical Education (ACGME)’s attempts to ensure graduates meet expected professional standards. Previous studies have found, however, that physicians make global ratings often by using a single criterion.

Methods

We use advanced statistical analysis to extend these studies by examining the validity of ACGME International competency measures for an international setting, across emergency medicine (EM) and neurology, and across evaluators. Confirmatory factor analysis (CFA) models were fitted to both EM and neurology data. A single-factor CFA was hypothesized to fit each dataset. This model was modified based on model fit indices. Differences in how different EM physicians perceived the core competencies were tested using a series of measurement invariance tests.

Results

Extremely high alpha reliability coefficients, factor coefficients (> .93), and item correlations indicated multicollinearity, that is, most items being evaluated could essentially replace the underlying construct itself. This was true for both EM and neurology data, as well as all six EM faculty.

Conclusions

Evaluation forms measuring the six core ACGME competencies did not possess adequate validity. Severe multicollinearity exists for the six competencies in this study. ACGME is introducing milestones with 24 sub-competencies. Attempting to measure these as discrete elements, without recognizing the inherent weaknesses in the tools used will likely serve to exacerbate an already flawed strategy. Physicians likely use their “gut feelings” to judge a resident’s overall performance. A better process could be conceived in which this subjectivity is acknowledged, contributing to more meaningful evaluation and feedback.