Standardized Letter of Recommendation: Can Everyone Be Awesome?

Optimization of a standardized letter of recommendation for faculty who wish to support candidates applying to surgical training programs

Letters of recommendation (LORs) play an important role in applicant selection for graduate medical education programs. LORs may be of increasing importance in the evaluation of applicants given the recent change of the USMLE Step 1 to pass/fail scoring and the relative lack of other objective measures by which to differentiate and stratify applicants. Narrative letters of recommendation (NLORs), although widely used, have certain limitations, namely variability in interpretation, introduction of gender/race bias, and performance inflation. Standardized letters of recommendation (SLOR) have been proposed as a potential corrective to these limitations. We conducted a series of semi-structured interviews and focus groups to gather perspectives from letter writers and readers to inform methods for improving information elicited by SLORs from which we collected and analyzed data using the constant comparative method and a process of iterative coding. We applied our findings to the development of a novel SLOR for use in surgical residency program applications and were subsequently invited to help revise existing SLORs for a surgical post-graduate training program.

Language Differences by Race on Letters of Recommendation for the Pediatric Surgery Match

OBJECTIVE

This study aims to evaluate if there is any significant linguistic difference in LoR based on applicant's race/ethnicity.

DESIGN

Retrospective review of applications to pediatric surgery fellowship at a single institution (2016-2020). Race was self-reported by applicants. LoR were analyzed via the Linguistic Inquiry and Word Count (LIWC) software program.

SETTING

Johns Hopkins All Children's Hospital, St. Petersburg, Florida USA. A free-standing tertiary pediatric hospital.

PARTICIPANTS

Pediatric surgery fellowship applicants from 2016 to 2020.

RESULTS

A total of 1086 LoR from 280 applicants (52% female) were analyzed. Racial distribution was Caucasians 62.1%, Asian 12.1%, Hispanics 7.1%, multiracial 6.4% African Americans 5%, and other/unknown 7.1%. Letter writers were largely male (84%), pediatric surgeons (63%) and professors (57%). There was no difference in LoR word count across races. LoR for female multiracial candidates contained higher use of affiliation and negative emotion terms compared to Hispanic females (p = 0.002 and 0.048, respectively), and past focus terms when compared to Caucasian and Asian female applicants (p < 0.001 and p = 0.003, respectively). Religion terms were more common in LoR for Asian females when compared to Caucasian females (p < 0.001).

CONCLUSION

This study demonstrates linguistic differences in LoR for pediatric surgery training programs based on applicant race/ethnicity. While differences are present, these do not suggest overt bias based on applicants race or ethnicity.

The Utility of the Plastic Surgery Standardized Letter of Recommendation Form in Predicting Residency Match Outcomes

BACKGROUND

Letters of recommendation play an important role in resident selection. While plastic surgery's Standardized Letter of Recommendation (SLOR) form most commonly serves as an adjunct to narrative letters, the SLOR provides objective data in the review process and could eventually replace narrative letters. The utility of the SLOR in predicting Match outcomes has not been studied.

METHODS

Applicant data from 225 first-time residency applicants in 2020-21 were collected. Logistic regression modeling was used to predict Match outcomes. This model was validated using 100 randomly selected applicants from 2019-20.

RESULTS

Rank placement (SLOR Question 6) was the most important factor when predicting Match outcomes (p<0.0001). All other SLOR questions were not predictive and subject to notable score inflation. No SLOR score differences were noted based on race; female applicants were rated higher in two of ten domains (p<0.05).

CONCLUSIONS

One question on the plastic surgery SLOR was highly predictive of an applicant matching. However, the remaining SLOR questions had little utility and were subject to gross score inflation. Further work should be done to optimize the utility of the SLOR in differentiating applicants. This has important implications in ensuring the selection of professional, competent residents according to the aims of the Accreditation Council of Graduate Medical Education.