1
|
Schrepel C, Sehdev M, Dubosh NM, Schnapp BH, Alvarez A, Pelletier‐Bui A, Hegarty C, Bord S, Shappell E. Decoding competitiveness: Exploring how emergency medicine faculty interpret standardized letters of evaluation. AEM EDUCATION AND TRAINING 2024; 8:e11019. [PMID: 39185031 PMCID: PMC11343972 DOI: 10.1002/aet2.11019] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/05/2024] [Revised: 08/03/2024] [Accepted: 08/12/2024] [Indexed: 08/27/2024]
Abstract
Background Given the importance of the standardized letter of evaluation (SLOE) for application to emergency medicine (EM) residency, it is important that SLOE developers and authors understand how reviewers determine SLOE competitiveness. To inform SLOE design and authorship, the authors set out to build a novel theory to explain how faculty holistically interpret SLOE competitiveness. Methods The authors used constructivist grounded theory to explore how EM faculty determine SLOE competitiveness. They used purposive sampling to recruit EM faculty participants with at least 1 year of experience in scoring SLOEs. One author conducted hour-long, semistructured interviews over Zoom between August 2023 and March 2024. Two authors iteratively coded the data to develop the initial codebook, organize codes into categories, and build connections to construct the resulting theory. Results The authors interviewed 11 EM faculty from throughout the United States. Participants described a complex process to determine SLOE competitiveness. They began by contextualizing the SLOE to determine its trustworthiness and value before using various components of the SLOE to stratify and refine their understanding of competitiveness. Finally, when participants noted the inconsistency between different aspects of the SLOE, they used various methods to reconcile discordances and determine competitiveness. Conclusions This study illuminates the framework used by EM faculty to determine applicant competitiveness based on the SLOE and highlights several factors that SLOE authors should consider to ensure the accurate and efficient transfer of information.
Collapse
Affiliation(s)
| | - Morgan Sehdev
- Harvard‐Affiliated Emergency Medicine ResidencyBostonMassachusettsUSA
| | - Nicole M. Dubosh
- Beth Israel Deaconess Medical Center/Harvard Medical SchoolBostonMassachusettsUSA
| | - Benjamin H. Schnapp
- University of Wisconsin School of Medicine and Public HealthMadisonWisconsinUSA
| | - Al’ai Alvarez
- Stanford University School of MedicinePalo AltoCaliforniaUSA
| | - Alexis Pelletier‐Bui
- Cooper Medical School of Rowan University/Cooper University HospitalCamdenNew JerseyUSA
| | - Cullen Hegarty
- University of Minnesota Medical SchoolMinneapolisMinnesotaUSA
| | - Sharon Bord
- The Johns Hopkins University School of MedicineBaltimoreMarylandUSA
| | - Eric Shappell
- Massachusetts General Hospital/Harvard Medical SchoolBostonMassachusettsUSA
| |
Collapse
|
2
|
Burkhart RJ, Lavu MS, Hecht II CJ, Ina JG, Gillespie RJ, Liu RW. How Prominent Are Gender Bias, Racial Bias, and Score Inflation in Orthopaedic Surgery Residency Recommendation Letters? A Systematic Review. Clin Orthop Relat Res 2024; 482:916-928. [PMID: 39031051 PMCID: PMC11124601 DOI: 10.1097/corr.0000000000003062] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/27/2023] [Accepted: 03/05/2024] [Indexed: 07/22/2024]
Abstract
INTRODUCTION Letters of recommendation are considered an important factor in the holistic ranking of orthopaedic residency applications. The standardized letter of recommendation (SLOR) was introduced in 2017 in response to the inherent bias and limited comparative value of traditional LORs. It includes standardized questions about an applicant's orthopaedic qualifications, designed to enable fair comparisons among candidates. However, disparate and inconsistent findings have made it difficult to draw meaningful conclusions from individual studies on the SLOR and narrative letter of recommendation. QUESTION/PURPOSES In this systematic review we asked: (1) What is the distribution of applicant SLOR rating among nine domains and summative scores? (2) Are there applicant characteristics associated with SLOR ratings? (3) Is there gender and racial bias in the SLOR and narrative letter of recommendation? METHODS PubMed, EBSCO, and Google Scholar electronic databases were queried on March 20, 2023, to identify all studies evaluating SLOR and narrative letter of recommendations between January 1, 2017, and March 20, 2023. Articles that evaluated orthopaedic SLOR or narrative LORs were included. Systematic reviews, case reports, duplicate studies between databases, and grey literature such as abstracts and articles on preprint servers were excluded. Eight studies including 6179 applicants and 18,987 letters of recommendation were included in the final analysis. The applicant classes ranged from years 2014 to 2020. Two reviewers independently evaluated the quality of the included studies using the Joanna Briggs Institute (JBI) tool for cross-sectional studies. The mean JBI score of included studies was 7.4 out of a maximum of 8, with higher scores indicating better quality. The primary outcome was to determine the distribution of applicant SLOR rankings in their summative score. Summative scores were separated into ranked to match, top one-third, middle one-third, lower one-third, and not a fit. Four studies reported the summative SLOR scores of applicants. Our secondary study goal was to assess applicant characteristics associated with SLOR summative scores and assess any bias present in the SLOR or narrative recommendation. Five studies compared SLOR ratings across applicant characteristics including gender, race, USMLE Step 1 score, USMLE Step 2 score, Alpha Omega Alpha (AΩA) membership, clerkship grades, and publications. Gender and racial bias were also assessed across five studies. Four studies utilized a linguistic analysis software to search letters of recommendation for differences in word category use by gender and race. RESULTS Studies consistently found that a higher percentage of candidates were identified in the top percentile blocks than is mathematically possible. For example, the two studies that tallied the proportion of candidates ranking in the top 10% of applicants found that 36% and 42% were rated as being in the top 10%. Similarly, articles found more than 87% of applicants scored in the top one-third. Studies had mixed results, but in general, they suggested that AΩA status, higher Step 1 scores, and more research publications have a slightly positive association with increased SLOR scores. We found no evidence of gender bias against women, and in fact, most studies evaluating word choices found word choices for women candidates generally were stronger. Similarly, no consistent disparities were identified in word categories utilized in SLORs based on applicant race. CONCLUSION Most notably, a mathematically impossible proportion of applicants were counted in the top percentiles in letters of recommendation. This compromises readers' abilities to differentiate candidates. Factors like AOA status and research publications displayed a modest positive correlation with SLOR scores. Gender bias against women or candidates from racial minority groups was not evident. CLINICAL RELEVANCE We suggest that group SLOR authorship, with a consensus assessment by clerkship or residency directors, should be considered. We also recommend the incorporation of mean and SD scores for each letter writer (or group) alongside their letters. Furthermore, studies indicate that submitting all four SLOR letters can substantially improve an applicant's chances of success. We advise students to choose strategically and submit the maximum allowable number of SLORs.
Collapse
Affiliation(s)
- Robert J. Burkhart
- Department of Orthopaedic Surgery, University Hospitals of Cleveland, Case Western Reserve University, Cleveland, OH, USA
| | - Monish S. Lavu
- Department of Orthopaedic Surgery, University Hospitals of Cleveland, Case Western Reserve University, Cleveland, OH, USA
| | - Christian J. Hecht II
- Department of Orthopaedic Surgery, University Hospitals of Cleveland, Case Western Reserve University, Cleveland, OH, USA
| | - Jason G. Ina
- Department of Orthopaedic Surgery, University Hospitals of Cleveland, Case Western Reserve University, Cleveland, OH, USA
| | - Robert J. Gillespie
- Department of Orthopaedic Surgery, University Hospitals of Cleveland, Case Western Reserve University, Cleveland, OH, USA
| | - Raymond W. Liu
- Department of Orthopaedic Surgery, University Hospitals of Cleveland, Case Western Reserve University, Cleveland, OH, USA
| |
Collapse
|
3
|
Leopold SS. Editor's Spotlight/Take 5: How Prominent Are Gender Bias, Racial Bias, and Score Inflation in Orthopaedic Surgery Residency Recommendation Letters? A Systematic Review. Clin Orthop Relat Res 2024; 482:912-915. [PMID: 38809669 PMCID: PMC11124659 DOI: 10.1097/corr.0000000000003078] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/07/2024] [Accepted: 03/18/2024] [Indexed: 05/31/2024]
Affiliation(s)
- Seth S Leopold
- Editor-in-Chief, Clinical Orthopaedics and Related Research ®, Park Ridge, IL, USA
| |
Collapse
|
4
|
Gordon AM, Mont MA, Choueka J. The Utility of the Standardized Letter of Recommendation for Orthopedic Surgery Residency Applicants: A Systematic Review. Orthopedics 2024; 47:e1-e5. [PMID: 37672777 DOI: 10.3928/01477447-20230901-02] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 09/08/2023]
Abstract
Despite widespread adoption for evaluating residency candidates, few studies have evaluated the orthopedic standardized letter of recommendation (SLOR). A systematic review using PubMed, Embase, and Web of Science was performed in June 2022. Study design and results from SLOR investigations were compiled. Common outcomes studied were summative rank statement scores and SLOR individual domains. Applicants were rated ranked to match or in the top one-third of rank lists in non-normally distributed frequencies. The association of summative rank statement score with match outcome was rarely studied. Applicants' ratings skew positively, the utility is reportedly limited, and influence on match outcome has been inadequately studied. [Orthopedics. 2024;47(1):e1-e5.].
Collapse
|
5
|
Moran CJ, Dzara K, Frey-Vogel AS, Flaherty M, Hall D, Nelson BA, Sparger K, Stanley T, Yager P, Scott-Vernaglia S. Confidence of Faculty in Writing Letters of Recommendation for Pediatric Fellowship Applicants. Cureus 2023; 15:e49750. [PMID: 38161882 PMCID: PMC10757671 DOI: 10.7759/cureus.49750] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 11/30/2023] [Indexed: 01/03/2024] Open
Abstract
BACKGROUND The assessment of pediatric residents applying to subspecialty fellowship programs relies on faculty letters of recommendation (LOR). However, it is unclear if pediatric faculty are confident that their LOR are effective. OBJECTIVE This study aims to assess the confidence of pediatric faculty in writing an effective LOR for pediatric residents applying to subspecialty fellowship programs. METHODS Survey development was conducted using evidence-based best practices. Surveys were distributed via email in 2021 to all full-time pediatric faculty members who taught pediatric residents in a large academic medical center. Categorical values were compared by chi-square test. RESULTS Eighty-five out of 150 (57%) faculty members completed the survey. Forty-one percent of participants were very confident that their LOR provided adequate content to assess residents during the application process. Confidence was associated with higher academic rank (p=0.02), frequent contact with residents (p=0.01), and writing >2 LOR in the last five years (p=0.0002). Confident LOR writers were more likely to describe their own background, details about the resident's scholarly activity, and the resident's ability to work as part of a team. Thirty-five percent of respondents reported never considering gender bias when writing LOR, whereas 28% reported always considering gender bias. Eighty-seven percent of respondents reported an interest in receiving LOR writing guidelines. CONCLUSION Half of the faculty respondents were not very confident in their ability to write an effective LOR for pediatric residents applying for a fellowship. Faculty development and standardized instructions on writing effective LOR may be helpful both at the institutional and national levels, including the importance of considering gender bias when writing LOR.
Collapse
Affiliation(s)
- Christopher J Moran
- Division of Pediatric Gastroenterology, Hepatology, and Nutrition, Mass General for Children, Boston, USA
- Department of Pediatrics, Harvard Medical School, Boston, USA
| | - Kristina Dzara
- Department of Family and Community Medicine, Saint Louis University School of Medicine, Saint Louis, USA
- Center for Educator Development, Advancement, and Research, Saint Louis University School of Medicine, Saint Louis, USA
| | - Ariel S Frey-Vogel
- Department of Pediatrics, Harvard Medical School, Boston, USA
- Division of General Pediatrics, Mass General for Children, Boston, USA
| | - Michael Flaherty
- Division of Pediatric Critical Care Medicine, Mass General for Children, Boston, USA
- Department of Pediatrics, Harvard Medical School, Boston, USA
| | - Daniel Hall
- Division of General Pediatrics, Mass General for Children, Boston, USA
- Department of Pediatrics, Harvard Medical School, Boston, USA
| | - Benjamin A Nelson
- Division of Pediatric Pulmonology, Mass General for Children, Boston, USA
- Department of Pediatrics, Harvard Medical School, Boston, USA
| | - Katherine Sparger
- Division of Neonatology, Mass General for Children, Boston, USA
- Department of Pediatrics, Harvard Medical School, Boston, USA
| | - Takara Stanley
- Division of Pediatric Endocrinology, Mass General for Children, Boston, USA
- Department of Pediatrics, Harvard Medical School, Boston, USA
| | - Phoebe Yager
- Division of Pediatric Critical Care Medicine, Mass General for Children, Boston, USA
- Department of Pediatrics, Harvard Medical School, Boston, USA
| | - Shannon Scott-Vernaglia
- Division of General Pediatrics, Mass General for Children, Boston, USA
- Department of Pediatrics, Harvard Medical School, Boston, USA
| |
Collapse
|
6
|
Jodeh DS, Miller-Ocuin JL, Ginesi M, Abelson JS, Saraidaridis JT, Stein SL, Steinhagen E. Standardized Letter of Recommendation: Can Everyone Be Awesome? JOURNAL OF SURGICAL EDUCATION 2023; 80:39-50. [PMID: 36085114 DOI: 10.1016/j.jsurg.2022.08.003] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/31/2022] [Revised: 07/27/2022] [Accepted: 08/07/2022] [Indexed: 06/15/2023]
Abstract
OBJECTIVE Standardized letters of recommendation (SLOR) are hypothesized to decrease bias and provide consistent domains for evaluation. However, their ability to differentiate among applicants is unknown. The utilization and functionality of SLOR and the impact of SLOR domain rating on matching for colon and rectal surgery (CRS) residency applicants have yet to be assessed. DESIGN Descriptive statistics and bivariate analysis were employed. Applicants were categorized into 3 groups; Top-tier(TT): applicants rated 100% Excellent/Very Good; Mid-Tier(MT) applicants rated 80-99% Excellent/Very Good; and Non-Top Tier(NTT) applicants rated <80% Excellent/Very Good. SETTING University of Hospitals Cleveland Medical Center. PARTICIPANTS SLORs submitted to a single colorectal surgery residency in 2019 were analyzed RESULTS: A total of 101 applicants were included, 54 (53.5%) of the applicants were male. 75 (74.2%) applicants who applied to our residency matched into a CRS residency, compared to the national rate of 66%. Of the 101 applicants with SLOR, 54 (53.5%) were categorized as TT, 26 (25.7%) as MT, and 21(20.8%) as NTT. The univariable analysis demonstrated a statistically significant difference in research experience (p=0.029) and match status (p=0.01) between applicant tiers. There were no statistically significant differences between applicant-tier and demographics, foreign medical graduates (FMG), H-indices, ABSITE scores, type of residency, preliminary year, completing an unaccredited CRS, and applicants with an additional degree. On multivariable analysis age (OR=0.65; CI=0.48-0.87) and FMG applicants (OR=0.05; CI=0.01-0.44) were inversely associated with successfully matching. Compared to TT applicants, MT (OR=0.07; CI=0.01-0.57) and NTT (OR=0.04; CI=0.01-0.34) applicants were inversely associated with a successful match. Individuals who completed research prior to residency but after medical school was associated with successfully matching (p=0.009). CONCLUSIONS The presence of MT and NTT ratings is associated with failure to match and may represent an area of concern for CRS programs rather than a tool to discern differences between candidates.
Collapse
Affiliation(s)
- Diana S Jodeh
- University Hospitals Research in Surgical Outcomes & Effectiveness Center (UH-RISES), Department of Surgery, University Hospitals Cleveland Medical Center, Cleveland, Ohio
| | - Jennifer L Miller-Ocuin
- University Hospitals Research in Surgical Outcomes & Effectiveness Center (UH-RISES), Department of Surgery, University Hospitals Cleveland Medical Center, Cleveland, Ohio
| | - Meridith Ginesi
- University Hospitals Research in Surgical Outcomes & Effectiveness Center (UH-RISES), Department of Surgery, University Hospitals Cleveland Medical Center, Cleveland, Ohio
| | - Jonathan S Abelson
- Lahey Hospital & Medical Center, Department of Surgery, Burlington, Massachusetts
| | - Julia T Saraidaridis
- Lahey Hospital & Medical Center, Department of Surgery, Burlington, Massachusetts
| | - Sharon L Stein
- University Hospitals Research in Surgical Outcomes & Effectiveness Center (UH-RISES), Department of Surgery, University Hospitals Cleveland Medical Center, Cleveland, Ohio
| | - Emily Steinhagen
- University Hospitals Research in Surgical Outcomes & Effectiveness Center (UH-RISES), Department of Surgery, University Hospitals Cleveland Medical Center, Cleveland, Ohio.
| |
Collapse
|
7
|
Feldman H, Blackmon S, Lawton JS, Antonoff MB. Dear sirs, your bias is showing: Implicit bias in letters of recommendation. J Thorac Cardiovasc Surg 2023; 165:398-400. [PMID: 35599208 DOI: 10.1016/j.jtcvs.2022.04.017] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/24/2022] [Revised: 04/05/2022] [Accepted: 04/07/2022] [Indexed: 12/16/2022]
Affiliation(s)
- Hope Feldman
- Department of Thoracic and Cardiovascular Surgery, University of Texas MD Anderson Cancer Center, Houston, Tex
| | - Shanda Blackmon
- Division of General Thoracic Surgery, Mayo Clinic, Rochester, Minn
| | - Jennifer S Lawton
- Division of Cardiac Surgery, Department of Surgery, Johns Hopkins University School of Medicine, Baltimore, Md
| | - Mara B Antonoff
- Department of Thoracic and Cardiovascular Surgery, University of Texas MD Anderson Cancer Center, Houston, Tex.
| |
Collapse
|
8
|
Tavarez MM, Baghdassarian A, Bailey J, Caglar D, Eckerle M, Fang A, McVety K, Nagler J, Ngo TL, Rose JA, Roskind CG, Benedict FT, Nesiama JAO, Thomas AA, Langhan ML. A Call to Action for Standardizing Letters of Recommendation. J Grad Med Educ 2022; 14:642-646. [PMID: 36591418 PMCID: PMC9765898 DOI: 10.4300/jgme-d-22-00131.1] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/23/2022] Open
Affiliation(s)
- Melissa M. Tavarez
- Melissa M. Tavarez, MD, MS, is Program Director, Pediatric Emergency Medicine (PEM) Fellowship, Department of Pediatrics, Division of PEM, University of Pittsburgh School of Medicine/UPMC Children's Hospital of Pittsburgh
| | - Aline Baghdassarian
- Aline Baghdassarian, MD, MPH, is Program Director, PEM Fellowship, Department of Pediatrics, Section of PEM, Inova LJ Murphy Children's Hospital
| | - Jessica Bailey
- Jessica Bailey, MD, is Program Director, PEM Fellowship, Department of Emergency Medicine, Oregon Health & Science University
| | - Derya Caglar
- Derya Caglar, MD, is Program Director, PEM Fellowship, Department of Pediatrics, Division of PEM, University of Washington School of Medicine/Seattle Children's Hospital
| | - Michelle Eckerle
- Michelle Eckerle, MD, MPH, is Program Director, PEM Fellowship, Department of Pediatrics, University of Cincinnati College of Medicine/Division of PEM, Cincinnati Children's Hospital Medical Center
| | - Andrea Fang
- Andrea Fang, MD, is Clinical Associate Professor, Department of Emergency Medicine, Stanford University School of Medicine
| | - Katherine McVety
- Katherine McVety, MD, is Associate Program Director, PEM Fellowship, Department of Pediatrics, Division of PEM, Children's Hospital of Michigan
| | - Joshua Nagler
- Joshua Nagler, MD, MHPEd, is Program Director, PEM Fellowship, Department of Pediatrics, Division of PEM, Boston Children's Hospital
| | - Thuy L. Ngo
- Thuy L. Ngo, DO, MEd, is Program Director, PEM Fellowship, Department of Pediatrics, Division of PEM, Johns Hopkins University School of Medicine
| | - Jerri A. Rose
- Jerri A. Rose, MD, is Program Director, PEM Fellowship, Department of Pediatrics, Division of PEM, UH Rainbow Babies & Children's Hospital
| | - Cindy Ganis Roskind
- Cindy Ganis Roskind, MD, is Program Director, PEM Fellowship, Department of Emergency Medicine, Division of PEM, Columbia University Irving Medical Center
| | - Frances Turcotte Benedict
- Frances Turcotte Benedict, MD, MPH, is Program Director, PEM Fellowship, Department of Pediatrics, Division of PEM, University of Missouri Kansas City School of Medicine/Children's Mercy Hospital
| | - Jo-Ann O. Nesiama
- Jo-Ann O. Nesiama, MD, MS, is Associate Program Director, PEM Fellowship, Department of Pediatrics, Division of Emergency Medicine, UT Southwestern Medical Center
| | - Anita A. Thomas
- Anita A. Thomas, MD, MPH, is Associate Program Director, PEM Fellowship, Department of Pediatrics, Division of PEM, University of Washington School of Medicine/Seattle Children's Hospital
| | - Melissa L. Langhan
- Melissa L. Langhan, MD, MHS, is Program Director, PEM Fellowship, Department of Pediatrics, Section of PEM, Yale University School of Medicine
| |
Collapse
|
9
|
Use of standardized letters of recommendation for orthopaedic surgery residency: a national survey study. CURRENT ORTHOPAEDIC PRACTICE 2022. [DOI: 10.1097/bco.0000000000001186] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/30/2022]
|
10
|
Preserve That Narrative! Supporting and Differentiating Applicants in an Increasingly Distanced and Stressful World. Acad Radiol 2022; 29:1455-1457. [PMID: 35965157 DOI: 10.1016/j.acra.2022.07.010] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/27/2022] [Revised: 07/13/2022] [Accepted: 07/13/2022] [Indexed: 12/14/2022]
|
11
|
Development and Assessment of Early Utilization of the Standardized Letter of Recommendation for Use in the Radiology Residency Match. Acad Radiol 2022; 29:1583-1589. [PMID: 35341687 DOI: 10.1016/j.acra.2022.02.004] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/14/2021] [Revised: 01/31/2022] [Accepted: 02/03/2022] [Indexed: 12/14/2022]
Abstract
RATIONALE AND OBJECTIVES Letters of recommendation are essential for residency applications. Traditionally, narrative letters have been used. Standardized letters of recommendation (SLOR) have been developed and have benefits to the writer and reader. The goal was to develop an informative, meaningful, and efficient SLOR for the radiology residency Match and to assess its early use. MATERIALS AND METHODS An Association of University Radiologists (AUR) Research and Education Venture Fund Grant was awarded for the development of a SLOR for use in the radiology residency Match. Grant recipients developed the letter and modifications were based on feedback from the Alliance of Medical Student Educators in Radiology (AMSER) SLOR task force and a larger task force including additional AUR and Association of Program Directors in Radiology (APDR) members. AUR and APDR members were surveyed in 2020 to assess the radiology SLOR. RESULTS The radiology SLOR became available for use in September 2018. It highlights the top six traits or abilities deemed most valued in a resident candidate and guides the writer to develop a concrete narrative. Top perceived benefits of the Radiology SLOR are ease of reading and interpretation. Top perceived drawbacks are an impersonal format and concerns regarding program directors' perceptions of the new SLOR. SLOR utilization increased in the second year of availability. CONCLUSION The radiology SLOR was developed and first used in 2018. Its use increased over two years and will likely continue to increase given the perceived benefits and increasing awareness of the SLOR.
Collapse
|
12
|
Feldman MJ, Ortiz AV, Roth SG, Dambrino RJ, Yengo-Kahn AM, Chitale RV, Chambless LB. Trends in utilization of standardized letters of recommendation in the 2021-2022 neurosurgery application process. J Neurosurg 2022; 138:1132-1138. [PMID: 36087327 DOI: 10.3171/2022.7.jns221186] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2022] [Accepted: 07/06/2022] [Indexed: 11/06/2022]
Abstract
OBJECTIVE Standardized letters of recommendation (SLORs) were introduced during the 2020-2021 neurosurgery residency application cycle, but suffered from poor interrater reliability and grade inflation. Changes were made to the SLOR template and utilization patterns in response to these shortcomings. The authors examined the second year of SLOR utilization. They hypothesized that grade inflation and interrater reliability would be improved from the first iteration. They also hypothesized that increased numbers of letters by single writers would correlate with broader rating distributions. METHODS This cross-sectional study analyzed all SLORs submitted to a single neurosurgery residency program over the 2021-2022 cycle. Data from 7 competency domains and the overall rating were recorded and stratified by academic category of the letter writer. Interrater reliability was evaluated using Krippendorff's alpha. The frequency of letters written was evaluated using the Kruskal-Wallis H test. RESULTS Ninety percent of SLORs rated applicants among the top 25%, but there was a significant decrease in the usage of the top 1% and top 2%-5% ratings. Interrater reliability was poor across all competencies. Writers who completed 1 SLOR rated applicants higher and had a narrower range than those who completed multiple SLORs. CONCLUSIONS Changes in the format and subsequent utilization patterns of SLORs have slightly decreased grade inflation; however, interrater reliability remains poor. The most wide-ranging evaluators submitted the highest number of SLORs, suggesting that future evaluation and usage of SLORs should emphasize letter-writer characteristics and numbers of SLORs written. Overall, SLORs have been well and broadly accepted with subtle improvements in the second year of utilization.
Collapse
Affiliation(s)
- Michael J Feldman
- 1Department of Neurological Surgery, Vanderbilt University Medical Center, Nashville; and
| | | | - Steven G Roth
- 1Department of Neurological Surgery, Vanderbilt University Medical Center, Nashville; and
| | - Robert J Dambrino
- 1Department of Neurological Surgery, Vanderbilt University Medical Center, Nashville; and
| | - Aaron M Yengo-Kahn
- 1Department of Neurological Surgery, Vanderbilt University Medical Center, Nashville; and
| | - Rohan V Chitale
- 1Department of Neurological Surgery, Vanderbilt University Medical Center, Nashville; and
| | - Lola B Chambless
- 1Department of Neurological Surgery, Vanderbilt University Medical Center, Nashville; and
| |
Collapse
|
13
|
Tidwell J, Yudien M, Rutledge H, Terhune KP, LaFemina J, Aarons CB. Reshaping Residency Recruitment: Achieving Alignment Between Applicants and Programs in Surgery. JOURNAL OF SURGICAL EDUCATION 2022; 79:643-654. [PMID: 35123913 DOI: 10.1016/j.jsurg.2022.01.004] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/29/2021] [Revised: 12/15/2021] [Accepted: 01/08/2022] [Indexed: 06/14/2023]
Abstract
OBJECTIVE The residency recruitment process has become increasingly challenging for both applicants and program directors, in part, due to the inflation in the number of applications per student. As a result, it has become more daunting for programs to design processes that evaluate applicants holistically. Furthermore, the existing methods used to evaluate and select applicants do not necessarily predict success in residency and may inadvertently lend to gender, racial, and ethnic bias. This narrative review aims to identify innovative tools used in residency recruitment that will allow programs and applicants to better determine concordance of interests and achieve value alignment while supporting improved, objective evaluation of an applicant's unique attributes and experiences. DESIGN PubMed was used to conduct a narrative review of recruitment strategies in admission processes of undergraduate and graduate medical education between 1975 and June 2021, using the designated Medical Subject Heading (MeSH0 terms. Inclusion criteria were established surrounding innovative tools to better objectively screen, evaluate, or select applicants. Strategies relying primarily on traditional metrics (United States Medical Licensing Examination (USMLE) scores, Alpha Omega Alpha status, and clerkship grades) were excluded. RESULTS Forty-two articles met specific inclusion criteria. Using these articles, a framework was created with two specific aims: (1) to allow applicants and programs to express or assess interest and (2) to foster objective review of unique applicant attributes, skills, experiences, and competencies that align with program mission and values. The following five innovative tools for recruitment were identified: preference signaling, secondary applications, standardized letters of recommendation, situational judgment testing, and surgical simulation. CONCLUSIONS As the number of applications continues to rise, strategies must be implemented to allow applicants and institutions to achieve better alignment or "fit," while also giving balanced consideration to all of an applicant's unique characteristics. A more holistic approach to applicant selection is a necessary tool in order to increase diversity and inclusion within the field of surgery.
Collapse
Affiliation(s)
- Jerica Tidwell
- Philadelphia College of Osteopathic Medicine, Philadelphia, Pennsylvania
| | - Mikhal Yudien
- University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania
| | - Hannah Rutledge
- Biomedical Library, University of Pennsylvania, Philadelphia, Pennsylvania
| | - Kyla P Terhune
- Department of Surgery, Vanderbilt University School of Medicine, Nashville, Tennessee
| | - Jennifer LaFemina
- Department of Surgery, University of Massachusetts Chan Medical School, Worcester, Massachusetts
| | - Cary B Aarons
- Department of Surgery, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania.
| |
Collapse
|
14
|
Gordon AM, Conway CA, Sheth BK, Magruder ML, Vakharia RM, Levine WN, Razi AE. How Did Coronavirus-19 Impact the Expenses for Medical Students Applying to an Orthopaedic Surgery Residency in 2020 to 2021? Clin Orthop Relat Res 2022; 480:443-451. [PMID: 34913886 PMCID: PMC8846343 DOI: 10.1097/corr.0000000000002042] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/28/2021] [Accepted: 10/13/2021] [Indexed: 01/31/2023]
Abstract
BACKGROUND Orthopaedic surgery is one of the most competitive specialties for residency applicants. For the 2021 residency match, the coronavirus-19 pandemic introduced complexity for programs and applicants because away rotations were limited and in-person interviews were cancelled. This may have changed the landscape in terms of expenses for candidates in important ways, but this topic has been insufficiently studied. QUESTIONS/PURPOSES Given that in 2021, students did not attend away rotations and all interviews were held virtually, we asked (1) What were the financial savings associated with this change? (2) Was medical school geographic region associated with differences in expenses when applying to residency? METHODS A retrospective, cross-sectional analysis of the 2020 and 2021 Texas Seeking Transparency in Application to Residency Dashboard database was performed. The data were derived from an online survey of a nationwide pool of applicants from 87% (123 of 141) of US allopathic medical schools upon conclusion of the match. The response percentage was 29% (521 of 1794). We believe this nationwide dataset represents the largest and most current data for this applicant group. Responses from applicants applying to orthopaedic surgery residency in the year before the COVID-19 pandemic application changes (2020) and during COVID-19 (2021) were queried and compared. After the orthopaedic surgery match, the database was evaluated for individual (application costs, away rotation expenses, and interview expenses) and total expenses for medical school seniors applying to orthopaedic surgery residency. Applicant characteristics were compared between application cycles. The 2020 to 2021 Texas Seeking Transparency in Application to Residency Dashboard database had 521 responses (n = 263 in 2020 and n = 258 in 2021) from applicants applying to orthopaedic surgery residency. Demographic and applicant characteristics were comparable between application cycles. Median expenses are reported with percentile distributions and geographic comparisons. A Mann-Whitney U test or Kruskal-Wallis H test was used to determine whether there were statistically significant differences in expenses between years and between medical school regions at a p value threshold of < 0.05. RESULTS For all applicants, the median total expenses (USD 7250 versus USD 2250), application costs (USD 2250 versus USD 1750), away rotation expenses (USD 2750 versus USD 250), and interview expenses (USD 2250 versus USD 75) declined in 2021 compared with 2020 (all p < 0.001). The median total savings in expenses for all applicants in 2021 compared with 2020 was USD 5000. In 2021, median total expenses were lower in all geographic regions with the greatest savings from applicants in the West (USD 6000); in addition, the difference in median total expenses between the geographic region with the highest total expenses and the lowest total expenses was lower in the pandemic year than it was in the year prior (USD 1000 versus USD 1500; p < 0.001). In 2021, there were differences in total expenses between the Northeast (USD 1750), West (USD 1750), and Central (USD 2750) regions (p < 0.001). From 2020 to 2021, only application fees from Northeast applicants differed (USD 2250 versus USD 1250; p < 0.001). In 2020, interview expenses were not different between all regions (USD 2250 Northeast and West versus USD 2750 Central and South; p = 0.19); similarly in 2021, interview expenses were similar between all regions (USD 75 versus USD 75; p = 0.82). Finally, in 2020, Northeast (USD 3250) and Western (USD 3250) applicants spent more for away rotations than Southern (USD 2750) and Central (USD 2250) applicants (p = 0.01). In 2021, applicants from schools in the South (USD 250) and Central (USD 250) regions spent more than their counterparts (USD 0; p = 0.028). CONCLUSION In the COVID-19 application cycle, the median expenditures of orthopaedic residency candidates were USD 5000 lower than they were in the previous year; the difference can be attributed to the use of virtual interviews and the lack of away rotations. There are geographic implications, with applicants from Western United States medical schools potentially saving the most. Despite the financial savings during the 2021 match, further study related to the long-term success of the current application process (both for applicants and programs) is needed. The recommendation in May 2020 by the AOA Council of Orthopaedic Residency Directors (CORD) to limit the number of applications submitted by candidates with USMLE Step 1 scores greater than 235 did not result in any considerable decline in applications submitted or expenses. A better understanding of how differences in these expenses may influence our specialty's ability to attract socioeconomically diverse candidates would be important, and we need to explore perceived and actual financial obstacles to obtaining this diversity in the application process. Finally, avenues should be explored by program directors and chairpersons to reduce the expenses of the traditional application process while maintaining recruitment of top candidates. LEVEL OF EVIDENCE Level IV, economic analysis.
Collapse
Affiliation(s)
- Adam M. Gordon
- Department of Orthopaedic Surgery, Maimonides Medical Center, New York, NY, USA
| | - Charles A. Conway
- Department of Orthopaedic Surgery, Maimonides Medical Center, New York, NY, USA
| | - Bhavya K. Sheth
- Department of Orthopaedic Surgery, Maimonides Medical Center, New York, NY, USA
| | - Matthew L. Magruder
- Department of Orthopaedic Surgery, Maimonides Medical Center, New York, NY, USA
| | - Rushabh M. Vakharia
- Department of Orthopaedic Surgery, Maimonides Medical Center, New York, NY, USA
| | - William N. Levine
- Department of Orthopaedic Surgery, Columbia University Medical Center, New York, NY, USA
| | - Afshin E. Razi
- Department of Orthopaedic Surgery, Maimonides Medical Center, New York, NY, USA
| |
Collapse
|
15
|
The Orthopaedic Resident Selection Process: Proposed Reforms and Lessons From Other Specialties. J Am Acad Orthop Surg 2022; 30:91-99. [PMID: 34288891 DOI: 10.5435/jaaos-d-21-00214] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/19/2021] [Accepted: 06/18/2021] [Indexed: 02/01/2023] Open
Abstract
INTRODUCTION Proposals for substantive reforms to the orthopaedic resident selection process are growing, given increasing applicant competitiveness, burgeoning inefficiencies and inequities of the current system, and impending transition of Step 1 to pass/fail. The COVID-19 pandemic has further catalyzed the need for reforms, offering unprecedented opportunities to pilot novel changes. However, a comprehensive collation of all proposed and implemented orthopaedic reforms is currently lacking. Thus, we aimed to characterize proposed orthopaedic-specific resident selection reforms in the context of reforms implemented by other specialties. METHODS EMBASE, MEDLINE, Scopus, and Web of Science databases were searched for references proposing reforms to the orthopaedic resident selection process published from 2005 to 2020. An inductive approach to qualitative content analysis was used to categorize reforms. RESULTS Twenty-six articles proposing 13 unique reforms to the orthopaedic resident selection process were identified. The most commonly proposed reforms included noncognitive assessments (n = 8), application caps (n = 7), standardized letters of recommendation (n = 5), program-specific supplemental applications (n = 5), creation of a centralized database of standardized program information (n = 4), use of a standardized applicant composite score (n = 4), and a moratorium on postinterview communication (n = 4). Importantly, nearly all of these reforms have also been proposed or implemented by other specialties. DISCUSSION Numerous reforms to the orthopaedic resident selection process have been suggested over the past 15 years, several of which have been implemented on a program-specific basis, including noncognitive assessments, supplemental applications, and standardized letters of recommendation. Careful examination of applicant and program experiences and Match outcomes after these reforms is imperative to inform future directions.
Collapse
|
16
|
Compliance with CDIM-APDIM Guidelines for Department of Medicine Letters: an Opportunity to Improve Communication Across the Continuum. J Gen Intern Med 2022; 37:125-129. [PMID: 33791934 PMCID: PMC8739400 DOI: 10.1007/s11606-021-06744-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/21/2020] [Revised: 02/04/2021] [Accepted: 03/17/2021] [Indexed: 01/03/2023]
Abstract
BACKGROUND With rising applications to internal medicine programs and pending changes in United States Medical Licensing Examination Step 1 score reporting, program directors desire transparent data for comparing applicants. The Department of Medicine Letters of Recommendation (DOM LORs) are frequently used to assess applicants and have the potential to provide clearly defined data on performance including stratification of a medical school class. Despite published guidelines on the expected content of the DOM LOR, these LORs do not always meet that need. OBJECTIVES To better understand the degree to which DOM LORs comply with published guidelines. METHODS We reviewed DOM LORs from 146 of 155 LCME-accredited medical schools in the 2019 Match cycle, assessing for compliance with published guidelines. RESULTS Adherence to the recommendation for DOM LORs to provide a final characterization of performance relative to peers was low (68/146, 47%). Of those that provided a final characterization, 19/68 (28%) provided a quantitative measure, and 49/68 (72%) provided a qualitative descriptor. Only 17/49 (35%) with qualitative terms described those terms, and thirteen distinct qualitative scales were identified. Ranking systems varied, with seven different titles given to highest performers. Explanations about determination of ranking groups were provided in 12% of cases. CONCLUSIONS Adherence to published guidelines for DOM LORs varies but is generally low. For program directors desiring transparent data to use in application review, clearly defined data on student performance, stratification groupings, and common language across schools could improve the utility of DOM LORs.
Collapse
|
17
|
Gordon AM, Ahlering TE. How Does Geographic Region Affect the Total and Individual Costs for Medical Students Applying to the Competitive Surgical Residencies? JOURNAL OF SURGICAL EDUCATION 2022; 79:147-156. [PMID: 34535435 DOI: 10.1016/j.jsurg.2021.08.016] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/24/2021] [Revised: 08/16/2021] [Accepted: 08/21/2021] [Indexed: 06/13/2023]
Abstract
OBJECTIVE Studies have individually assessed the expenses for medical students desiring a surgical residency in neurosurgery, orthopaedic surgery, otolaryngology, plastic surgery, and urology. A detailed comparison of expenses by geographic region have not been reported. The aim was to geographically compare the expenses of United States medical students applying to the 5 most competitive surgical residencies. SETTING/PARTICIPANTS Anonymous nationwide survey of plastic surgery, orthopaedic surgery, otolaryngology, urology, and neurological surgery residency applicants. DESIGN A cross sectional, retrospective analysis of the 2019-2020 Texas STAR Dashboard database, an online tool generated from a nationwide survey of students. Individual (application, away rotation, interview) and total costs for medical school seniors were recorded in addition to applicant characteristics. Mean and median costs were reported for each specialty with percentile distributions and geographic comparisons. A Kruskal-Wallis H test was performed to compare differences in costs between surgical specialties and medical school region. RESULTS In total, 1136 applicants to surgical residency were included. The number of applicants to orthopaedic surgery (OS) (n = 459), neurological surgery (NS) (n = 121), urology (UR) (n = 191), plastic surgery (PS) (n = 117), and otolaryngology (OTO) (n = 248) were reported. Mean total costs were (OS; $8,205), (NS; $11,882), (UR; $8,207), (PS; $10,845), and (OTO; $7,516) (p ≤ 0.029). Application fees were only significantly different in the northeast and southern applicants between the different specialties, notably orthopaedic surgery applicants spent the most. In all geographic regions excluding the west, neurosurgery and plastic surgery applicants spent significantly more than other specialties for interview costs and away rotation costs. In all geographic regions neurosurgery and plastic surgery applicants spent significantly more than other specialties in total costs. CONCLUSIONS Orthopaedic surgery applicants spend the most on application fees in select geographic regions. Neurosurgery and plastic surgery applicants spend more on interviews, away rotations, and total costs in nearly all geographic regions.
Collapse
Affiliation(s)
| | - Thomas E Ahlering
- University of California, Irvine, Department of Urology, Orange, California
| |
Collapse
|
18
|
Feldman MJ, Ortiz AV, Roth SG, Dambrino RJ, Yengo-Kahn AM, Chitale RV, Chambless LB. An Examination of Standardized Letters of Recommendation Rating Scales Among Neurosurgical Residency Candidates During the 2020-2021 Application Cycle. Neurosurgery 2021; 89:1005-1011. [PMID: 34624075 PMCID: PMC8600167 DOI: 10.1093/neuros/nyab346] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2021] [Accepted: 07/16/2021] [Indexed: 11/12/2022] Open
Abstract
BACKGROUND Letters of recommendation (LORs) are historically an important, though subjective, component of the neurosurgery residency application process. Standardized LORs (SLORs) were introduced during the 2020 to 2021 application cycle. The intent of SLORs is to allow objective comparison of applicants and to reduce bias. OBJECTIVE To examine the utility of SLORs during this application cycle. We hypothesized that "grade inflation" and poor inter-rater reliability, as described by other specialties using SLORs, would limit the utility of SLORs in their current form. METHODS This cross-sectional study analyzed all SLORs submitted to a single neurosurgery residency program over the 2020 to 2021 cycle. Data from 7 competency domains and the overall rating were recorded and stratified by academic category of letter writer. Inter-rater reliability was evaluated using Krippendorff's alpha. RESULTS One or more SLORs was submitted as part of 298 of 393 applications (76%). Approximately 58.3% of letters written by neurosurgery chairpersons rated a given applicant as being within the top 5% across all competencies. Approximately 44.4% of program director letters similarly rated applicants as amongst the top 5%, while 73.2% and 81.4% of letters by other neurosurgeons and general surgery evaluators, respectively, rated applicants in the top 5%. Inter-rater reliability was poor (<0.33) in all rating categories, including overall (α = 0.18). CONCLUSION The utility of the first iteration of SLORs in neurosurgery applications is undermined by significant "grade inflation" and poor inter-rater reliability. Improvements are necessary for SLORs if they are to provide meaningful information in future application cycles.
Collapse
Affiliation(s)
- Michael J Feldman
- Department of Neurological Surgery, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Alexander V Ortiz
- School of Medicine, Vanderbilt University, Nashville, Tennessee, USA
| | - Steven G Roth
- Department of Neurological Surgery, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Robert J Dambrino
- Department of Neurological Surgery, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Aaron M Yengo-Kahn
- Department of Neurological Surgery, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Rohan V Chitale
- Department of Neurological Surgery, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Lola B Chambless
- Department of Neurological Surgery, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| |
Collapse
|
19
|
London DA. Letter to the Editor: More Than One-third of Orthopaedic Applicants Are in the Top 10%: The Standardized Letter of Recommendation and Evaluation of Orthopaedic Resident Applicants. Clin Orthop Relat Res 2021; 479:2102-2103. [PMID: 33979306 PMCID: PMC8373548 DOI: 10.1097/corr.0000000000001822] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/29/2021] [Accepted: 04/22/2021] [Indexed: 01/31/2023]
|
20
|
Kukulski P, Ahn J. Validity Evidence for the Emergency Medicine Standardized Letter of Evaluation. J Grad Med Educ 2021; 13:490-499. [PMID: 34434509 PMCID: PMC8370378 DOI: 10.4300/jgme-d-20-01110.1] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/21/2020] [Revised: 02/04/2021] [Accepted: 04/15/2021] [Indexed: 11/06/2022] Open
Abstract
BACKGROUND The standardized letter of evaluation (SLOE) is the application component that program directors value most when evaluating candidates to interview and rank for emergency medicine (EM) residency. Given its successful implementation, other specialties, including otolaryngology, dermatology, and orthopedics, have adopted similar SLOEs of their own, and more specialties are considering creating one. Unfortunately, for such a significant assessment tool, no study to date has comprehensively examined the validity evidence for the EM SLOE. OBJECTIVE We summarized the published evidence for validity for the EM SLOE using Messick's framework for validity evidence. METHODS A scoping review of the validity evidence of the EM SLOE was performed in 2020. A scoping review was chosen to identify gaps and future directions, and because the heterogeneity of the literature makes a systematic review difficult. Included articles were assigned to an aspect of Messick's framework and determined to provide evidence for or against validity. RESULTS There have been 22 articles published relating to validity evidence for the EM SLOE. There is evidence for content validity; however, there is a lack of evidence for internal structure, relation to other variables, and consequences. Additionally, the literature regarding response process demonstrates evidence against validity. CONCLUSIONS Overall, there is little published evidence in support of validity for the EM SLOE. Stakeholders need to consider changing the ranking system, improving standardization of clerkships, and further studying relation to other variables to improve validity. This will be important across GME as more specialties adopt a standardized letter.
Collapse
Affiliation(s)
- Paul Kukulski
- Both authors are with the University of Chicago Medical Center
- is Assistant Professor and Assistant Clerkship Director, Section of Emergency Medicine, Department of Medicine
| | - James Ahn
- Both authors are with the University of Chicago Medical Center
- is Associate Professor and Program Director, Section of Emergency Medicine, Department of Medicine
| |
Collapse
|
21
|
Egan CR, Dashe J, Hussein AI, Tornetta P. Are Narrative Letters of Recommendation for Medical Students Interpreted as Intended by Orthopaedic Surgery Residency Programs? Clin Orthop Relat Res 2021; 479:1679-1687. [PMID: 34252917 PMCID: PMC8277265 DOI: 10.1097/corr.0000000000001691] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/29/2020] [Accepted: 02/02/2021] [Indexed: 01/31/2023]
Abstract
BACKGROUND Narrative letters of recommendation are an important component of the residency application process. However, because narrative letters of recommendation are almost always positive, it is unclear whether those reviewing the letters understand the writer's intended strength of support for a given applicant. QUESTIONS/PURPOSES (1) Is the perception of letter readers for narrative letters of recommendation consistent with the intention of the letter's author? (2) Is there inter-reviewer consistency in selection committee members' perceptions of the narrative letters of recommendation? METHODS Letter writers who wrote two or more narrative letters of recommendation for applicants to one university-based orthopaedic residency program for the 2014 to 2015 application cycle were sent a survey linked to a specific letter of recommendation they authored to assess the intended meaning regarding the strength of an applicant. A total of 247 unstructured letters of recommendation and accompanying surveys were sent to their authors, and 157 surveys were returned and form the basis of this study (response percentage 64%). The seven core members of the admissions committee (of 22 total reviewers) at a university-based residency program were sent a similar survey regarding their perception of the letter. To answer our research question about whether letter readers' perceptions about a candidate were consistent with the letter writer's intention, we used kappa values to determine agreement for survey questions involving discrete variables and Spearman correlation coefficients (SCCs) to determine agreement for survey questions involving continuous variables. To answer our research question regarding inter-reviewer consistency among the seven faculty members, we compared the letter readers' responses to each survey question using intraclass correlation coefficients (ICCs). RESULTS There was a negligible to moderate correlation between the intended and perceived strength of the letters (SCC 0.26 to 0.57), with only one of seven letter readers scoring in the moderate correlation category. When stratifying the applicants into thirds, there was only slight agreement (kappa 0.07 to 0.19) between the writers and reviewers. There were similarly low kappa values for agreement about how the writers and readers felt regarding the candidate matching into their program (kappa 0.14 to 0.30). The ICC for each question among the seven faculty reviewers ranged from poor to moderate (ICC 0.42 to 0.52). CONCLUSION Our results demonstrate that the reader's perception of narrative letters of recommendation did not correlate well with the letter writer's intended meaning and was not consistent between letter readers at a single university-based urban orthopaedic surgery residency program. CLINICAL RELEVANCE Given the low correlation between the intended strength of the letter writers and the perceived strength of those letters, we believe that other options such as a slider bar or agreed-upon wording as is used in many dean's letters may be helpful.
Collapse
Affiliation(s)
- Cameron R. Egan
- Harvard Combined Orthopaedic Surgery Residency Program, Massachusetts General Hospital, Boston, MA, USA
| | - Jesse Dashe
- Department of Orthopaedic Surgery, Boston Medical Center, Boston, MA, USA
| | - Amira I. Hussein
- Department of Orthopaedic Surgery, Boston Medical Center, Boston, MA, USA
| | - Paul Tornetta
- Department of Orthopaedic Surgery, Boston Medical Center, Boston, MA, USA
| |
Collapse
|
22
|
Zastrow RK, Burk-Rafel J, London DA. Systems-Level Reforms to the US Resident Selection Process: A Scoping Review. J Grad Med Educ 2021; 13:355-370. [PMID: 34178261 PMCID: PMC8207920 DOI: 10.4300/jgme-d-20-01381.1] [Citation(s) in RCA: 15] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/08/2020] [Revised: 01/18/2021] [Accepted: 02/18/2021] [Indexed: 11/06/2022] Open
Abstract
BACKGROUND Calls to reform the US resident selection process are growing, given increasing competition and inefficiencies of the current system. Though numerous reforms have been proposed, they have not been comprehensively cataloged. OBJECTIVE This scoping review was conducted to characterize and categorize literature proposing systems-level reforms to the resident selection process. METHODS Following Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines, searches of Embase, MEDLINE, Scopus, and Web of Science databases were performed for references published from January 2005 to February 2020. Articles were included if they proposed reforms that were applicable or generalizable to all applicants, medical schools, or residency programs. An inductive approach to qualitative content analysis was used to generate codes and higher-order categories. RESULTS Of 10 407 unique references screened, 116 met our inclusion criteria. Qualitative analysis generated 34 codes that were grouped into 14 categories according to the broad stages of resident selection: application submission, application review, interviews, and the Match. The most commonly proposed reforms were implementation of an application cap (n = 28), creation of a standardized program database (n = 21), utilization of standardized letters of evaluation (n = 20), and pre-interview screening (n = 13). CONCLUSIONS This scoping review collated and categorized proposed reforms to the resident selection process, developing a common language and framework to facilitate national conversations and change.
Collapse
Affiliation(s)
- Ryley K. Zastrow
- Ryley K. Zastrow, BS, is a Fourth-Year Medical Student, Department of Medical Education, Icahn School of Medicine at Mount Sinai
| | - Jesse Burk-Rafel
- Jesse Burk-Rafel, MD, MRes, is Assistant Professor, Department of Internal Medicine, and Assistant Director of UME-GME Innovation, Institute for Innovations in Medical Education, NYU Grossman School of Medicine
| | - Daniel A. London
- At the time of writing, Daniel A. London, MD, MS, was an Orthopaedic Surgery Resident, Department of Orthopaedic Surgery, Icahn School of Medicine at Mount Sinai, and is currently a Hand Surgery Fellow, Mary S. Stern Hand Surgery Fellowship, TriHealth
| |
Collapse
|
23
|
Gordon AM, Malik AT, Scharschmidt TJ, Goyal KS. Cost Analysis of Medical Students Applying to Orthopaedic Surgery Residency: Implications for the 2020 to 2021 Application Cycle During COVID-19. JB JS Open Access 2021; 6:JBJSOA-D-20-00158. [PMID: 34386683 PMCID: PMC8352616 DOI: 10.2106/jbjs.oa.20.00158] [Citation(s) in RCA: 16] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/17/2022] Open
Abstract
Although studies have evaluated the economic burden to medical students desiring an orthopaedic residency broadly, no study has evaluated in detail the application costs, away rotation costs, interview costs, and total costs. Given that the Association of American Medical Colleges and the American Orthopaedic Association's Council of Orthopaedic Residency Directors have recommended orthopaedic surgery residency programs for the 2021 residency application cycle cancel away rotations and in-person interviews, our objective was to evaluate the cost savings to medical students applying during this time and the potential implications.
Collapse
Affiliation(s)
- Adam M Gordon
- Department of Orthopaedics, The Ohio State University Wexner Medical Center, Columbus, Ohio
| | - Azeem Tariq Malik
- Department of Orthopaedics, The Ohio State University Wexner Medical Center, Columbus, Ohio
| | - Thomas J Scharschmidt
- Department of Orthopaedics, The Ohio State University Wexner Medical Center, Columbus, Ohio
| | - Kanu S Goyal
- Department of Orthopaedics, The Ohio State University Wexner Medical Center, Columbus, Ohio
| |
Collapse
|
24
|
Powers A, Gerull KM, Rothman R, Klein SA, Wright RW, Dy CJ. Race- and Gender-Based Differences in Descriptions of Applicants in the Letters of Recommendation for Orthopaedic Surgery Residency. JB JS Open Access 2020; 5:JBJSOA-D-20-00023. [PMID: 32803104 PMCID: PMC7386551 DOI: 10.2106/jbjs.oa.20.00023] [Citation(s) in RCA: 40] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/14/2022] Open
Abstract
Background Letters of recommendation (LOR) are an important component of trainee advancement and assessment. Examination of word use in LOR has demonstrated significant differences in how letter writers describe female and male applicants. Given the emphasis on increasing both gender and racial diversity among orthopaedic surgeons, we aimed to study gender and racial differences in LOR for applicants to orthopaedic surgery residencies. Methods All applications submitted to a single, academic orthopaedic residency program in 2018 were included. Self-identified gender and race were recorded. The LOR were analyzed via a text analysis software program using previously described categories of communal, agentic, grindstone, ability, and standout words. We examined the relative frequency of word use in letters for (1) male and female applicants and (2) white and underrepresented in orthopaedics (UiO) applicants, with the subgroup analysis based on whether standardized (using the American Orthopaedic Association template) or traditional (narrative) LOR were used. Results Two thousand six hundred twenty-five LOR were submitted for 730 applicants (79% men). Fifty-nine percent of applicants were self-identified as white, and 34% were self-identified as UiO. In traditional LOR, standout words (odds ratio [OR] 1.07; p = 0.01) were more likely to be used in letters for women compared with men, with no difference in any other word-use category. In standardized LOR, there were no gender-based differences in any word category. In traditional LOR, grindstone words (OR = 0.96; p = 0.02) were more likely to be used in letters for UiO than white applicants, whereas standout words (OR = 1.05; p = 0.04) were more likely to be used in letters for white candidates. In standardized LOR, there were no race-based differences in any word category use. Conclusions Small differences were found in the categories of words used to describe male and female candidates and white and UiO candidates. These differences were not present in the standardized LOR compared with traditional LOR. It is possible that the use of standardized LOR may reduce gender- and race-based bias in the narrative assessment of applicants.
Collapse
Affiliation(s)
- Alexa Powers
- Department of Orthopaedic Surgery, Washington University School of Medicine, St. Louis, Missouri.,Saint Louis University School of Medicine, St. Louis, Missouri
| | - Katherine M Gerull
- Department of Orthopaedic Surgery, Washington University School of Medicine, St. Louis, Missouri
| | - Rachel Rothman
- Department of Orthopaedic Surgery, Washington University School of Medicine, St. Louis, Missouri
| | - Sandra A Klein
- Department of Orthopaedic Surgery, Washington University School of Medicine, St. Louis, Missouri
| | - Rick W Wright
- Department of Orthopaedic Surgery, Washington University School of Medicine, St. Louis, Missouri
| | - Christopher J Dy
- Department of Orthopaedic Surgery, Washington University School of Medicine, St. Louis, Missouri
| |
Collapse
|
25
|
Abstract
Over recent months, coronavirus disease 2019 (COVID-19) has swept the world as a global pandemic, largely changing the practice of medicine as it was previously known. Physician trainees have not been immune to these changes-uncertainty during this time is undeniable for medical students at all levels of training. Of particular importance is the potential impact of COVID-19 on the upcoming residency application process for rising fourth-year students; a further source of added complexity in light of the newly integrated allopathic and osteopathic match in the 2020 to 2021 cycle. Owing to the impact COVID-19 could have on the residency match, insight regarding inevitable alterations to the application process and how medical students can adapt is in high demand. Furthermore, it is very possible that programs will inquire about how applicants spent their time while not in the hospital because of COVID-19, and applicants should be prepared to provide a meaningful answer. Although competitive at a basal level, the complexity of COVID-19 now presents an unforeseen, superimposed development in the quest to match. In this article, we aim to discuss and provide potential strategies for navigating the impact of COVID-19 on the residency application process for orthopaedic surgery.
Collapse
|