Revisiting the Gender Gap in Orthopaedic Surgery: Investigating the Relationship Between Orthopaedic Surgery Female Faculty and Female Residency Applicants

Sex Diversity and Equity Among Fellows of the American Orthopaedic Association

BACKGROUND

We hypothesized that women are underrepresented among fellows of the American Orthopaedic Association (AOA) relative to their prevalence among faculty at departments of orthopaedic surgery.

METHODS

Clinical faculty at departments of orthopaedic surgery that are affiliated with residency training programs were analyzed for AOA membership. Participation-to-prevalence ratios (PPRs) were calculated for men and women with AOA membership relative to their prevalence among orthopaedic surgeon faculty. A PPR of <0.8 or >1.2 indicated under- or overrepresentation, respectively. PPR values between 0.8 and 1.2 indicated equivalent representation. Sex disparities in AOA membership were explored by academic rank and orthopaedic subspecialty.

RESULTS

There were 4,310 orthopaedic surgery faculty, and 998 (23.2%) had AOA membership. Overall, women formed the minority of members in the AOA study cohort (11.4%), but they had equivalent representation relative to their prevalence among faculty (10.0%, PPR = 1.14, p = 0.210). In the AOA study cohort, women were the minority at each academic rank, including assistant professor (20.2% versus 79.8%, p < 0.001), associate professor (16.8% versus 83.2%, p < 0.001), and full professor (7.0% versus 93.0%, p < 0.001). However, women had greater representation in the AOA study cohort at each academic rank than would be expected based on their prevalence among faculty, including the assistant (PPR = 1.45), associate (PPR = 1.56), and full (PPR = 1.27) professor levels. Similarly, in the AOA study cohort, women were the minority in each orthopaedic subspecialty, but were only underrepresented in pediatric orthopaedics (PPR = 0.74). Women had greater representation in the AOA study cohort than would be expected by their prevalence among faculty in hand surgery (PPR = 1.51) and orthopaedic sports medicine (PPR = 1.33). In contrast, men had equivalent representation in each orthopaedic subspecialty.

CONCLUSIONS

Women form a minority group in the AOA, but they exhibit greater representation in the AOA than would be expected based on their prevalence among faculty at each academic rank, and they have greater representation than men, who exhibit equivalent representation across all academic ranks. Collectively, these results disproved our hypothesis, and suggest that women are participating in the AOA at rates that are greater than their prevalence among faculty.

CLINICAL RELEVANCE

Equitable participation of women in orthopaedic surgery is necessary to promote diversity and inclusion in the specialty. The AOA may be an important catalyst for future positive diversity and inclusion efforts within orthopaedic surgery.

The Sex or Race of Program Directors May Not Play a Significant Role in Impacting Diversity Among Orthopaedic Surgery Residents

PURPOSE

To identify the influence of residency program characteristics, including the presence of under-represented minorities in medicine (URiM) and/or female program directors (PDs), on the race and sex distribution of orthopaedic surgery residency trainees.

METHODS

All active and Accreditation Council for Graduate Medical Education-accredited orthopaedic surgery residency programs from 2017-2021 that reported usable information in the Residency Explorer Tool were included. Data collected included program characteristics, as well as faculty and resident sex distribution, ethnicity, race, and demographic characteristics. The PDs' specific sex, ethnicity, race, and demographic characteristics were collected using residency program websites. The prevalence of factors in programs with the top quartile of female and URiM residents was compared with that in programs with the bottom 3 quartiles.

RESULTS

Data were obtained from 148 of 200 Accreditation Council for Graduate Medical Education-accredited programs (3,694 residents). The 52 excluded programs had no usable information in the Residency Explorer Tool or on an identifiable program website. Overall, 15.9% of residents in orthopaedic surgery residency programs were women and 14% were under-represented minorities. The rates of female PDs and chairs were 12.4% and 6.9%, respectively, whereas those of URiM PDs and chairs were 8.3% and 4.6%, respectively. Programs with more female residents were not associated with female PDs (P = .79) or URiM PDs (P = .48). Programs with a greater percentage of URiM residents were not associated with URiM PDs (P = .16). Larger programs (P = .021) and university-based programs (P = .048) had a greater percentage of female residents. Orthopaedic residency programs with visa sponsorship had a greater percentage of URiM residents (P = .017).

CONCLUSIONS

Programs with a higher percentage of female or URiM residents did not show a significant association with having female or URiM PDs. Larger programs and university-affiliated programs were more likely to have a larger percentage of female residents, whereas programs that offered visa sponsorship had a higher percentage of URiM residents.

CLINICAL RELEVANCE

This study highlights factors influencing diversity among orthopaedic surgery residents. Although the presence of female or URiM PDs does not influence the percentage of female or URiM residents, other program characteristics such as size, affiliation, and visa sponsorship offer potentially actionable insights for promoting greater diversity in orthopaedic training. Doing so may increase under-represented groups within the field and could ultimately impact patient care and improve health care equity.

Orthopedic Surgery Residency Program Rankings and Gender Diversity

Background Orthopedic surgery residency programs are some of the least gender-diverse specialty programs in medicine. Despite strong representation at the undergraduate and medical school levels and increased applications to orthopedic surgery residency programs by women, there is still a substantial gender gap at the resident level. This study explores the relationship between the gender diversity of orthopedic surgery residency programs and program rankings. Methodology Program rank, program director gender identity, and gender diversity data were collected for the top 100 programs by reputation in Doximity. Gender diversity was measured as the proportion of female residents in the program and alumni. Results The greatest percentage of women in a program was 33% and the smallest was 3%. After linear regression analysis, we found that there was a statistically significant positive correlation between program rank and the proportion of women. The higher ranked a program was, the greater the proportion of women. There was no significant correlation between program director gender, appointment year, and program rank. Conclusions These results suggest that, although there is still a long way to go before closing the gender gap in orthopedic surgery residency programs, higher-ranked programs are associated with greater gender diversity than their lower-ranked counterparts.

Using a Modern Linked Research Database to Examine Gender Disparities in Orthopaedic Grant Funding from 2010 to 2022

BACKGROUND

Gender disparities in research grant funding persist in many disciplines. With use of the Dimensions database, we sought to examine the extent of gender disparities in U.S. orthopaedic grant funding from 2010 onward. Our aim was to provide insights into the extent of gender disparities in the field of orthopaedic research and to highlight the potential need for future action to address these disparities.

METHODS

Using orthopaedic-related search terms, we queried all U.S. grants awarded for orthopaedic research from 2010 to 2022. A total of 22,326 results were then manually screened to exclude those without a direct focus on orthopaedic research. The amounts received per principal investigator were reported in U.S. dollars and adjusted for inflation. Author gender was predicted with use of the Genderize.io algorithm application programming interface. The iCite Relative Citation Ratio (RCR) was utilized to assess the impact of the publications linked to each grant.

RESULTS

A total of 1,723 grants were included. Men principal investigators received significantly higher median funding per grant in 2011, 2012, and 2013; however, this trend reversed with women receiving nonsignificantly higher funding in 2015, 2017, 2018, 2021, and 2022. In 2020, women received significantly higher median funding per grant than men ($166,234 versus $121,384; p = 0.04). Throughout the 13-year period, men principal investigators accounted for approximately 71% of grants, with a very weak increasing trend in the percent of grants attributed to women (R 2 = 0.16; p < 0.001). Grants with men principal investigators resulted in more publications than those with women principal investigators (mean publications, 11.1 versus 6.6; p = 0.001). Publications resulting from grants awarded to men had a significantly higher mean RCR than those resulting from grants awarded to women (2.42 versus 2.09; p = 0.04).

CONCLUSIONS

There was no significant difference in the median amounts of funding per grant awarded to men and to women in 7 of the past 8 years, despite significantly greater funding per grant having been awarded to men from 2011 to 2013. Men principal investigators accounted for the majority of grants received during the study period, although this proportion was lower than the proportion of men among orthopaedic surgeons in 2022. This study could inform initiatives aimed at promoting equity in grant funding for orthopaedic research.

Team Physicians in Men's and Women's Professional Sports Leagues: Gender Representation and Career Path Analysis

Background

The most common orthopaedic fellowship is for sports medicine, but few fellowship-trained orthopaedic surgeons fill roles as team physicians. Gender disparities within the field of orthopaedics, coupled with male-dominated professional sports leagues in the United States, may lead to lower representation of women as professional team physicians.

Purpose

To (1) determine the career path trajectories of current head team physicians in professional sports, (2) quantify gender disparities across team physician representation, and (3) further characterize professional profiles of team physicians appointed to women's and men's professional sports leagues in the United States.

Study Design

Cross-sectional study.

Methods

This is a cross-sectional study of professional sports head team physicians in 8 major American sports leagues: American football (National Football League), baseball (Major League Baseball), basketball (National Basketball Association and Women's National Basketball Association), hockey (National Hockey League and National Women's Hockey League), and soccer (Major League Soccer and National Women's Soccer League). Online searches were used to compile information on gender, specialty, medical school, residency, fellowship, years in practice, clinical practice type, practice setting, and research productivity. Differences according to league type (men's vs women's leagues) were analyzed with the chi-square test for categorical variables, t test for continuous variables, and Mann-Whitney U test for nonparametric means. Bonferroni correction was applied for multiple comparisons.

Results

Within the 172 professional sports teams, 183 head team physicians were identified, including 170 men (92.9%) and 13 women (7.1%). Team physicians in both men's and women's sports leagues were predominantly men. Overall, 96.7% of team physicians in men's leagues were men, and 73.3% of team physicians in women's leagues were men (P < .001). The most common physician specialties were orthopaedic surgery (70.0%) and family medicine (19.1%). Compared with team physicians in women's leagues, those in men's leagues were more likely to be orthopaedic surgeons (40.0% vs 71.9%, respectively; P = .001) and to have more experience (15.9 vs 22.4 years, respectively; P < .001).

Conclusion

Study findings indicated disparities in gender, practice experience, and physician specialty representation among team physicians in men's versus women's professional sports leagues.

New Patient Referral Patterns May Reflect Gender Biases in Orthopedics

Background Orthopedic surgery traditionally has been a male-dominant specialty with the lowest percentage of female residents and female faculty of all medical specialties. Prior studies demonstrate gender biases from both referring providers and patients. This study investigates surgeon, referring provider, and patient demographic differences in new patient orthopedic referrals. Methodology A retrospective chart review was performed to analyze the demographics of new patients referred to male and female orthopedic surgeons within adult reconstruction and shoulder/elbow specialties at a single academic institution. Patients and referring provider demographics were compared for male and female orthopedic surgeons. Statistical analysis utilized Student's t-test and chi-square analyses for quantitative and qualitative data, respectively. Results In total, 2,642 new patients were analyzed, with 2,084 patients being referred from a provider, and 306 patients requesting specific providers. When compared to male surgeons, female surgeons had fewer referrals from male providers (45.3% vs. 50.3%, p = 0.03) and no difference from female providers (30.6% vs, 29.9%, p = 0.72). The female adult reconstruction surgeon had fewer internal referrals compared to a male surgeon of similar experience and time at the institution (8.4% vs. 12.8%, p = 0.03). Female patients requested male surgeons more frequently than female surgeons (76.7% vs. 23.3%, p = 0.02). Conclusions New patient demographics differed between male and female orthopedic surgeons at a single academic institution with more male referring providers referring to male surgeons. Female patients requesting male orthopedic providers may reflect patient and specialty-driven biases. There remains a need for additional female representation in orthopedic surgery, and new patient referral patterns may be a marker to assess and monitor gender biases.

Representation of women in orthopaedic surgery: perception of barriers among undergraduate medical students in Saudi Arabia

BACKGROUND

While female participation has improved in several surgical specialties over time globally, no such increase has been observed in orthopaedic surgery over the past decades. The potential barriers to female participation are likely present from the beginning of medical education. Therefore, this study assessed the apparent lag in equal representation among men and women in orthopaedic surgery in the Kingdom of Saudi Arabia.

METHODS

This cross-sectional study used a questionnaire survey to investigate medical students' and interns' perceptions of women participating in orthopaedic surgery, their subspeciality preferences, and barriers preventing them from pursuing an orthopaedic career. The responses were analysed to understand general perceptions, gender-based differences, impact of clinical experiential learning, and exposure to orthopaedic surgery.

RESULTS

Approximately 565 medical students (49% females, 51% males) participated in the survey. Only 17% of students (11% females, 23% males) considered orthopaedic surgery as their future career option. While 31% of female and 17% of male students disagreed with the concept of female-appropriate orthopaedic subspecialties, most of the remaining male and female students perceived paediatric orthopaedics as a female-appropriate subspecialty. Concerning equal representation of women, gender bias and lack of a strong physique were the most frequently selected barriers by female and male students, respectively. Patient preference for male orthopaedicians, gender discrimination, social and family commitments, and need for physical strength were all perceived as barriers for women in orthopaedics. Overall, clinical experience and orthopaedic exposure did not significantly improve the likelihood of female students in choosing orthopaedic surgery as a career.

CONCLUSIONS

The bias against women in orthopaedic careers is prevalent among medical students early in their academic years. Clinical experience and exposure to orthopaedic surgery should be improved to make a significant impact on female participation in orthopaedic careers. Career building efforts in terms of improved career opportunities, career counselling, flexible working hours, social and family related adjustments and implementation of mentorship/research/fellowship programmes for females are needed to reduce gender discrimination and improve female orthopaedic participation. Furthermore, process improvements may yield greater flexibility for women pursuing the challenging field while accommodating other barriers faced by women in orthopaedic surgery.

Is There Unconscious Bias in the Orthopaedic Residency Interview Selection Process?

OBJECTIVE

Orthopaedic surgery has historically been a white male-dominated field. Given the diverse patient population presenting to providers with musculoskeletal pathology, it is thought that it would be beneficial for the orthopaedic workforce to more closely mirror this patient population. This study aims to elucidate whether unconscious bias may have an effect on the scoring of applications for residency interview selection.

DESIGN

Applications for the 2019-2020 residency match cycle were initially reviewed and scored by faculty members. Applications were then redacted of all information suggestive of race or gender and returned to evaluators for rescoring after at least 6 months. The pre and post-redaction data was compared using ANOVA and student's two-tailed t tests.

SETTING

Department of Orthopaedic Surgery, Medical College of Georgia at Augusta University.

PARTICIPANTS

Thirteen attending surgeons scored 320 2019-2020 Electronic Residency Application System (ERAS) applications, unblinded and blinded of applicant identifying information.

RESULTS

Interviewed applicants were similar to the non-interviewed group in all measured variables except for higher pre-redaction scores (8.73-7.81; p = 0.02) which was expected (Table 2). Minority applicants had significant differences in Step 1 scores (243 vs 247; p < 0.01), Step 2 scores (251 vs 254; p = 0.01), articles (5.9 vs 3.8; p < 0.01), posters (5.9 vs 3.5; p < 0.01), and pre-redaction scores (7.44 vs 8.07; p = 0.01) compared to white applicants (Table 4). There was no relationship noted between step score and number or type of research items (Table 5). Pre-redaction and post-redaction scores were significantly different in white applicants who experienced a negative change (8.07-7.88; p = 0.03 (Table 6)). Males had statistically significant differences compared to females in Step 1 score (246 vs 243; p = 0.01) (Table 7).

CONCLUSIONS

This study was unable to prove unconscious bias based on a lack of statistically significant change of score when blinded, however the direction in change of scores was unlikely to be accounted for exclusively by objective differences between applicants, suggesting a trend toward unconscious bias. It remains unclear how influential subjective portions of the ERAS application such as personal statements, Letters of Recommendation, hobbies, and activities are on the overall assessment of an applicant and whether or not unconscious bias manifests in these subjective portions. Further investigation is needed in this area. Until then, residency programs should take immediate measures to mitigate potential implicit bias in the residency interview selection process. Actions can include implicit bias training for all faculty members involved in resident selection, standardization of application scoring and possibly redacting all or portions of the ERAS application so that only objective academic markers are presented to evaluators. Gaining a better understanding of these barriers is not only essential for their removal, but also allows for better preparation of applicants for success in the match with the ultimate goal being to correct the persistent disparity in the field of orthopaedic surgery.

Women in Surgery Events Alone do not Change Medical Student Perceptions of Gender Bias and Discrimination in Orthopaedic Surgery

Aims

This study investigated the perceptions of medical students regarding the barriers to pursuing a career in trauma and orthopaedics (T&O); and whether these perceptions were altered by attending an event promoting women in T&O.

Methods

An event consisting of presentations and interactive sessions from two female T&O trainees was hosted online. Attendees completed pre and post-event questionnaires. Students were asked about their previous exposure to T&O, perceptions of gender imbalances in T&O and what barriers they perceived prevented women from entering T&O. Univariate analysis was performed to identify changes in perceptions following the event.

Results

Pre-event questionnaires were completed by 102 people; and post-event by 52. Although 64/102 respondents were considering a career in T&O, 26/102 were dissuaded by perceived gender disparities. Perceptions of gender disparities were significantly higher in UK based attendees compared to other nationalities (p = 0.047). Attendees were more likely to want to pursue a career in T&O if they had been directly exposed at medical school (p = 0.044), but exposure did not alter perceptions of women in T&O. The most common perceived barrier was the orthopaedic stereotype followed by male dominated workplace culture, and lack of female role models. Pre and post-event responses did not differ significantly for any areas examined.

Conclusion

There are significant concerns amongst medical students regarding gender based discrimination within T&O, and these perceptions were not altered by attending a one-off women in T&O event. Early exposure to T&O appears important to improve interest in orthopaedics, whereas negative stereotyping is a barrier.

Representation of Women in Academic Orthopaedic Leadership: Where Are We Now?

BACKGROUND

Women have long been underrepresented in orthopaedic surgery; however, there is a lack of quantitative data on the representation of women in orthopaedic academic program leadership.

QUESTIONS/PURPOSES

(1) What is the proportion of women in leadership roles in orthopaedic surgery departments and residency programs in the United States (specifically, chairs, vice chairs, program directors, assistant program directors, and subspecialty division chiefs)? (2) How do women and men leaders compare in terms of years in position in those roles, years in practice, academic rank, research productivity as represented by publications, and subspecialty breakdown? (3) Is there a difference between men and women in the chair or program director role in terms of whether they are working in that role at institutions where they attended medical school or completed their residency or fellowship?

METHODS

We identified 161 academic orthopaedic residency programs from the Accreditation Council for Graduate Medical Education (ACGME) website. Data (gender, length of time in position, length of time in practice, professorship appointment, research productivity as indirectly measured via PubMed publications, and subspecialty) were collected for chairs, vice chairs, program directors, assistant program directors, and subspecialty division chiefs in July 2020 to control for changes in leadership. Information not provided by the ACGME and PubMed was found using orthopaedic program websites and the specific leader's curriculum vitae. Complete data were obtained for chairs and program directors, but there were missing data points for vice chairs, assistant program directors, and division chiefs. All statistical analysis was performed using SPSS using independent t-tests for continuous variables and the Pearson chi-square test for categorical variables, with p < 0.05 considered significant.

RESULTS

Three percent (4 of 153) of chairs, 8% (5 of 61) of vice chairs, 11% (18 of 161) of program directors, 27% (20 of 75) of assistant program directors, and 9% (45 of 514) of division chiefs were women. There were varying degrees of missing data points for vice chairs, assistant program directors, and division chiefs as not all programs reported or have those positions. Women chairs had fewer years in their position than men (2 ± 1 versus 9 ± 7 [95% confidence interval -9.3 to -5.9]; p < 0.001). Women vice chairs more commonly specialized in hand or tumor compared with men (40% [2 of 5] and 40% [2 of 5] versus 11% [6 of 56] and 4% [2 of 56], respectively; X2(9) = 16; p = 0.04). Women program directors more commonly specialized in tumor or hand compared with men (33% [6 of 18] and 17% [3 of 18] versus 6% [9 of 143] and 11% [16 of 143], respectively; X2(9) = 20; p = 0.02). Women assistant program directors had fewer years in practice (9 ± 4 years versus 14 ± 11 years [95% CI -10.5 to 1.6]; p = 0.045) and fewer publications (11 ± 7 versus 30 ± 48 [95% CI -32.9 to -5.8]; p = 0.01) than men. Women division chiefs had fewer years in practice and publications than men and were most prevalent in tumor and pediatrics (21% [10 of 48] and 16% [9 of 55], respectively) and least prevalent in spine and adult reconstruction (2% [1 of 60] and 1% [1 of 70], respectively) (X2(9) = 26; p = 0.001). Women program directors were more likely than men to stay at the same institution they studied at for medical school (39% [7 of 18] versus 14% [20 of 143]; odds ratio 3.9 [95% CI 1.4 to 11.3]; p = 0.02) and trained at for residency (61% [11 of 18] versus 42% [60 of 143]; OR 2.2 [95% CI 0.8 to 5.9]; p = 0.01).

CONCLUSION

The higher percentage of women in junior leadership positions in orthopaedic surgery, with the data available, is a promising finding. Hand, tumor, and pediatrics appear to be orthopaedic subspecialties with a higher percentage of women. However, more improvement is needed to achieve gender parity in orthopaedics overall, and more information is needed in terms of publicly available information on gender representation in orthopaedic leadership.

CLINICAL RELEVANCE

Proportional representation of women in orthopaedics is essential for quality musculoskeletal care, and proportional representation in leadership may help encourage women to apply to the specialty. Our findings suggest movement in an improving direction in this regard, though more progress is needed.

Bibliometric Analysis of the English Musculoskeletal Literature over the Last 30 Years

Publication and authorship are important in academia for career advancement, obtaining grants, and improved patient care. There has been a recent interest in bibliometric changes over time, especially regarding the gender gap. The purpose of this study was to explore bibliometric changes in the musculoskeletal literature. Bibliometric variables (number of authors, institutions, countries, pages, references, corresponding author position, author gender, geographic region of origin, and editorial board makeup) were analyzed for 5 basic science and 12 clinically oriented musculoskeletal journals from 1985 through 2016. Statistical analyses comprised bivariate analyses, multifactorial ANOVAs, and logistic regression analyses. A p < 0.005 was considered significant. Nearly, all variables increased over time. Asia had the highest number of authors and corresponding author positions, Australia/New Zealand the highest number of institutions and references, North America the highest number of pages, and Europe the highest number of countries. Those with a female first author had more authors, institutions, countries, references, and pages. Likewise, those with a female corresponding author had more authors, institutions, countries, references, and pages. Single-authored manuscripts decreased over time. The percentage of female first authors rose from 10.8% in 1985-1987 to 23.7% in 2015-2016. There were more female 1^st authors in the basic science journals compared to the clinical journals (33.2% vs. 12.7%). Single-authored manuscripts were more likely to be written by males (5.1 vs. 2.4%) and decreased over time. The many differences by geographic region of origin likely reflect different socio/cultural attitudes regarding academia and research, as well as the gender composition of the disciplines by geographic region. Overall, there has been an increase in the number of female 1^st and corresponding authors, editorial board members, and chief editors, indicating a slow but progressive narrowing of the gender gap.

Gender, Race, Age, Allopathic Degree, Board Score, and Research Experience Among Applicants Matching to General and Orthopedic Surgery Residencies, 2015-2019

BACKGROUND

Surgical fields are historically dominated by male physicians. Increasing the diversity of the physician workforce improves training and patient experiences. We aim to investigate any differences in qualifications and match rates between male and female applicants to general surgery (GS) and orthopedic surgery (OS) residencies in the United States.

METHODS

A retrospective cohort analysis was performed utilizing the Association of American Medical Colleges data regarding Electronic Residency Application Service (ERAS) applicants and matched Accreditation Council for Graduate Medical Education (ACGME) residents into GS and OS residencies from 2015 to 2019. Descriptive statistics and independent sample T-tests were performed with significance defined as P < .05.

RESULTS

26 568 GS and 7076 OS ERAS applicants matched at a rate of 25.2% and 55.3%, respectively. Men and women matched into GS at rates of 23.0% and 29.2%, respectively. Men and women matched into OS at rates of 55.2% and 56.2%, respectively. Men aged ≥36 years matched into OS at a significantly higher rate than women aged years ≥36 (11.9% vs. 1.4%, P = .009). Female GS ERAS applicants and entering ACGME residents had a higher mean number of research experiences than male GS ERAS applicants (2.66 vs. 2.26, P < .001) and entering male GS ACGME residents (2.96 vs. 2.56, P = .008).

CONCLUSIONS

Male and female GS and OS applicants have similar qualifications. Women match into GS and OS at higher rates than men but comprise disproportionately lower numbers of applicants. Greater mentorship opportunities and recruitment of female applicants are needed to expand, diversify, and increase representation of women in surgery.

When (Almost) Everyone is Above Average: A Critical Analysis of American Orthopaedic Association Committee of Residency Directors Standardized Letters of Recommendation

The American Orthopaedic Association introduced standardized letters of recommendations (SLORs) to improve on traditional letters of recommendations by “providing a global prospective on an applicant.” However, no study has defined the utilization of SLORs, the distribution of applicant ratings in SLORs, or the impact of sex, race, or degree of involvement between the letter writer and applicant on SLOR domain ratings.