A critical evaluation of the algorithm behind the Relative Citation Ratio (RCR)

Factors Associated with Research Productivity and National Institutes of Health Funding in Academic Otology

OBJECTIVES

Bibliometrics, such as the Hirsch index (h-index) and the more recently developed relative citation ratio (RCR), are utilized to evaluate research productivity. Our study evaluates demographics, research productivity, and National Institutes of Health (NIH) funding in academic otology.

METHODS

Academic otologists were identified, and their demographics were collected using institutional faculty profiles (N = 265). Funding data were obtained using the NIH Research Portfolio Online Reporting Tools Expenditures and Reports Database. The h-index was calculated using Scopus and mean (m-RCR) and weighted RCR (w-RCR) were calculated using the NIH iCite tool.

RESULTS

H-index (aOR 1.18, 95% CI 1.10-1.27, p < 0.001), but not m-RCR (aOR 1.50, 95% CI 0.97-2.31, p = 0.069) or w-RCR (aOR 1.00, 95% CI 0.99-1.00, p = 0.231), was associated with receiving NIH funding. Men had greater h-index (16 vs. 9, p < 0.001) and w-RCR (51.8 vs. 23.0, p < 0.001), but not m-RCR (1.3 vs. 1.3, p = 0.269) than women. Higher academic rank was associated with greater h-index and w-RCR (p < 0.001). Among assistant professors, men had greater h-index than women (9.0 vs. 8.0, p = 0.025). At career duration 11-20 years, men had greater h-index (14.0 vs. 8.0, p = 0.009) and w-RCR (52.7 vs. 25.8, p = 0.022) than women.

CONCLUSION

The h-index has a strong relationship with NIH funding in academic otology. Similar h-index, m-RCR, and w-RCR between men and women across most academic ranks and career durations suggests production of similarly impactful research. The m-RCR may correct some deficiencies of time-dependent bibliometrics and its consideration in academic promotion and research funding allocation may promote representation of women in otology.

LEVEL OF EVIDENCE

N/A Laryngoscope, 134:3786-3794, 2024.

Examination of the Novel National Institutes of Health-Supported Relative Citation Ratio, a Measure of Research Productivity, Among Academic Plastic Surgeons

BACKGROUND

Considerable interest has been devoted to quantifying research productivity for the purposes of academic appointment and promotion in plastic surgery. A novel bibliometric, the relative citation ratio (RCR), integrates features unavailable in prior metrics, such as the h-index, including the ability to compare researchers in distinct fields. This investigation examines the RCR in relation to established measures of academic productivity and provides the benchmark data in plastic surgery.

METHODS

Online sources were queried to identify the characteristics of 955 academic plastic surgeons from 94 programs, ie, academic rank, gender, degrees, and fellowships. Bibliometric data were acquired using the iCite and Scopus databases. Comparative and correlational analyses of variables were performed.

RESULTS

Academic plastic surgeons were exceptionally productive, with a mean RCR of 1.20 (interquartile range, 0.79-1.67) and a weighted RCR of 17.68 (interquartile range, 5.14-52.48). Increased mean RCR was significantly associated with advanced academic rank and fellowship training. Increased weighted RCR was significantly associated with advanced academic rank, male gender, PhD acquisition, publication experience, and fellowship training. The h-index was weakly correlated with mean RCR but strongly correlated with weighted RCR and publication experience.

CONCLUSIONS

The RCR was associated with established markers of academic productivity, indicating its validity as a reliable field-normalized measure for the evaluation of plastic surgery faculty. Because scholarly output is a potential factor with respect to decisions of hiring, promotion, and allocation of funding, this modality of standardized comparison is paramount for plastic surgeons who exist within a larger general surgery department.

Association of literature metrics in gynecologic oncology with country classification by income level

Objective

Bibliometric literature in gynecologic oncology is limited. We aimed to study the association between the level of income of the country of authorship and citation metrics.

Methods

A retrospective study including all articles and reviews published during 1977–2022 in theInternational Journal of Gynecological Cancer(IJGC) andGynecologic Oncologyjournals. Country of origin was defined as the corresponding author’s address. We classified articles into groups by level of income of the country of origin, as defined by the World Bank. The primary outcome measure was the median number of citations per year.

Results

A total of 9835 articles were included in the analysis (IJGCn=3786 (38.5%),Gynecologic Oncologyn=6049 (61.5%)). There were 8587 (87.3%) publications from high income countries, 1134 (11.5%) from upper-middle income countries, and 114 (1.2%) from lower-middle income countries. There were no publications from countries of low income. Most publications originated in the United States with 4089 (41.6%), followed by China (n=730, 7.4%), Italy (n=533, 5.4%), Canada (n=467, 4.7%), and Japan (n=461, 4.7%). Over the most recent 5 years there was a decrease in the representation of upper-middle income countries and lower-middle income countries; 16.3% (91/557) in 2018 versus 9.1% (38/417) in 2022 (p=0.005). In a multivariable regression analysis that included year of publication, open access publication model, study being supported by funding, publishing journal, review article, and level of income, all factors were associated with high citation per year score except the income classification of the article’s country of origin (adjusted OR 1.59–1.72, 95% CI 0.61 to 4.30).

Conclusion

High income countries have a disproportionate representation in gynecologic oncology publications. After adjusting for confounders, the country’s level of income was not independently associated with a high citation per year score. This implies that the number of citations per year is not compromised by the country’s level of income.

Current Trends in Subspecialty Fellowship Training for 1691 Academic Neurological Surgeons

OBJECTIVE

Neurosurgical subspecialty fellowship training has become increasingly popular in recent decades. However, few studies have evaluated recent trends in postgraduate subspecialty education. This study aims to provide a detailed cross-sectional analysis of subspecialty fellowship training completion trends and demographics among U.S. academic neurosurgeons.

METHODS

Academic clinical faculty (M.D. or D.O.) teaching at accredited neurosurgery programs were included. Demographic, career, and fellowship data were collected from departmental physician profiles and the American Association of Neurological Surgeons (AANS) membership database. Relative citation ratio scores were retrieved using the National Institutes of Health iCite tool.

RESULTS

This study included 1691 surgeons (1756 fellowships) from 125 institutions. The majority (79.13%) reported fellowship training. Fellowship completion was more common among recent graduates (residency year >2000), as was training in multiple subspecialties (P < 0.0001). Spine was the most popular subspecialty (16.04%), followed by pediatrics (11.18%), and cerebrovascular (9.46%). The least common were trauma/critical care (2.52%) and peripheral nerve (1.26%). Spine, neuroradiology, and endovascular subspecialties grew in popularity over time. Pediatrics and spine were the most popular for females and males, respectively. Epilepsy and cerebrovascular had the most full professors, while endovascular and spine had the most assistant professors. Stereotactic/functional and epilepsy had the most Ph.Ds. Fellowship training correlated with higher weighted, but not mean, relative citation ratio scores among associate (P = 0.002) and full professors (P = 0.005).

CONCLUSIONS

There is an emerging proclivity for additional fellowship training among young neurosurgeons, often in multiple subspecialties. These findings are intended to help guide professional decision-making and optimize the delivery of postgraduate education.

Evaluation of the National Institutes of Health-Supported Relative Citation Ratio Among Fellowship-Trained American Orthopaedic Joint Reconstruction Surgery Faculty: A New Bibliometric Measure of Scientific Influence

BACKGROUND

The relative citation ratio (RCR), a novel National Institutes of Health-Supported measure of research productivity, allows for accurate interdisciplinary comparison of publication influence. This study evaluates the RCR of fellowship-trained adult reconstructive orthopaedic surgeons with the goal of analyzing potentially influential physician demographics.

METHODS

Adult Reconstruction Accreditation Council for Graduate Medical Education fellowship-trained faculty for orthopaedic residency programs were identified via departmental websites. The National Institutes of Health's iCite database was retrospectively reviewed for mean RCR, weighted RCR, and publication count by surgeon. Multivariate analyses were performed using the Wilcoxon rank-sum tests and analyses of variance testing to compare sex, career length, academic rank, and professional degrees in addition to an MD or DO. Significance was considered P < .05.

RESULTS

A total of 488 fellowship-trained adult reconstruction faculty from 144 programs were included in the analysis. Overall, the faculty recorded a median RCR of 1.65 (interquartile range: 1.01-2.28) and a median weighted RCR of 16.59 (interquartile range: 3.98-61.92). The weighted RCR and total number of publications were associated with academic rank and career longevity, while the mean RCR was associated with academic rank. The median RCR ranged from 1.12 to 1.87 for all subgroups.

CONCLUSION

Adult reconstruction faculty are exceptionally productive and generate highly impactful studies as evidenced by the high median RCR value relative to the National Institutes of Health standard value of 1.0. Our data have important implications in the assessment of grant outcomes, promotion, and continued evaluation of research influence within the hip and knee community.

The relative citation ratio (RCR) as a novel bibliometric among 2511 academic orthopedic surgeons

Objectively measuring research output is important for grant awards, promotion, and tenure, or self-evaluation of productivity. However, certain shortcomings limit common bibliometric indicators. The time- and field-independent relative citation ratio (RCR) was proposed to overcome these limitations. The objective of this study was to determine whether the RCR correlates with academic rank, gender, and PhD degree status among US academic orthopedic surgeons. Full-time faculty surgeons at Accreditation Council for Graduate Medical Education-accredited orthopedic surgery residency programs were included in this study. Mean (mRCR) and weighted (wRCR) RCR scores were collected from the National Institutes of Health iCite database to quantify scholarly "impact" and "production," respectively, and were compared by academic rank, gender, and PhD status. All information was collected from publicly available faculty listings on departmental websites. A total of 2511 orthopedic surgeons from 132 residency programs were assessed. Overall, the median (interquartile range) mRCR score was 1.56 (1.05-2.12) and the median wRCR score was 27.6 (6.97-88.44). Both metrics increased with each successive academic rank, except for department chairs. There was no difference in mRCR between male and female surgeons. Among assistant professors, males had higher wRCR scores. Both metrics were higher among surgeons with a PhD degree. The RCR offers key advantages over other indices, which are reflected in differences in score distributions compared with the widely used h-index. Nevertheless, implementation of the RCR should be preceded with careful consideration of its own limitations.

Co-citation Percentile Rank and JYUcite: a new network-standardized output-level citation influence metric and its implementation using Dimensions API

Judging value of scholarly outputs quantitatively remains a difficult but unavoidable challenge. Most of the proposed solutions suffer from three fundamental shortcomings: they involve (i) the concept of journal, in one way or another, (ii) calculating arithmetic averages from extremely skewed distributions, and (iii) binning data by calendar year. Here, we introduce a new metric Co-citation Percentile Rank (CPR), that relates the current citation rate of the target output taken at resolution of days since first citable, to the distribution of current citation rates of outputs in its co-citation set, as its percentile rank in that set. We explore some of its properties with an example dataset of all scholarly outputs from University of Jyväskylä spanning multiple years and disciplines. We also demonstrate how CPR can be efficiently implemented with Dimensions database API, and provide a publicly available web resource JYUcite, allowing anyone to retrieve CPR value for any output that has a DOI and is indexed in the Dimensions database. Finally, we discuss how CPR remedies failures of the Relative Citation Ratio (RCR), and remaining issues in situations where CPR too could potentially lead to biased judgement of value.

A bibliometric and visual analysis of obesity and polycystic ovary syndrome from 2012 to 2022

BACKGROUND

Obesity is associated with polycystic ovary syndrome (PCOS). We aimed to elucidate the research status and explore research trends and future directions of research on obesity and PCOS.

METHODS

A bibliometric analysis of the published papers in the field of obesity and PCOS between 2012 and 2022 was conducted on the basis of the Web of Science Core Collection database. The collaboration networks, research trends, literature sources, citation analysis, co-citation analysis, and keywords analysis were statistically analyzed and visualized using the VOSviewer software.

RESULTS

We retrieved 2843 records from 681 journals by 12307 authors from 2942 institutes in 99 countries. The number of published papers and citations had a roughly increasing trend annually. The United States and China contributed the majority of the records. Monash University, Shanghai Jiaotong University, Aristotle University of Thessaloniki, Karolinska Institute, University of São Paulo, and Tehran University of Medical Sciences were the biggest nodes in their cluster of the collaboration network map, and Moran LJ, Teede HJ, Joham AE, Escobar-Morreale HF, and Macut D were prolific authors. Research trends and hotspots were identified and visualized in the field of obesity and PCOS. Research hotspots in this field focused on insulin resistance (IR), metabolic syndrome, metformin, and inflammation. Bariatric surgery, mitochondrial dysfunction, binding globulins, and comorbidities may be the frontiers of future research.

CONCLUSIONS

We concluded the research status and trends in the field of obesity and PCOS. A better understanding of collaboration patterns, research hotspots, and frontiers may be useful for researchers.

Structure, function, and productivity from the National Dental Practice-Based Research Network

Introduction:

Following inception in 2005 as a multiregional practice-based research network (PBRN), the “National Dental PBRN” expanded nationwide in 2012, and in 2019 implemented additional organizational changes. The objectives are to: (1) describe the new structure and function of the network; and (2) quantify its scientific productivity since 2005.

Methods:

A national Administrative and Resource Center is based in Alabama; regional and specialty nodes are based in Alabama, Florida, Illinois, Minnesota, Oregon, New York, and Texas. A Network Coordinating Center is based in Oregon. Studies are funded via investigator-initiated grants. Scientific productivity is assessed using specific metrics, including the Relative Citation Ratio.

Results:

To date, 58 studies have been completed or are in data collection or development. These studies have investigated a broad range of topics using a wide variety of study designs. Of the studies that have completed enrollment, 70,665 patients were enrolled, as were 19,827 practitioners (some participated in multiple studies), plus electronic records for 790,493 patients in two data-only studies. To date, these studies have led to 193 peer-reviewed scientific publications in 62 different journals. The mean (1.40) Relative Citation Ratio of Network publications connotes a greater-than-average influence in their fields.

Conclusions:

These metrics demonstrate that the PBRN research context can successfully engage practitioners and patients from diverse settings nationally with a high and sustained level of scientific productivity. This infrastructure has enabled clinical scientists in oral health and nonoral health topics and provided additional recruitment venues outside of the typical academic health center research context.

Requiem for impact factors and high publication charges

Journal impact factors, publication charges and assessment of quality and accuracy of scientific research are critical for researchers, managers, funders, policy makers, and society. Editors and publishers compete for impact factor rankings, to demonstrate how important their journals are, and researchers strive to publish in perceived top journals, despite high publication and access charges. This raises questions of how top journals are identified, whether assessments of impacts are accurate and whether high publication charges borne by the research community are justified, bearing in mind that they also collectively provide free peer-review to the publishers. Although traditional journals accelerated peer review and publication during the COVID-19 pandemic, preprint servers made a greater impact with over 30,000 open access articles becoming available and accelerating a trend already seen in other fields of research. We review and comment on the advantages and disadvantages of a range of assessment methods and the way in which they are used by researchers, managers, employers and publishers. We argue that new approaches to assessment are required to provide a realistic and comprehensive measure of the value of research and journals and we support open access publishing at a modest, affordable price to benefit research producers and consumers.

Use of the journal impact factor for assessing individual articles: Statistically flawed or not?

Most scientometricians reject the use of the journal impact factor for assessing individual articles and their authors. The well-known San Francisco Declaration on Research Assessment also strongly objects against this way of using the impact factor. Arguments against the use of the impact factor at the level of individual articles are often based on statistical considerations. The skewness of journal citation distributions typically plays a central role in these arguments. We present a theoretical analysis of statistical arguments against the use of the impact factor at the level of individual articles. Our analysis shows that these arguments do not support the conclusion that the impact factor should not be used for assessing individual articles. Using computer simulations, we demonstrate that under certain conditions the number of citations an article has received is a more accurate indicator of the value of the article than the impact factor. However, under other conditions, the impact factor is a more accurate indicator. It is important to critically discuss the dominant role of the impact factor in research evaluations, but the discussion should not be based on misplaced statistical arguments. Instead, the primary focus should be on the socio-technical implications of the use of the impact factor.

Interdisciplinarity Metric Based on the Co-Citation Network

Quantifying the interdisciplinarity of a research is a relevant problem in the evaluative bibliometrics. The concept of interdisciplinarity is ambiguous and multidimensional. Thus, different measures of interdisciplinarity have been propose in the literature. However, few studies have proposed interdisciplinary metrics without previously defining classification sets, and no one has used the co-citation network for this purpose. In this study we propose an interdisciplinary metric based on the co-citation network. This is a way to define the publication’s field without resorting to pre-defined classification sets. We present a characterization of a publication’s field and then we use this definition to propose a new metric of the interdisciplinarity degree for publications (papers) and journals as units of analysis. The proposed measure has an aggregative property that makes it scalable from a paper individually to a set of them (journal) without more than adding the numerators and denominators in the proportions that define this new indicator. Moreover, the aggregated value of two or more units is strictly among all the individual values.

Gender disparities are present in academic rank and leadership positions despite overall equivalence in research productivity indices among senior members of American Head and Neck Society (AHNS) Fellowship Faculty

BACKGROUND

This study aims to examine potential disparities in scholarly performance based on sex, academic rank, leadership positions, and regional distribution of faculty in accredited Head and Neck Surgery fellowships in the United States.

METHODS

Online faculty listings for 37 accredited fellowships were organized according to academic rank, leadership position, sex, and institutional location. Academic productivity was measured with three bibliometric indices: h-index, m-index, and the weighted relative citation ratio.

RESULTS

A total of 732 faculty members were included, of which 153 (21%) were female. Fifty-eight males (89.2%) held leadership positions, compared to seven females (10.8%). There was no significant difference in overall productivity between male and female senior faculty. There were regional differences in productivity by sex.

CONCLUSIONS

Females are underrepresented in senior faculty and within three common leadership positions, although scholarly productivity for male and female senior faculty and for those in leadership positions is similar.

The relative citation ratio: what is it and why should medical librarians care?

Bibliometrics is becoming increasingly prominent in the world of medical libraries. The number of presentations related to research impact at the Medical Library Association (MLA) annual meeting has been increasing in past years. Medical centers have been using institutional dashboards to track clinical performance for over a decade, and more recently, these institutional dashboards have included measures of academic performance. This commentary reviews current practices and considers the role for a newer metric, the relative citation ratio.

Scientific productivity: An exploratory study of metrics and incentives

Competitive pressure to maximize the current bibliometric measures of productivity is jeopardizing the integrity of the scientific literature. Efforts are underway to address the ‘reproducibility crisis’ by encouraging the use of more rigorous, confirmatory methods. However, as long as productivity continues to be defined by the number of discoveries scientists publish, the impact factor of the journals they publish in and the number of times their papers are cited, they will be reluctant to accept high quality methods and consistently conduct and publish confirmatory/replication studies. This exploratory study examined a sample of rigorous Phase II-IV clinical trials, including unpublished studies, to determine if more appropriate metrics and incentives can be developed. The results suggest that rigorous procedures will help reduce false positives, but to the extent that higher quality methods are accepted as the standard of practice, the current bibliometric incentives will discourage innovative studies and encourage scientists to shift their research to less informative studies of subjects that are already being more actively investigated. However, the results also suggest that it is possible to develop a more appropriate system of rewards. In contrast to the current bibliometric incentives, evaluations of the quality of the methods and reproducibility of the results, innovation and diversity of thought, and amount of information produced may serve as measures and incentives that maintain the integrity of the scientific literature and maximize scientific progress.

Assessing scientists for hiring, promotion, and tenure

Assessment of researchers is necessary for decisions of hiring, promotion, and tenure. A burgeoning number of scientific leaders believe the current system of faculty incentives and rewards is misaligned with the needs of society and disconnected from the evidence about the causes of the reproducibility crisis and suboptimal quality of the scientific publication record. To address this issue, particularly for the clinical and life sciences, we convened a 22-member expert panel workshop in Washington, DC, in January 2017. Twenty-two academic leaders, funders, and scientists participated in the meeting. As background for the meeting, we completed a selective literature review of 22 key documents critiquing the current incentive system. From each document, we extracted how the authors perceived the problems of assessing science and scientists, the unintended consequences of maintaining the status quo for assessing scientists, and details of their proposed solutions. The resulting table was used as a seed for participant discussion. This resulted in six principles for assessing scientists and associated research and policy implications. We hope the content of this paper will serve as a basis for establishing best practices and redesigning the current approaches to assessing scientists by the many players involved in that process.