The relative citation ratio: what is it and why should medical librarians care?

Evaluation of Research Productivity among Academic Glaucoma Specialists using the Relative Citation Ratio

PURPOSE

To provide relative citation ratio (RCR) benchmark data for the field of glaucoma.

DESIGN

Cross-sectional bibliometric analysis.

SUBJECTS

Fellowship-trained glaucoma faculty at ACGME-accredited institutions.

METHODS

Glaucoma faculty were individually indexed using the NIH iCite website. Publication count, mean RCR score, and weighted RCR score were collected for each author between May and August 2023 and included PubMed-listed articles from 1980 to 2023. Data were compared by gender, career duration, academic rank, and acquisition of a Doctor of Philosophy (PhD).

MAIN OUTCOME MEASURES

Total number of publications, mean RCR value, weighted RCR value.

RESULTS

526 academic glaucoma specialists from 113 institutions were indexed. These physicians produced highly impactful research with a median publication count of 13 (IQR 4-38), median RCR of 1.41 (IQR 0.97-1.98), and median weighted RCR of 16.89 (4.80-63.39). Academic rank, career duration, and having a PhD were associated with increased publication count, mean RCR, and weighted RCR. Publication count and weighted RCR differed significantly by gender, however, no difference was observed with mean RCR.

CONCLUSIONS

Current academic glaucoma specialists have high mean RCR values relative to the NIH standard RCR value of 1. This benchmark data serves as a more accurate gauge of research impact within the glaucoma community and can be used to inform self, institutional, and departmental evaluations. Additionally, the mean RCR may provide an accurate metric for quantifying research productivity among historically underrepresented groups that are disadvantaged by time-dependent factors such as number of publications.

Evaluation of Research Productivity Among Academic Cornea, External Diseases, and Refractive Surgery Ophthalmologists Using the Relative Citation Ratio

PURPOSE

The purpose of this study was to provide relative citation ratio (RCR) benchmark data for cornea and external diseases specialists.

DESIGN

This is a cross-sectional bibliometric analysis.

SUBJECTS

Subjects included were fellowship-trained cornea and external diseases faculty at Accreditation Council for Graduate Medical Education-accredited institutions in the United States.

METHODS

Academic specialists were indexed using the National Institutes of Health iCite Web site. Publication count, mean RCR score, and weighted RCR score were obtained between October 2022 and January 2023 by examining PubMed-listed publications from 1980 to 2022. Data were compared by sex, career duration, academic rank, and acquisition of a Doctor of Philosophy.

MAIN OUTCOME MEASURES

The main outcome measures were publication count, mean RCR value, and weighted RCR value.

RESULTS

The cohort included 602 specialists from 112 Accreditation Council for Graduate Medical Education-accredited institutions. These clinician-scientists produced highly impactful research with a median publication count of 15 (interquartile ranges 4-41), median RCR of 1.4 (interquartile ranges 0.91-1.88), and median-weighted RCR of 20.28 (5.3-66.69). Both academic rank and career length were associated with greater publication count and RCR values. Male sex was also associated with greater publications counts and RCR scores compared with female faculty. Acquisition of a Doctor of Philosophy was associated with greater publication counts and weighted RCR scores but no difference in mean RCR scores.

CONCLUSIONS

Academic cornea and external diseases specialists conduct high-impact research, with a median RCR of 1.4, exceeding the NIH standard value of 1. These data provide RCR benchmark data for the field to inform self, institutional, and departmental evaluations. These results also highlight a significant gender disparity in the field necessitating efforts to increase female representation and ensure equal opportunities.

Research Productivity and Impact in Foot and Ankle Surgery: Insights From Relative Citation Ratio Analysis of Recent Fellowship Graduates

INTRODUCTION

The relative citation ratio (RCR), a novel bibliometric tool supported by the National Institute of Health, provides a standardized approach to evaluate research productivity and impact across different fields. This study aims to evaluate RCR of fellowship-trained foot and ankle orthopaedic surgeons to analyze the influence of various surgeon demographics.

METHODS

Fellow names listed on the American Orthopaedic Foot and Ankle Society website were extracted from the year 2008 to 2009 to the year 2022 to 2023. Demographic information for each fellow was collected including sex, degree type, and academic title. The iCite database developed by the National Institute of Health was used to obtain total publications, mean RCR, weighted RCR, and change in RCR after fellowship graduation for each fellow. Univariate and multivariate analysis was conducted to predict these four parameters based on sex, degree type, academic position, and career longevity.

RESULTS

Of the 820 fellows, 674 (82%) were male. Most fellows (n = 587, 71%) did not go on to hold academic positions. Multivariate analysis revealed that male sex (β = 2.32, P < 0.001), holding an academic position (β = 6.44, P < 0.001), holding a PhD (β = 22.96, P < 0.001), and a shorter length time since graduation (β = -0.50, P < 0.001) were independent predictors of number of total publications. Holding a DO degree was an independent predictor of decreased mean RCR (β = 0.39, P = 0.039). Finally, multivariate analysis revealed that male sex (β = 4.05, P = 0.003), a career in academics (β = 4.61, P < 0.001), and a shorter time since graduation (β = -0.45, P = 0.001) were associated with a larger weighted RCR.

DISCUSSION

The findings highlight the importance of addressing gender disparities and promoting research opportunities across different programs. Moreover, academic institutions should provide adequate support and mentorship to early-career foot and ankle-trained orthopaedic surgeons to foster sustained research productivity.

Recognizing the value of meta-research and making it easier to find

Meta-research is a bourgeoning field studying topics with significant relevance to health sciences librarianship, such as research reproducibility, peer review, and open access. As a discipline that studies research itself and the practices of researchers, meta-research spans disciplines and encompasses a broad spectrum of topics and methods. The breadth of meta-research presents a significant challenge for identifying published meta-research studies. Introducing a subject heading for meta-research in the controlled vocabularies of literature databases has the potential to increase the visibility of meta-research, further advance the field, and expand its impact on research practices. Given the relatively recent designation of meta-research as a field and its expanding use as a term, now is the time to develop appropriate indexing vocabulary. We seek to call attention to the value of meta-research for health sciences librarianship, describe the challenges of identifying meta-research literature with currently available key terms, and highlight the need to establish controlled vocabulary specific to meta-research.

The positive impact on translational research of Fondazione italiana di ricerca per la Sclerosi Laterale Amiotrofica (AriSLA), a non-profit foundation focused on amyotrophic lateral sclerosis. Convergence of ex-ante evaluation and ex-post outcomes when goals are set upfront

Charities investing on rare disease research greatly contribute to generate ground-breaking knowledge with the clear goal of finding a cure for their condition of interest. Although the amount of their investments may be relatively small compared to major funders, the advocacy groups' clear mission promotes innovative research and aggregates highly motivated and mission-oriented scientists. Here, we illustrate the case of Fondazione italiana di ricerca per la Sclerosi Laterale Amiotrofica (AriSLA), the main Italian funding agency entirely dedicated to amyotrophic lateral sclerosis research. An international benchmark analysis of publications derived from AriSLA-funded projects indicated that their mean relative citation ratio values (iCite dashboard, National Institutes of Health, U.S.) were very high, suggesting a strong influence on the referring international scientific community. An interesting trend of research toward translation based on the "triangle of biomedicine" and paper citations (iCite) was also observed. Qualitative analysis on researchers' accomplishments was convergent with the bibliometric data, indicating a high level of performance of several working groups, lines of research that speak of progression toward clinical translation, and one study that has progressed from the investigation of cellular mechanisms to a Phase 2 international clinical trial. The key elements of the success of the AriSLA investment lie in: (i) the clear definition of the objectives (research with potential impact on patients, no matter how far), (ii) a rigorous peer-review process entrusted to an international panel of experts, (iii) diversification of the portfolio with ad hoc selection criteria, which also contributed to bringing new experts and younger scientists to the field, and (iv) a close interaction of AriSLA stakeholders with scientists, who developed a strong sense of belonging. Periodic review of the portfolio of investments is a vital practice for funding agencies. Sharing information between funding agencies about their own policies and research assessment methods and outcomes help guide the international debate on funding strategies and research directions to be undertaken, particularly in the field of rare diseases, where synergy is a relevant enabling factor.

Use of the Relative Citation Ratio in Conjunction With H-Index to Promote Equity in Academic Orthopaedics

INTRODUCTION

Quantification of a researcher's productivity relies on objective bibliometric measurements, such as the Hirsch index (h-index). However, h-index is not field and time-normalized and possesses bias against newer researchers. Our study is the first to compare the relative citation ratio (RCR), a new article-level metric developed by the National Institutes of Health, with h-index in academic orthopaedics.

METHODS

Academic orthopaedic programs in the United States were identified using the 2022 Fellowship and Residency Electronic Interactive Database. Available demographic and training data for surgeons were collected. RCR was calculated using the National Institutes of Health iCite tool, and h-index was calculated using Scopus.

RESULTS

Two thousand eight hundred twelve academic orthopaedic surgeons were identified from 131 residency programs. H-index, weighted RCR (w-RCR), and mean RCR (m-RCR) all significantly differed by faculty rank and career duration. However, while h-index and w-RCR varied between sexes (P < 0.001), m-RCR did not (P = 0.066), despite men having a longer career duration (P < 0.001).

DISCUSSION

We propose that m-RCR be used in conjunction with w-RCR or h-index to promote a fairer, comprehensive depiction of an orthopaedic surgeon's academic effect and productivity. Use of m-RCR may reduce the historic bias against women and younger surgeons in orthopaedics, which has implications in employment, promotion, and tenure.

Impact of National Institutes of Health and Food and Drug Administration Tobacco Research Funding: A Bibliometrics Analyses

INTRODUCTION

Conduct bibliometric analyses documenting the output of National Institutes of Health (NIH) tobacco-related and Food and Drug Administration (FDA) tobacco regulatory science (FDA-TRS) research portfolios.

AIMS AND METHODS

PubMed identifiers for publications between 2015 and 2020 citing tobacco funding by NIH and/or FDA were imported into NIH iCite generating measures of productivity and influence, including number of citations, journal, relative citation ratios (RCR), and comparison of research influence across Web of Science (WoS) disciplines. Coauthorship and measures of centrality among and between NIH and FDA-supported investigators gauged collaboration.

RESULTS

Between FY 2015 and 2020, 8160 publications cited funding from NIH tobacco-related grants, 1776 cited FDA-TRS grants and 496 cited Common funding (ie, both NIH and FDA-TRS funding). The proportion of publications citing NIH grants declined while those citing FDA-TRS or Common funding rose significantly. Publications citing Common funding showed the highest influence (mean RCR = 2.52). Publications citing FDA-TRS funding displayed higher median RCRs than publications citing NIH funding in most WoS categories. Higher translational progress was estimated over time for FDA-TRS and Common publications compared to NIH publications. Authors citing Common funding scored highest across all collaboration measures.

CONCLUSIONS

This study demonstrates the high bibliometric output of tobacco research overall. The rise in publications citing FDA-TRS and Common likely reflects increased funding for TRS research. Higher RCRs across WoS subject categories and trends towards human translation among FDA-TRS and Common publications indicate focus on research to inform regulation. This analysis suggests that FDA support for TRS has expanded the field of tobacco control resulting in sustained productivity, influence, and collaboration.

IMPLICATIONS

This paper is the first effort to better describe the impact of tobacco research resulting from the addition of FDA funding for TRS in the past decade. The analysis provides impetus for further investigation into the publication topics and their focus which would offer insight into the specific evidence generated on tobacco control and regulation.

Evaluation of Research Productivity among Academic Vitreoretinal Surgeons Using the Relative Citation Ratio

PURPOSE

To provide relative citation ratio (RCR) benchmark data for the field of vitreoretinal surgery.

DESIGN

Cross-sectional bibliometric analysis.

SUBJECTS

Fellowship-trained vitreoretinal faculty at Accreditation Council for Graduate Medical Education-accredited institutions.

METHODS

Academic vitreoretinal surgeons were individually indexed using the National Institutes of Health iCite Website. Publication count, mean RCR score, and weighted RCR score were collected for each author between June and July 2022 and included PubMed-listed articles from 1980 to 2022. Data were compared by gender, career duration, academic rank, and acquisition of a Doctor of Philosophy (PhD).

MAIN OUTCOME MEASURES

Total number of publications, mean RCR value, and weighted RCR value.

RESULTS

Our sample consisted of 677 academic vitreoretinal surgeons from 113 institutions. These physicians produced highly impactful research with a median publication count of 30 (interquartile range [IQR], 11-82), median RCR of 1.78 (IQR, 1.09-3.00), and median weighted RCR of 59.83 (14.31-195.78). Academic rank and career duration were associated with increased publication count, mean RCR, and weighted RCR. Publication count and weighted RCR differed significantly by gender; however, no difference was observed with mean RCR.

CONCLUSIONS

Current academic vitreoretinal surgeons have high mean RCR values relative to the National Institutes of Health standard RCR value of 1. This benchmark data serves as a more accurate gauge of research impact within the vitreoretinal community and can be used to inform self, institutional, and departmental evaluations. Additionally, the mean RCR may provide an accurate metric for quantifying research productivity among historically underrepresented groups that are disadvantaged by time-dependent factors, such as number of publications.

FINANCIAL DISCLOSURE(S)

The author(s) have no proprietary or commercial interest in any materials discussed in this article.

Stem cell therapy for type 1 diabetes: a scientometric assessment of global research during the twenty-first century

Purpose

We aimed to provide a scientometric assessment of global research in stem cell therapy (SCT) for type 1 diabetes (T1D) during 1999-2020.

Methods

The published data on SCT in T1D were retrieved from Elsevier's Scopus database and analyzed using select bibliometric tools. We used VOSviewer software and the Biblioshiny app to construct and visualize bibliometric networks.

Results

The global yield totaled 1806 publications in the 22-year study period, registering a 17.7% annual growth peaking at 196.9% in the last 11 years. The average citations per publication (CPP) decreased from 62.0 during 1999-2009 to 24.3 during 2010-2020. The funded publications were 727 (40.2%). Randomized controlled trials (RCTs) were only 2.4% (45). Amongst 70 participating countries, the USA led with a 38.6% share. Of the 388 global organizations, Harvard Medical School, USA, San Raffaele Scientific Institute, Italy, and the University of Florida, USA were the topmost contributors. Florina, Couri, and Trucco were the top productive authors, whereas Melton, Abdi, and Simoes were the most impactful. Only 129 (3.1%) publications were highly-cited; their total and average CPP were 31,228 and 214.0 (range 101-1841), respectively.

Conclusions

The quantity of research in SCT for T1D has increased during the last two decades while the quality has dipped. The research landscape is dominated by high-income North-American and Western-European countries. There is a need for conducting large-scale RCTs and promoting research collaborations between high- and low-income countries for long-term sustainability and global impact.

The relative citation ratio (RCR) as a novel bibliometric among 2511 academic orthopedic surgeons

Objectively measuring research output is important for grant awards, promotion, and tenure, or self-evaluation of productivity. However, certain shortcomings limit common bibliometric indicators. The time- and field-independent relative citation ratio (RCR) was proposed to overcome these limitations. The objective of this study was to determine whether the RCR correlates with academic rank, gender, and PhD degree status among US academic orthopedic surgeons. Full-time faculty surgeons at Accreditation Council for Graduate Medical Education-accredited orthopedic surgery residency programs were included in this study. Mean (mRCR) and weighted (wRCR) RCR scores were collected from the National Institutes of Health iCite database to quantify scholarly "impact" and "production," respectively, and were compared by academic rank, gender, and PhD status. All information was collected from publicly available faculty listings on departmental websites. A total of 2511 orthopedic surgeons from 132 residency programs were assessed. Overall, the median (interquartile range) mRCR score was 1.56 (1.05-2.12) and the median wRCR score was 27.6 (6.97-88.44). Both metrics increased with each successive academic rank, except for department chairs. There was no difference in mRCR between male and female surgeons. Among assistant professors, males had higher wRCR scores. Both metrics were higher among surgeons with a PhD degree. The RCR offers key advantages over other indices, which are reflected in differences in score distributions compared with the widely used h-index. Nevertheless, implementation of the RCR should be preceded with careful consideration of its own limitations.

Research Promotion is Associated with Broader Influence and Higher Impact of Plastic Surgery Publications

BACKGROUND

Social media has altered the mechanisms by which published research is disseminated and accessed. The objective of this study was to measure the effect of promotion on research article dissemination, influence, and impact in Plastic and Reconstructive Surgery.

METHODS

All articles published in Plastic and Reconstructive Surgery from January 1, 2016-December 31, 2018 were obtained and reviewed to determine inclusion/exclusion and for the Altmetric Attention Score (AAS), citations, relative citation rate (RCR), and 16 unique promotional tags (journal club, editor's pick, press release, patient safety, etc.) as indexed on the Plastic and Reconstructive Surgery website. 1,502 articles were included in the analysis. Statistical analysis was completed using descriptive statistics, Pearson's correlations, and Student t-tests where appropriate with a predetermined level of significance of p≤0.05.

RESULTS

A total of 637 articles (42.4%) had a promotional tag, while 252 (16.8%) had multiple tags. Articles with promotional tags had higher AAS (30.35 vs 8.22; p<0.001), more citations (11.96 vs 8.47; p<0.001), and a higher RCR (2.97 vs 2.06; p<0.001) compared to articles without a tag. Articles with multiple tags had higher AAS (50.17 vs 17.39; p<0.001), more citations (15.78 vs 9.47; p<0.001), and a higher RCR (3.67 vs 2.51; p<0.001) compared to articles with only one tag. As the number of tags increased for an article, AAS (p<0.001), citation count (p<0.001), and RCR (p<0.001) likewise increased.

CONCLUSIONS

This analysis strongly suggests that promotion of research articles is associated with significantly wider dissemination, broader visibility, and more subsequent citations in the literature.

The Influence of Orthopedic Journals in Knee Arthritis Treatment Research: Evaluation Using Relative Citation Ratio

Introduction

The iCite database, developed by the National Institute of Health (NIH), utilizes a bibliometric known as the relative citation ratio (RCR) to gauge scholarly impact. The goal of this study was to use the RCR to evaluate the influence of orthopedic journals in regard to knee arthritis treatment literature, as no such studies exist to date.

Materials and methods

The 100 highest RCR-rated articles published between 2007 and 2017 were obtained in the following categories: physical therapy (PT), viscosupplementation (VS), nonsteroidal anti-inflammatory drugs (NSAIDs), corticosteroid injection (CSI), results of total knee arthroplasty (TKA), platelet-rich plasma (PRP), and meniscectomy (MS). Journals were categorized with respect to the following specialties: general orthopedics (GO), orthopedic subspecialty (OSS), nonsurgical musculoskeletal (NSMSK), general medicine (GM), and basic science/nonclinical (BS/NC).

Results

Across the seven domains, GO journals held the highest median RCR, while OSS ranked fourth (RCR, 6.60 versus 3.95; p=0.0027). GO journals were considered the most influential specialty in CSI (RCR, 2.99), while OSS journals held the highest median RCR in PRP (RCR, 4.10). OSS and GO journals ranked third (RCR, 4.79) and fourth (RCR, 4.21), respectively, in NSAIDs, lagging behind NSMSK and GM journals.

Conclusions

Bibliometric tools, such as the RCR, can inform the orthopedic field of current and future research trends and help guide further research efforts. Currently, publications in GO journals hold a strong influence in CSI but less so in PT and NSAIDs. The use of bibliometrics allows the identification of highly influential non-orthopedic articles and journals to read while identifying influential non-orthopedic researchers to promote interdisciplinary collaboration.

The relative citation ratio and the h-index among academic ophthalmologists: A retrospective cross-sectional analysis

Background

Limited research has evaluated the relative citation ratio (RCR), a novel measure of research productivity. Accordingly, there remains a minimal understanding of its practical value relative to established metrics such as the h-index. Here, we examined correlations between the mean and weighted RCR scores and the h-index and explore the influence of academic rank, career duration, PhD acquisition, and fellowship training on these metrics.

Methods

Data regarding the academic rank (e.g. assistant professor, associate professor, professor, or “other”), career duration, degrees, fellowship training, and research yield were collected for 1018 academic ophthalmologists practicing in the southern United States of America. The iCite and Scopus databases were utilized to quantify research yield via calculations of mean and weighted RCR, and h-index, respectively.

Results

Significant correlations were observed between the h-index and the mean (ρ = 0.62, P < 0.001) and weighted RCR (ρ = 0.84, P < 0.001). Advanced academic rank was associated with increased indices values. In a subset of ophthalmologists excluding members of the “other” category, career duration was moderately correlated with h-index (ρ = 0.45, P < 0.001), and weakly correlated with mean (ρ = 0.14, P < 0.001) and weighted (ρ = 0.26, P < 0.001) RCR. PhD and fellowship acquisition were associated with increased research yield.

Conclusion

The findings suggest that the RCR is an effective measure of research yield, while resolving deficiencies present in the h-index. Further research remains to characterize the RCR's value relative to other established markers of research productivity.

•

The relative citation ratio (RCR) is a novel NIH-supported research metric.

•

Advanced academic rank and PhD acquisition is associated with increased RCR and h-index.

•

The h-index is poorer at delineating differences due to career duration.

•

Overall, the RCR is a more representative measure of research productivity.

Root Canal Disinfection Articles with the Highest Relative Citation Ratios. A Bibliometric Analysis from 1990 to 2019

The relative citation rate (RCR) is a normalized article-level metric useful to assess the impact of research articles. The objective of this bibliometric study is to identify and analyze, in root canal disinfection, the 100 articles having the highest RCRs in the period 1990–2019, then compare them with the top 100 articles most cited. A cross-sectional study was performed, and the search strategy ((Disinfection AND root canal) AND ((“1990/01/01”[Date-Publication]: “2019/12/31”[Date-Publication]))) relied on PubMed (n = 4294 documents), and article data were downloaded from the iCite database. The 100 articles with the highest RCRs and the top 100 cited were selected and evaluated in bibliometric terms. Among the 100 articles with the highest RCRs, there were no differences in the three decades for RCRs values, but there were in citations, being 2000–2009 the most cited. The USA was the predominant country (n = 30), followed by Brazil (n = 14). The most frequent study designs were reviews (n = 27) and in vitro (n = 25) and ex vivo (n = 24) studies. All subfields were well represented, although they varied over time. In 2010–2019, regenerative procedures and irrigation/disinfection techniques were predominant. Considering the RCR’s top 100 articles, 76 were common with the 100 most cited articles. Using the RCR metric allowed us to identify influential articles in root canal disinfection, a research field with topics of significance that fluctuate over time. Compared to citations, RCR reduces the time from publication to detection of its importance for the readership and could be a valid alternative to citation counts.

The Relative Citation Ratio: Examining a Novel Measure of Research Productivity among Southern Academic Ophthalmologists

PurposeTo analyze research productivity, as assessed by the National Institutes of Health-supported relative citation ratio (RCR), for a cohort of Southern academic ophthalmologists.DesignA descriptive and cross-sectional design was used. Data on gender, academic rank (assigned as an assistant professor, associate professor, professor, or "other"), degrees, and career duration were collected using online resources. Research yield was quantified using mean and weighted RCR data queried from the iCite database. Significant between-group differences were calculated using the Mann-Whitney U-test and the Kruskal-Wallis test.SettingsPracticing academic ophthalmologists at Accreditation Council for Graduate Medical Education-accredited ophthalmology programs in the Southern United States (n = 1018).ResultsFor all Southern academic ophthalmologists, median mean RCR was 0.90 (IQR 0.18-1.71) and median weighted RCR was 5.12 (IQR 0.34-33.18). Advanced academic rank and PhD acquisition were significantly associated with increased mean and weighted RCR. After exclusion of faculty within the "other" category, median mean RCR was 1.12 (IQR 0.54-1.80) and median weighted RCR was 11.65 (IQR 2.03-45.58). Furthermore, effects of career duration and gender emerged. Ophthalmologists with longer careers had significantly higher mean and weighted RCR than their younger counterparts. Males had significantly higher mean and weighted RCR than females.ConclusionsAcademic rank and attainment of a PhD degree were correlated with increased research productivity. When analyses focused exclusively on faculty not in the "other" subgroup, male gender, and lengthier career were associated with increased mean and weighted RCR, the former of which potentially highlights differences in professional advancement between genders.

Requiem for impact factors and high publication charges

Journal impact factors, publication charges and assessment of quality and accuracy of scientific research are critical for researchers, managers, funders, policy makers, and society. Editors and publishers compete for impact factor rankings, to demonstrate how important their journals are, and researchers strive to publish in perceived top journals, despite high publication and access charges. This raises questions of how top journals are identified, whether assessments of impacts are accurate and whether high publication charges borne by the research community are justified, bearing in mind that they also collectively provide free peer-review to the publishers. Although traditional journals accelerated peer review and publication during the COVID-19 pandemic, preprint servers made a greater impact with over 30,000 open access articles becoming available and accelerating a trend already seen in other fields of research. We review and comment on the advantages and disadvantages of a range of assessment methods and the way in which they are used by researchers, managers, employers and publishers. We argue that new approaches to assessment are required to provide a realistic and comprehensive measure of the value of research and journals and we support open access publishing at a modest, affordable price to benefit research producers and consumers.

Journal- or article-based citation measure? A study of academic promotion at a Swiss university

In academia, decisions on promotions are influenced by the citation impact of the works published by the candidates. The Medical Faculty of the University of Bern used a measure based on the journal impact factor (JIF) for this purpose: the JIF of the papers submitted for promotion should rank in the upper third of journals in the relevant discipline (JIF rank >0.66). The San Francisco Declaration on Research Assessment (DORA) aims to eliminate the use of journal-based metrics in academic promotion. We examined whether the JIF rank could be replaced with the relative citation ratio (RCR), an article-level measure of citation impact developed by the National Institutes of Health (NIH). An RCR percentile >0.66 corresponds to the upper third of citation impact of articles from NIH-sponsored research. We examined 1525 publications submitted by 64 candidates for academic promotion at University of Bern. There was only a moderate correlation between the JIF rank and RCR percentile (Pearson correlation coefficient 0.34, 95% CI 0.29-0.38). Among the 1,199 articles (78.6%) published in journals ranking >0.66 for the JIF, less than half (509, 42.5%) were in the upper third of the RCR percentile. Conversely, among the 326 articles published in journals ranking <0.66 regarding the JIF, 72 (22.1%) ranked in the upper third of the RCR percentile. Our study demonstrates that the rank of the JIF is a bad proxy measure for the actual citation impact of individual articles. The Medical Faculty of University of Bern has signed DORA and replaced the JIF rank with the RCR percentile to assess the citation impact of papers submitted for academic promotion.

Using bibliometrics to evaluate translational science training: evidence for early career success of KL2 scholars

Introduction:

Evaluating clinical and translational research (CTR) mentored training programs is challenging because no two programs are alike. Careful selection of appropriate metrics is required to make valid comparisons between individuals and between programs. The KL2 program provides mentored-training for early-stage CTR investigators. Clinical and Translational Awards across the country have unique KL2 programs. The evaluation of KL2 programs has begun to incorporate bibliometrics to measure KL2 scholar and program impact.

Methods:

This study investigated demographic differences in bibliometric performance and post-K award funding of KL2 scholars and compared the bibliometric performance and post-K award federal funding of KL2 scholars and other mentored-K awardees at the same institution. Data for this study included SciVal and iCite bibliometrics and National Institutions of Health RePORTER grant information for mentored-K awardees (K08, K23, and KL2) at Case Western Reserve University between 2005 and 2013.

Results:

Results showed no demographics differences within the KL2 program scholars. Bibliometric differences between KL2 and other mentored-K awardee indicated an initial KL2 advantage for the number of publications at 5 years’ post-matriculation (i.e., the start of the K award). Regression analyses indicated the number of initial publications was a significant predictor of federal grant funding at the same time point. Analysis beyond the 5-year post-matriculation point did not result in a sustained, significant KL2 advantage.

Conclusions:

Factors that contributed to the grant funding advantage need to be determined. Additionally, differences between translational and clinical bibliometrics must be interpreted with caution, and appropriate metrics for translational science must be established.

Characteristics of Drug Intervention Clinical Trials and Scientific Impact of the Trial Outcome: A Bibliometric Analysis Using the Relative Citation Ratio in Non-small Cell Lung Cancer from 2007 to 2016

BACKGROUND

Although a large number of clinical trials have been conducted, the types of clinical trials that are scientifically influential, frequently utilized by society, and contribute to the progress of evidence-based medicine (EBM) have not been studied. Thus, we aimed to investigate the relationship between the characteristics of clinical trials and the scientific impact of the outcome in non-small cell lung cancer (NSCLC) by performing a bibliometric analysis using relative citation ratio (RCR), a newly developed bibliometric index by the National Institutes of Health (NIH).

METHODS

Primary publications of drug intervention clinical trials for NSCLC between 2007 and 2016 were included in the study. The characteristics of clinical trials were compared among four RCR categories with 50 trials in each [LOW50, 50 NIH percentile (50NIH%ile), 95 NIH percentile (95NIH%ile), and TOP50], totaling to 200 trials.

RESULTS

Median RCRs of LOW50, 50NIH%ile, 95NIH%ile, and TOP50 were 0.03, 1.00, 5.76, and 26.89, respectively. Publications of Phase 3, randomized, blinded, for-profit-company supported/sponsored, multi-center trials, and trials with a larger number of subjects were shown to have a higher scientific impact. Publications of clinical trials of newly developed molecular target drugs, including epidermal growth factor receptor-tyrosine kinase inhibitors, anaplastic lymphoma kinase inhibitors, and immune checkpoint inhibitors demonstrated a higher scientific impact than those of traditional chemotherapies.

CONCLUSION

Clinical trials designed to have a high evidence level would improve the scientific impact of the outcome, and novel interventions would be another factor to improve the clinical trials' influence.

Assessment of the NIH-supported relative citation ratio as a measure of research productivity among 1687 academic neurological surgeons

OBJECTIVE

Publication metrics such as the Hirsch index (h-index) are often used to evaluate and compare research productivity in academia. The h-index is not a field-normalized statistic and can therefore be dependent on overall rates of publication and citation within specific fields. Thus, a metric that adjusts for this while measuring individual contributions would be preferable. The National Institutes of Health (NIH) has developed a new, field-normalized, article-level metric called the "relative citation ratio" (RCR) that can be used to more accurately compare author productivity between fields. The mean RCR is calculated as the total number of citations per year of a publication divided by the average field-specific citations per year, whereas the weighted RCR is the sum of all article-level RCR scores over an author's career. The present study was performed to determine how various factors, such as academic rank, career duration, a Doctor of Philosophy (PhD) degree, and sex, impact the RCR to analyze research productivity among academic neurosurgeons.

METHODS

A retrospective data analysis was performed using the iCite database. All physician faculty affiliated with Accreditation Council for Graduate Medical Education (ACGME)-accredited neurological surgery programs were eligible for analysis. Sex, career duration, academic rank, additional degrees, total publications, mean RCR, and weighted RCR were collected for each individual. Mean RCR and weighted RCR were compared between variables to assess patterns of analysis by using SAS software version 9.4.

RESULTS

A total of 1687 neurosurgery faculty members from 125 institutions were included in the analysis. Advanced academic rank, longer career duration, and PhD acquisition were all associated with increased mean and weighted RCRs. Male sex was associated with having an increased weighted RCR but not an increased mean RCR score. Overall, neurological surgeons were highly productive, with a median RCR of 1.37 (IQR 0.93-1.97) and a median weighted RCR of 28.56 (IQR 7.99-85.65).

CONCLUSIONS

The RCR and its derivatives are new metrics that help fill in the gaps of other indices for research output. Here, the authors found that advanced academic rank, longer career duration, and PhD acquisition were all associated with increased mean and weighted RCRs. Male sex was associated with having an increased weighted, but not mean, RCR score, most likely because of historically unequal opportunities for women within the field. Furthermore, the data showed that current academic neurosurgeons are exceptionally productive compared to both physicians in other specialties and the general scientific community.