Feasibility of common bibliometrics in evaluating translational science

Translation in action: Influence, collaboration, and evolution of COVID-19 research with Clinical and Translational Science Awards consortium support

The National Institutes of Health (NIH)'s Clinical and Translational Science Awards (CTSA) consortium aims to accelerate translational processes that move discoveries from bench to bedside. The coronavirus disease 2019 (COVID-19) pandemic presented unmatched challenges and applications for CTSA hubs nationwide. Our study used bibliometrics to assess features of COVID-19 publications supported by the national CTSA program to characterize the consortium's response to the pandemic. Our goal was to understand relative scientific influence, collaboration across hubs, and trends in research emphasis over time. We identified publications from NIH's curated iSearch COVID-19 Publication Portfolio from February 2020 to February 2023; 3234 peer-reviewed articles relevant to COVID-19 cited a CTSA grant. All 66 CTSA hubs were represented, with large-size and longstanding hubs contributing more publications. Most publications cited UL1 grants, 457 cited KL2/TL1 training grants, and 164 cited multiple hub grants. Compared to a random sample of non-CTSA-supported COVID-19 publications, the CTSA portfolio exhibited greater clinical relevance, more human research, and higher altmetric and citation influence. Results were similar for multi-hub publications involving networked initiatives like multi-site clinical trials or the National COVID-19 Cohort Collaborative. Shifts from molecular/cellular-oriented research toward human-oriented research over time were evident, demonstrating translation in action. Results illuminate how the CTSA consortium confronted the pandemic through high-quality projects oriented toward human research, working across hubs on high-value collaborations, advancing along the translational spectrum over time. Findings validate CTSA hubs as critical support structures during health emergencies.

Evaluating the impact of a CTSA program from 2008 to 2021 through bibliometrics, social network analysis, and altmetrics

Introduction

We evaluate a CTSA program hub by applying bibliometrics, social network analysis (SNA), and altmetrics and examine the changes in research productivity, citation impact, research collaboration, and CTSA-supported research topics since our pilot study in 2017.

Methods

The sampled data included North Carolina Translational and Clinical Science Institute (NC TraCS)-supported publications produced between September 2008 and March 2021. We applied measures and metrics from bibliometrics, SNA, and altmetrics to the dataset. In addition, we analyzed research topics and correlations between different metrics.

Results

1154 NC TraCS-supported publications generated over 53,560 citation counts by April 2021. The average cites per year and the relative citation ratio (RCR) mean of these publications improved from 33 and 2.26 in 2017 to 48 and 2.58 in 2021. The number of involved UNC units in the most published authors' collaboration network increased from 7 (2017) to 10 (2021). NC TraCS-supported co-authorship involved 61 NC organizations. PlumX metrics identified articles with the highest altmetrics scores. About 96% NC TraCS-supported publications have above the average SciVal Topic Prominence Percentile; the average approximate potential to translate of the included publication was 54.2%; and 177 publications addressed health disparity issues. Bibliometric measures (e.g., citation counts, RCR) and PlumX metrics (i.e., Citations, Captures, and Social-Media) are positively correlated (p < .05).

Conclusion

Bibliometrics, SNA, and altmetrics offer distinctive but related perspectives to examine CTSA research performance and longitudinal growth, especially at the individual program hub level. These perspectives can help CTSAs build program foci.

The Rare Diseases Clinical Research Network: a model for clinical trial readiness

Background

The current road to developing treatments for rare diseases is often slow, expensive, and riddled with risk. Change is needed to improve the process, both in how we think about rare disease treatment development and the infrastructure we build to support ongoing science. The National Institutes of Health (NIH)-supported Rare Diseases Clinical Research Network (RDCRN) was established to advance the diagnosis, management, and treatment of rare diseases and to promote highly collaborative, multi-site, patient-centric, translational, and clinical research. The current iteration of the RDCRN intends to build upon and enhance successful approaches within the network while identifying innovative methods to fill gaps and address needs in the approach to the rare disease treatment development process through innovation, collaboration, and clinical trial readiness.

Objective

The objective of this paper is to provide an overview of the productivity and influence of the RDCRN since it was first established 20 years ago.

Design and methods

Using a suite of tools available to NIH staff that provides access to a comprehensive, curated, extensively linked data set of global grants, patents, publications, clinical trials, and FDA-approved drugs, a series of queries were executed that conducted bibliometric, co-author, and co-occurrence analysis.

Results

The results demonstrate that the entire RDCRN consortia and network has been highly productive since its inception. They have produced 2763 high-quality publications that have been cited more than 100,000 times, expanded international networks, and contributed scientifically to eight FDA-approved treatments for rare diseases.

Conclusion

The RDCRN program has successfully addressed some significant challenges while developing treatments for rare diseases. However, looking to the future and being agile in facing new challenges that arise as science progresses is important.

Translational research: from basic research to regional biomedical entrepreneurship

This paper examines the effect of translational research on knowledge production and biomedical entrepreneurship across U.S. regions. Researchers have earlier investigated the outputs of translational research by focusing on academic publications. Little attention has been paid to linking translational research to biomedical entrepreneurship. We construct an analytical model based on the knowledge spillover theory of entrepreneurship and the entrepreneurial ecosystem approach to examine the relationship between translational research, biomedical patents, clinical trials, and biomedical entrepreneurship. We test the model across 381 U.S. metropolitan statistical areas using 10 years of panel data related to the NIH Clinical and Translational Science Awards (CTSA) program. CTSA appears to increase the number of biomedical patents and biomedical entrepreneurship as proxied by the NIH Small Business Innovation Research (SBIR) grants. However, the magnitudes of the effects are relatively small. Path analysis shows that the effect of translational research on regional biomedical entrepreneurship is not strongly conveyed through biomedical patents or clinical trials.

Methods for identifying biomedical translation: a systematic review

Translational medicine is an important area of biomedicine, and has significantly facilitated the development of biomedical research. Despite its relevance, there is no consensus on how to evaluate its progress and impact. A systematic review was carried out to identify all the methods to evaluate translational research. Seven methods were found according to the established criteria to analyze their characteristics, advantages, and limitations. They allow us to perform this type of evaluation in different ways. No relevant advantages were found between them; each one presented its specific limitations that need to be considered. Nevertheless, the Triangle of Biomedicine could be considered the most relevant method, concerning the time since its publication and usefulness. In conclusion, there is still a lack of a gold-standard method for evaluating biomedical translational research.

A large National Institute for Health Research (NIHR) Biomedical Research Centre facilitates impactful cross-disciplinary and collaborative translational research publications and research collaboration networks: a bibliometric evaluation study

Background

The evaluation of translational health research is important for various reasons such as the research impact assessment, research funding allocation, accountability, and strategic research policy formulation. The purpose of this study was to evaluate the research productivity, strength and diversity of research collaboration networks and impact of research supported by a large biomedical research centre in the United Kingdom (UK).

Methods

Bibliometric analysis of research publications by translational researchers affiliated with the National Institute for Health Research (NIHR) Oxford Biomedical Research Centre (BRC) from April 2012 to March 2017.

Results

Analysis included 2377 translational research publications that were published during the second 5-year funding period of the NIHR Oxford BRC. Author details were available for 99.75% of the publications with DOIs (2359 of 2365 with DOIs), and the number of authors per publication was median 9 (mean  = 18.03, SD  = 3.63, maximum  = 2467 authors). Author lists also contained many consortia, groups, committees, and teams (n  = 165 in total), with 1238 additional contributors, where membership was reported. The BRC co-authorship i.e., research collaboration network for these publications involved 20,229 nodes (authors, of which 1606 nodes had Oxford affiliations), and approximately 4.3 million edges (authorship linkages). Articles with a valid DOIs (2365 of 2377, 99.5%) were collectively cited more than 155,000 times and the average Field Citation Ratio was median 6.75 (geometric mean  = 7.12) while the average Relative Citation Ratio was median 1.50 (geometric mean  = 1.83) for the analysed publications.

Conclusions

The NIHR Oxford BRC generated substantial translational research publications and facilitated a huge collaborative network of translational researchers working in complex structures and consortia, which shows success across the whole of this BRC funding period. Further research involving continued uptake of unique persistent identifiers and the tracking of other research outputs such as clinical innovations and patents would allow a more detailed understanding of large research enterprises such as NIHR BRCs in the UK.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12967-021-03149-x.

Using bibliometrics to evaluate outcomes and influence of translational biomedical research centers

Introduction

Federal grant funding to support infrastructure development of translational biomedical research centers is a form of public health intervention. Establishing rigorous methods for measuring center success and outcomes is essential to justify continued funding.

Methods

Bibliometric data compiled from a 5-year funding cycle of neurodegeneration and translational neuroscience research center were analyzed using the package bibliometrix for open-source software R and the NIH-developed research tool iCite.

Results

The research team and their collaborators (n = 485) produced 157 grant-citing publications from 2015-2020. The science was produced by small research teams clustered around three main communities of topics: Alzheimer's Disease, brain imaging, and neuropsychological testing in the elderly. Using the relative citation ratio, the publications produced by the research team were found to be influential when compared to other R01-funded publications.

Conclusion

Recent developments in bibliometric analysis expand beyond traditional measurement capabilities to better understand the characteristics, outcomes, and influences of research teams. These findings can be used to inform researchers and institutions about research team composition, productivity, and success. Measures of research influence may be used to justify return on investment to funders.

Scholarly Productivity Evaluation of KL2 Scholars Using Bibliometrics and Federal Follow-on Funding: Cross-Institution Study

Background

Evaluating outcomes of the clinical and translational research (CTR) training of a Clinical and Translational Science Award (CTSA) hub (eg, the KL2 program) requires the selection of reliable, accessible, and standardized measures. As measures of scholarly success usually focus on publication output and extramural funding, CTSA hubs have started to use bibliometrics to evaluate the impact of their supported scholarly activities. However, the evaluation of KL2 programs across CTSAs is limited, and the use of bibliometrics and follow-on funding is minimal.

Objective

This study seeks to evaluate scholarly productivity, impact, and collaboration using bibliometrics and federal follow-on funding of KL2 scholars from 3 CTSA hubs and to define and assess CTR training success indicators.

Methods

The sample included KL2 scholars from 3 CTSA institutions (A-C). Bibliometric data for each scholar in the sample were collected from both SciVal and iCite, including scholarly productivity, citation impact, and research collaboration. Three federal follow-on funding measures (at the 5-year, 8-year, and overall time points) were collected internally and confirmed by examining a federal funding database. Both descriptive and inferential statistical analyses were computed using SPSS to assess the bibliometric and federal follow-on funding results.

Results

A total of 143 KL2 scholars were included in the sample with relatively equal groups across the 3 CTSA institutions. The included KL2 scholars produced more publications and citation counts per year on average at the 8-year time point (3.75 publications and 26.44 citation counts) than the 5-year time point (3.4 publications vs 26.16 citation counts). Overall, the KL2 publications from all 3 institutions were cited twice as much as others in their fields based on the relative citation ratio. KL2 scholars published work with researchers from other US institutions over 2 times (5-year time point) or 3.5 times (8-year time point) more than others in their research fields. Within 5 years and 8 years postmatriculation, 44.1% (63/143) and 51.7% (74/143) of KL2 scholars achieved federal funding, respectively. The KL2-scholars of Institution C had a significantly higher citation rate per publication than the other institutions (P<.001). Institution A had a significantly lower rate of nationally field-weighted collaboration than did the other institutions (P<.001). Institution B scholars were more likely to have received federal funding than scholars at Institution A or C (P<.001).

Conclusions

Multi-institutional data showed a high level of scholarly productivity, impact, collaboration, and federal follow-on funding achieved by KL2 scholars. This study provides insights on the use of bibliometric and federal follow-on funding data to evaluate CTR training success across institutions. CTSA KL2 programs and other CTR career training programs can benefit from these findings in terms of understanding metrics of career success and using that knowledge to develop highly targeted strategies to support early-stage career development of CTR investigators.

Practical publication metrics for academics

Research organizations are becoming more reliant on quantitative approaches to determine how to recruit and promote researchers, allocate funding, and evaluate the impact of prior allocations. Many of these quantitative metrics are based on research publications. Publication metrics are not only important for individual careers, but also affect the progress of science as a whole via their role in the funding award process. Understanding the origin and intended use of popular publication metrics can inform an evaluative strategy that balances the usefulness of publication metrics with the limitations of what they can convey about the productivity and quality of an author, a publication, or a journal. This paper serves as a brief introduction to citation networks like Google Scholar, Web of Science Core Collection, Scopus, Microsoft Academic, and Dimensions. It also explains two of the most popular publication metrics: the h-index and the journal impact factor. The purpose of this paper is to provide practical information on using citation networks to generate publication metrics, and to discuss ideas for contextualizing and juxtaposing metrics, in order to help researchers in translational science and other disciplines document their impact in as favorable a light as may be justified.

A Bibliometric Measure of Translational Science

Science funders are increasingly requiring evidence of the broader impacts of even basic research. Initiatives such as NIH's CTSA program are designed to shift the research focus toward more translational research. However, tracking the effectiveness of such programs depends on developing indicators that can track the degree to which basic research is influencing clinical research. We propose a new bibliometric indicator, the TS score, that is relatively simple to calculate, can be implemented at scale, is easy to replicate, and has good reliability and validity properties. This indicator is broadly applicable in settings where the goal is to estimate the degree to which basic research is used in more applied downstream research, relative to use in basic research. The TS score should be of use for a variety of policy analysis and research evaluation purposes.

Scope, Influence, and Interdisciplinary Collaboration: The Publication Portfolio of the NIH Clinical and Translational Science Awards (CTSA) Program From 2006 Through 2017

The Clinical and Translational Science Awards (CTSA) program sponsors an array of innovative, collaborative research. This study uses complementary bibliometric approaches to assess the scope, influence, and interdisciplinary collaboration of publications supported by single CTSA hubs and those supported by multiple hubs. Authors identified articles acknowledging CTSA support and assessed the disciplinary scope of research areas represented in that publication portfolio, their citation influence, interdisciplinary overlap among research categories, and characteristics of publications supported by multihub collaborations. Since 2006, CTSA hubs supported 69,436 articles published in 4,927 journals and 189 research areas. The portfolio is well distributed across diverse research areas with above-average citation influence. Most supported publications involved clinical/health sciences, for example, neurology and pediatrics; life sciences, for example, neuroscience and immunology; or a combination of the two. Publications supported by multihub collaborations had distinct content emphasis, stronger citation influence, and greater interdisciplinary overlap. This study characterizes the CTSA consortium's contributions to clinical and translational science, identifies content areas of strength, and provides evidence for the success of multihub collaborations. These methods lay the foundation for future investigation of the best policies and priorities for fostering translational science and allow hubs to understand their progress benchmarked against the larger consortium.

Using bibliometrics to evaluate translational science training: evidence for early career success of KL2 scholars

Introduction:

Evaluating clinical and translational research (CTR) mentored training programs is challenging because no two programs are alike. Careful selection of appropriate metrics is required to make valid comparisons between individuals and between programs. The KL2 program provides mentored-training for early-stage CTR investigators. Clinical and Translational Awards across the country have unique KL2 programs. The evaluation of KL2 programs has begun to incorporate bibliometrics to measure KL2 scholar and program impact.

Methods:

This study investigated demographic differences in bibliometric performance and post-K award funding of KL2 scholars and compared the bibliometric performance and post-K award federal funding of KL2 scholars and other mentored-K awardees at the same institution. Data for this study included SciVal and iCite bibliometrics and National Institutions of Health RePORTER grant information for mentored-K awardees (K08, K23, and KL2) at Case Western Reserve University between 2005 and 2013.

Results:

Results showed no demographics differences within the KL2 program scholars. Bibliometric differences between KL2 and other mentored-K awardee indicated an initial KL2 advantage for the number of publications at 5 years’ post-matriculation (i.e., the start of the K award). Regression analyses indicated the number of initial publications was a significant predictor of federal grant funding at the same time point. Analysis beyond the 5-year post-matriculation point did not result in a sustained, significant KL2 advantage.

Conclusions:

Factors that contributed to the grant funding advantage need to be determined. Additionally, differences between translational and clinical bibliometrics must be interpreted with caution, and appropriate metrics for translational science must be established.

Scientometric analysis of the Puerto Rico Clinical and Translational Research Consortium (PRCTRC) research publications, 2010-2018

We analyzed the publication productivity supported by the Puerto Rico Consortium for Clinical and Translational Research (PRCTRC) using the structured process of scientometrics. The objective of this study was to evaluate the impact of the research and collaborations as presented in publications. Manuscripts published from 2010 to 2018 and that had the PRCTRC award number and a PMCID number were retrieved from the Science Citation Index database. Scientometric indicators included h-index (HI), average citation (AC), collaboration coefficient (CC), collaboration index (CI), and degree of collaboration (DC) analysis, and relative citation ratio (RCR) was done with Web of Science Platform, iCite, and Stata software. Joinpoint Trend Analysis Software was used to calculate the annual percent change (APC). From 2010 to 2018, 341 publications were identified with an average of 38 publications per year and a total of 3569 citations excluding self-citations. A significant growth (APC: 17.76%, P < 0.05) of scientific production was observed. The overall HI was 31, and the AC per item was 11.04. The overall CC was 0.82, the CI was 8.59, and the DC was 99.1%. This study demonstrates a statistically significant increase in the PRCTRC scientific production. Results allow for the assessment of the progress resulting from the provided support and to plan further strategies accordingly.

Bibliometrics approach to evaluating the research impact of CTSAs: A pilot study

INTRODUCTION

To enhance the performance evaluation of Clinical and Translational Science Award (CTSA) hubs, we examined the utility of advanced bibliometric measures that go beyond simple publication counts to demonstrate the impact of translational research output.

METHODS

The sampled data included North Carolina Translational and Clinical Science Institute (NC TraCS)-supported publications produced between September 2008 and March 2017. We adopted advanced bibliometric measures and a state-of-the-art bibliometric network analysis tool to assess research productivity, citation impact, the scope of research collaboration, and the clusters of research topics.

RESULTS

Totally, 754 NC TraCS-supported publications generated over 24,000 citation counts by April 2017 with an average of 33 cites per article. NC TraCS-supported research papers received more than twice as many cites per year as the average National Institute of Health-funded research publications from the same field and time. We identified the top productive researchers and their networks within the CTSA hub. Findings demonstrated the impact of NC TraCS in facilitating interdisciplinary collaborations within the CTSA hub and across the CTSA consortium and connecting researchers with right peers and organizations.

CONCLUSION

Both improved bibliometrics measures and bibliometric network analysis can bring new perspectives to CTSA evaluation via citation influence and the scope of research collaborations.

Bibliometric Analysis of Electronic Cigarette Publications: 2003⁻2018

Electronic cigarettes are at the center of a public health policy debate which leverages scientific publications. This study characterizes e-cigarette publication trends over the past 15 years via a bibliometric analysis. Scopus was searched for “electronic cigarette”, “e-cig”, “e-cigarette”, “vape”, “vaping”, “juul”, or “electronic nicotine delivery system” between 2003–2018. Data included Hirsch index, document type and frequency, and publications by institution, journal, and country. VOSviewer was used to visualize authorship network maps. A total of 4490 e-cigarette publications were identified, most (62.8%) being articles. After 2009, the annual growth rate for e-cigarette publications was the largest in 2014. The annual growth rate was nearly flat in 2017 but increased in 2018. The U.S. produced 51.6% of publications. Annual National Institutes of Health NIH funding for tobacco research mapped closely with the annual volume of e-cigarette publications. Author network analyses illustrated investigator collaborative patterns. The frequency of e-cigarette publications increased significantly in the past decade. A strong relationship of NIH funding for tobacco research and e-cigarette publications demonstrates the importance of e-cigarettes in tobacco research.

The relative citation ratio: what is it and why should medical librarians care?

Bibliometrics is becoming increasingly prominent in the world of medical libraries. The number of presentations related to research impact at the Medical Library Association (MLA) annual meeting has been increasing in past years. Medical centers have been using institutional dashboards to track clinical performance for over a decade, and more recently, these institutional dashboards have included measures of academic performance. This commentary reviews current practices and considers the role for a newer metric, the relative citation ratio.

Charting the Publication and Citation Impact of the NIH Clinical and Translational Science Awards (CTSA) Program From 2006 Through 2016

PURPOSE

The authors evaluated publication and citation patterns for articles supported by Clinical and Translational Science Awards (CTSA) hub investment over the first decade of the CTSA program. The aim was to elucidate a pivotal step in the translational process by providing an account of how time, hub maturity, and hub attributes were related to productivity and influence in the academic literature.

METHOD

In 2017, the authors collected bibliometric data from PubMed, Web of Science InCites, and National Institutes of Health (NIH) iCite for articles citing any CTSA hub grants published from hub inception through 2016. They compiled data on publication and citation rates and indices of relative citation impact aggregated by hub funding year cohort. They compared hub-level bibliometric activity by multi- versus single-institution structure and total monetary award sums, compiled from NIH RePORTER.

RESULTS

From 2006-2016, CTSA hubs supported over 66,000 publications, with publication rates accelerating as hubs matured. These publications accumulated over 1.2 million citations, with some articles cited over 1,000 times. Indices of relative citation impact indicated CTSA-supported publications were cited more than twice as often as expected for articles of their publication years and disciplines. Multi-institutional hubs and those awarded higher grant sums exhibited significantly higher publication and citation activity.

CONCLUSIONS

The CTSA program is yielding a robust and growing body of influential research findings with consistently high indices of relative citation impact. Preliminary evidence suggests multi-institutional collaborations and more monetary resources are associated with elevated bibliometric activity and, therefore, may be worth their investment.

A snapshot of translational research funded by the National Institutes of Health (NIH): A case study using behavioral and social science research awards and Clinical and Translational Science Awards funded publications

The translation of biomedical research from basic knowledge to application has been a priority at the National Institute of Health (NIH) for many years. Tracking the progress of scientific research and knowledge through the translational process is difficult due to variation in the definition of translational research as well as the identification of benchmarks for the spread and application of biomedical research; quantitatively tracking this process is even more difficult. Using a simple and reproducible method to assess whether publications are translational, we examined NIH R01 behavioral and social science research (BSSR) awards funded between 2008 and 2014 to determine whether there are differences in the percent of translational research publications produced by basic and applied research awards. We also assessed the percent of translational research publications produced by the Clinical and Translational Science Awards (CTSA) program to evaluate whether targeted translational research awards result in increased translational research. We found that 3.9% of publications produced by basic research awards were translational; that the percent of translational research publications produced by applied research awards is approximately double that of basic research awards (7.4%); and that targeted translational research awards from the CTSA program produced the highest percentage of translational research publications (13.4%). In addition, we assessed differences in time to first publication, time to first citation, and publication quality by award type (basic vs. applied), and whether an award (or publication) is translational.

Using publication data to evaluate a Clinical and Translational Science Award (CTSA) career development program: Early outcomes from KL2 scholars

Introduction

This study uses KL2 scholars' publications to evaluate the types of research the KL2 program supports and to assess the initial productivity and impact of its scholars.

Methods

We illustrate the feasibility of 3 different approaches to bibliometrics, one viable method for determining the types of research a program or hub supports, and demonstrate how these data can be further combined with internal data records.

Results

Gender differences were observed in the types of research scholars undertake. Overall KL2 scholars are performing well, with their publications being cited more than the norm for National Institutes of Health publications. Favorable results were also observed in scholars' continued engagement in research.

Conclusion

This study illustrates that linking bibliometric data and data categorizing publications along the translational spectrum with a Clinical and Translational Science Award hub's internal data records is feasible and offers a number of innovative possibilities for the evaluation of a Clinical and Translational Science Award hub's programs and investigators.