1
|
Llewellyn N, Nehl EJ, Dave G, DiazGranados D, Flynn D, Fournier D, Hoyo V, Pelfrey C, Casey S. Translation in action: Influence, collaboration, and evolution of COVID-19 research with Clinical and Translational Science Awards consortium support. Clin Transl Sci 2024; 17:e13700. [PMID: 38156426 PMCID: PMC10777432 DOI: 10.1111/cts.13700] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/08/2023] [Revised: 11/20/2023] [Accepted: 11/27/2023] [Indexed: 12/30/2023] Open
Abstract
The National Institutes of Health (NIH)'s Clinical and Translational Science Awards (CTSA) consortium aims to accelerate translational processes that move discoveries from bench to bedside. The coronavirus disease 2019 (COVID-19) pandemic presented unmatched challenges and applications for CTSA hubs nationwide. Our study used bibliometrics to assess features of COVID-19 publications supported by the national CTSA program to characterize the consortium's response to the pandemic. Our goal was to understand relative scientific influence, collaboration across hubs, and trends in research emphasis over time. We identified publications from NIH's curated iSearch COVID-19 Publication Portfolio from February 2020 to February 2023; 3234 peer-reviewed articles relevant to COVID-19 cited a CTSA grant. All 66 CTSA hubs were represented, with large-size and longstanding hubs contributing more publications. Most publications cited UL1 grants, 457 cited KL2/TL1 training grants, and 164 cited multiple hub grants. Compared to a random sample of non-CTSA-supported COVID-19 publications, the CTSA portfolio exhibited greater clinical relevance, more human research, and higher altmetric and citation influence. Results were similar for multi-hub publications involving networked initiatives like multi-site clinical trials or the National COVID-19 Cohort Collaborative. Shifts from molecular/cellular-oriented research toward human-oriented research over time were evident, demonstrating translation in action. Results illuminate how the CTSA consortium confronted the pandemic through high-quality projects oriented toward human research, working across hubs on high-value collaborations, advancing along the translational spectrum over time. Findings validate CTSA hubs as critical support structures during health emergencies.
Collapse
Affiliation(s)
- Nicole Llewellyn
- Georgia Clinical and Translational Science AllianceEmory University School of MedicineAtlantaGeorgiaUSA
| | - Eric J. Nehl
- Emory University Rollins School of Public HealthAtlantaGeorgiaUSA
| | - Gaurav Dave
- University of North CarolinaChapel HillNorth CarolinaUSA
| | - Deborah DiazGranados
- Wright Regional Center for Clinical and Translational ScienceVirginia Commonwealth UniversityRichmondVirginiaUSA
| | | | | | | | | | | |
Collapse
|
2
|
Yu F, Patel T, Carnegie A, Dave G. Evaluating the impact of a CTSA program from 2008 to 2021 through bibliometrics, social network analysis, and altmetrics. J Clin Transl Sci 2023; 7:e44. [PMID: 36845314 PMCID: PMC9947612 DOI: 10.1017/cts.2022.530] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/15/2022] [Revised: 11/21/2022] [Accepted: 12/27/2022] [Indexed: 01/12/2023] Open
Abstract
Introduction We evaluate a CTSA program hub by applying bibliometrics, social network analysis (SNA), and altmetrics and examine the changes in research productivity, citation impact, research collaboration, and CTSA-supported research topics since our pilot study in 2017. Methods The sampled data included North Carolina Translational and Clinical Science Institute (NC TraCS)-supported publications produced between September 2008 and March 2021. We applied measures and metrics from bibliometrics, SNA, and altmetrics to the dataset. In addition, we analyzed research topics and correlations between different metrics. Results 1154 NC TraCS-supported publications generated over 53,560 citation counts by April 2021. The average cites per year and the relative citation ratio (RCR) mean of these publications improved from 33 and 2.26 in 2017 to 48 and 2.58 in 2021. The number of involved UNC units in the most published authors' collaboration network increased from 7 (2017) to 10 (2021). NC TraCS-supported co-authorship involved 61 NC organizations. PlumX metrics identified articles with the highest altmetrics scores. About 96% NC TraCS-supported publications have above the average SciVal Topic Prominence Percentile; the average approximate potential to translate of the included publication was 54.2%; and 177 publications addressed health disparity issues. Bibliometric measures (e.g., citation counts, RCR) and PlumX metrics (i.e., Citations, Captures, and Social-Media) are positively correlated (p < .05). Conclusion Bibliometrics, SNA, and altmetrics offer distinctive but related perspectives to examine CTSA research performance and longitudinal growth, especially at the individual program hub level. These perspectives can help CTSAs build program foci.
Collapse
Affiliation(s)
- Fei Yu
- Health Sciences Library, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA
| | - Tanha Patel
- North Carolina Translational and Clinical Institute, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA
- School of Medicine, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA
| | - Andrea Carnegie
- North Carolina Translational and Clinical Institute, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA
- School of Medicine, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA
| | - Gaurav Dave
- North Carolina Translational and Clinical Institute, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA
- School of Medicine, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA
| |
Collapse
|
3
|
Lumsden JM, Urv TK. The Rare Diseases Clinical Research Network: a model for clinical trial readiness. THERAPEUTIC ADVANCES IN RARE DISEASE 2023; 4:26330040231219272. [PMID: 38152157 PMCID: PMC10752072 DOI: 10.1177/26330040231219272] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 08/07/2023] [Accepted: 11/15/2023] [Indexed: 12/29/2023]
Abstract
Background The current road to developing treatments for rare diseases is often slow, expensive, and riddled with risk. Change is needed to improve the process, both in how we think about rare disease treatment development and the infrastructure we build to support ongoing science. The National Institutes of Health (NIH)-supported Rare Diseases Clinical Research Network (RDCRN) was established to advance the diagnosis, management, and treatment of rare diseases and to promote highly collaborative, multi-site, patient-centric, translational, and clinical research. The current iteration of the RDCRN intends to build upon and enhance successful approaches within the network while identifying innovative methods to fill gaps and address needs in the approach to the rare disease treatment development process through innovation, collaboration, and clinical trial readiness. Objective The objective of this paper is to provide an overview of the productivity and influence of the RDCRN since it was first established 20 years ago. Design and methods Using a suite of tools available to NIH staff that provides access to a comprehensive, curated, extensively linked data set of global grants, patents, publications, clinical trials, and FDA-approved drugs, a series of queries were executed that conducted bibliometric, co-author, and co-occurrence analysis. Results The results demonstrate that the entire RDCRN consortia and network has been highly productive since its inception. They have produced 2763 high-quality publications that have been cited more than 100,000 times, expanded international networks, and contributed scientifically to eight FDA-approved treatments for rare diseases. Conclusion The RDCRN program has successfully addressed some significant challenges while developing treatments for rare diseases. However, looking to the future and being agile in facing new challenges that arise as science progresses is important.
Collapse
Affiliation(s)
- Joanne M. Lumsden
- Division of Rare Diseases Research Innovation, National Center for Advancing Translational Sciences, National Institutes of Health, 6801 Democracy Boulevard, Bethesda, MD 20892-0001, USA
| | - Tiina K. Urv
- Division of Rare Diseases Research Innovation, National Center for Advancing Translational Sciences, National Institutes of Health, Bethesda, MD, USA
| |
Collapse
|
4
|
Park SM, Vonortas NS. Translational research: from basic research to regional biomedical entrepreneurship. SMALL BUSINESS ECONOMICS 2022; 60:1761-1783. [PMID: 38625332 PMCID: PMC9425788 DOI: 10.1007/s11187-022-00676-9] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Accepted: 08/02/2022] [Indexed: 04/17/2024]
Abstract
This paper examines the effect of translational research on knowledge production and biomedical entrepreneurship across U.S. regions. Researchers have earlier investigated the outputs of translational research by focusing on academic publications. Little attention has been paid to linking translational research to biomedical entrepreneurship. We construct an analytical model based on the knowledge spillover theory of entrepreneurship and the entrepreneurial ecosystem approach to examine the relationship between translational research, biomedical patents, clinical trials, and biomedical entrepreneurship. We test the model across 381 U.S. metropolitan statistical areas using 10 years of panel data related to the NIH Clinical and Translational Science Awards (CTSA) program. CTSA appears to increase the number of biomedical patents and biomedical entrepreneurship as proxied by the NIH Small Business Innovation Research (SBIR) grants. However, the magnitudes of the effects are relatively small. Path analysis shows that the effect of translational research on regional biomedical entrepreneurship is not strongly conveyed through biomedical patents or clinical trials.
Collapse
Affiliation(s)
- Sang-Min Park
- Science, Technology and Innovation Support Team, Ministry of Science and ICT, Sejong Finance Center II, 194, Gareum-ro, Sejong-si, 30121 South Korea
| | - Nicholas S. Vonortas
- Institute for International Science and Technology Policy & Department of Economics, George Washington University, 1957 E Street NW Suite 403, Washington DC, 20052 USA
- São Paulo Excellence Chair, University of Campinas, Campinas, Brazil
| |
Collapse
|
5
|
Padilla-Cabello J, Santisteban-Espejo A, Heradio R, Cobo MJ, Martin-Piedra MA, Moral-Munoz JA. Methods for identifying biomedical translation: a systematic review. Am J Transl Res 2022; 14:2697-2708. [PMID: 35559386 PMCID: PMC9091120] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2021] [Accepted: 12/30/2021] [Indexed: 06/15/2023]
Abstract
Translational medicine is an important area of biomedicine, and has significantly facilitated the development of biomedical research. Despite its relevance, there is no consensus on how to evaluate its progress and impact. A systematic review was carried out to identify all the methods to evaluate translational research. Seven methods were found according to the established criteria to analyze their characteristics, advantages, and limitations. They allow us to perform this type of evaluation in different ways. No relevant advantages were found between them; each one presented its specific limitations that need to be considered. Nevertheless, the Triangle of Biomedicine could be considered the most relevant method, concerning the time since its publication and usefulness. In conclusion, there is still a lack of a gold-standard method for evaluating biomedical translational research.
Collapse
Affiliation(s)
- Javier Padilla-Cabello
- Program of Biomedicine, University of GranadaGranada, Spain
- Department of Otorhinolaryngology, Hospital Universitario Clínico San CecilioGranada, Spain
| | - Antonio Santisteban-Espejo
- Department of Pathology, Puerta del Mar University HospitalCadiz, Spain
- Institute of Research and Innovation in Biomedical Sciences of The Province of Cadiz (INiBICA)Cadiz, Spain
| | - Ruben Heradio
- School of Computer Science, Universidad Nacional de Educación a Distancia (UNED)Madrid, Spain
| | - Manuel J Cobo
- Department of Computer Science and Engineering, University of CadizPuerto Real, Spain
| | | | - Jose A Moral-Munoz
- Institute of Research and Innovation in Biomedical Sciences of The Province of Cadiz (INiBICA)Cadiz, Spain
- Department of Nursing and Physiotherapy, University of CadizCadiz, Spain
| |
Collapse
|
6
|
Kiparoglou V, Brown LA, McShane H, Channon KM, Shah SGS. A large National Institute for Health Research (NIHR) Biomedical Research Centre facilitates impactful cross-disciplinary and collaborative translational research publications and research collaboration networks: a bibliometric evaluation study. J Transl Med 2021; 19:483. [PMID: 34838033 PMCID: PMC8626935 DOI: 10.1186/s12967-021-03149-x] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/04/2021] [Accepted: 11/15/2021] [Indexed: 11/10/2022] Open
Abstract
Background The evaluation of translational health research is important for various reasons such as the research impact assessment, research funding allocation, accountability, and strategic research policy formulation. The purpose of this study was to evaluate the research productivity, strength and diversity of research collaboration networks and impact of research supported by a large biomedical research centre in the United Kingdom (UK). Methods Bibliometric analysis of research publications by translational researchers affiliated with the National Institute for Health Research (NIHR) Oxford Biomedical Research Centre (BRC) from April 2012 to March 2017. Results Analysis included 2377 translational research publications that were published during the second 5-year funding period of the NIHR Oxford BRC. Author details were available for 99.75% of the publications with DOIs (2359 of 2365 with DOIs), and the number of authors per publication was median 9 (mean = 18.03, SD = 3.63, maximum = 2467 authors). Author lists also contained many consortia, groups, committees, and teams (n = 165 in total), with 1238 additional contributors, where membership was reported. The BRC co-authorship i.e., research collaboration network for these publications involved 20,229 nodes (authors, of which 1606 nodes had Oxford affiliations), and approximately 4.3 million edges (authorship linkages). Articles with a valid DOIs (2365 of 2377, 99.5%) were collectively cited more than 155,000 times and the average Field Citation Ratio was median 6.75 (geometric mean = 7.12) while the average Relative Citation Ratio was median 1.50 (geometric mean = 1.83) for the analysed publications. Conclusions The NIHR Oxford BRC generated substantial translational research publications and facilitated a huge collaborative network of translational researchers working in complex structures and consortia, which shows success across the whole of this BRC funding period. Further research involving continued uptake of unique persistent identifiers and the tracking of other research outputs such as clinical innovations and patents would allow a more detailed understanding of large research enterprises such as NIHR BRCs in the UK. Supplementary Information The online version contains supplementary material available at 10.1186/s12967-021-03149-x.
Collapse
Affiliation(s)
- Vasiliki Kiparoglou
- NIHR Oxford Biomedical Research Centre, John Radcliffe Hospital, Oxford University Hospitals NHS Foundation Trust, Oxford, OX3 9DU, UK.,Nuffield Department of Primary Care Health Sciences, University of Oxford, Radcliffe Primary Care Building, Woodstock Road, Oxford, OX2 6GG, UK
| | - Laurence A Brown
- Research Support Team, IT Services, University of Oxford, Oxford, OX1 2JD, UK.,Nuffield Department of Clinical Neurosciences, John Radcliffe Hospital, University of Oxford, Oxford, OX3 9DU, UK
| | - Helen McShane
- NIHR Oxford Biomedical Research Centre, John Radcliffe Hospital, Oxford University Hospitals NHS Foundation Trust, Oxford, OX3 9DU, UK.,Nuffield Department of Medicine, The Jenner Institute, University of Oxford, Old Road Campus, Oxford, OX3 7BN, UK
| | - Keith M Channon
- NIHR Oxford Biomedical Research Centre, John Radcliffe Hospital, Oxford University Hospitals NHS Foundation Trust, Oxford, OX3 9DU, UK.,Division of Cardiovascular Medicine, British Heart Foundation (BHF) Centre of Research Excellence, John Radcliffe Hospital, University of Oxford, Oxford, OX3 9DU, UK
| | - Syed Ghulam Sarwar Shah
- NIHR Oxford Biomedical Research Centre, John Radcliffe Hospital, Oxford University Hospitals NHS Foundation Trust, Oxford, OX3 9DU, UK. .,Radcliffe Department of Medicine, Medical Sciences Division, John Radcliffe Hospital, University of Oxford, Oxford, OX3 9DU, UK.
| |
Collapse
|
7
|
Bragg KM, Marchand GC, Hilpert JC, Cummings JL. Using bibliometrics to evaluate outcomes and influence of translational biomedical research centers. J Clin Transl Sci 2021; 6:e72. [PMID: 35836786 PMCID: PMC9257775 DOI: 10.1017/cts.2021.863] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2021] [Revised: 09/24/2021] [Accepted: 09/27/2021] [Indexed: 11/24/2022] Open
Abstract
Introduction Federal grant funding to support infrastructure development of translational biomedical research centers is a form of public health intervention. Establishing rigorous methods for measuring center success and outcomes is essential to justify continued funding. Methods Bibliometric data compiled from a 5-year funding cycle of neurodegeneration and translational neuroscience research center were analyzed using the package bibliometrix for open-source software R and the NIH-developed research tool iCite. Results The research team and their collaborators (n = 485) produced 157 grant-citing publications from 2015-2020. The science was produced by small research teams clustered around three main communities of topics: Alzheimer's Disease, brain imaging, and neuropsychological testing in the elderly. Using the relative citation ratio, the publications produced by the research team were found to be influential when compared to other R01-funded publications. Conclusion Recent developments in bibliometric analysis expand beyond traditional measurement capabilities to better understand the characteristics, outcomes, and influences of research teams. These findings can be used to inform researchers and institutions about research team composition, productivity, and success. Measures of research influence may be used to justify return on investment to funders.
Collapse
Affiliation(s)
- Kristine M. Bragg
- Department of Educational Psychology and Higher Education, College of Education, University of Nevada, Las Vegas, Las Vegas, NV, USA
| | - Gwen C. Marchand
- Department of Educational Psychology and Higher Education, College of Education, University of Nevada, Las Vegas, Las Vegas, NV, USA
| | - Jonathan C. Hilpert
- Department of Educational Psychology and Higher Education, College of Education, University of Nevada, Las Vegas, Las Vegas, NV, USA
| | - Jeffrey L. Cummings
- Chambers-Grundy Center for Transformative Neuroscience, Department of Brain Health, School of Integrated Health Sciences, University of Nevada, Las Vegas, Las Vegas, NV, USA
| |
Collapse
|
8
|
Qua K, Yu F, Patel T, Dave G, Cornelius K, Pelfrey CM. Scholarly Productivity Evaluation of KL2 Scholars Using Bibliometrics and Federal Follow-on Funding: Cross-Institution Study. J Med Internet Res 2021; 23:e29239. [PMID: 34586077 PMCID: PMC8515229 DOI: 10.2196/29239] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2021] [Revised: 07/04/2021] [Accepted: 07/27/2021] [Indexed: 11/13/2022] Open
Abstract
Background Evaluating outcomes of the clinical and translational research (CTR) training of a Clinical and Translational Science Award (CTSA) hub (eg, the KL2 program) requires the selection of reliable, accessible, and standardized measures. As measures of scholarly success usually focus on publication output and extramural funding, CTSA hubs have started to use bibliometrics to evaluate the impact of their supported scholarly activities. However, the evaluation of KL2 programs across CTSAs is limited, and the use of bibliometrics and follow-on funding is minimal. Objective This study seeks to evaluate scholarly productivity, impact, and collaboration using bibliometrics and federal follow-on funding of KL2 scholars from 3 CTSA hubs and to define and assess CTR training success indicators. Methods The sample included KL2 scholars from 3 CTSA institutions (A-C). Bibliometric data for each scholar in the sample were collected from both SciVal and iCite, including scholarly productivity, citation impact, and research collaboration. Three federal follow-on funding measures (at the 5-year, 8-year, and overall time points) were collected internally and confirmed by examining a federal funding database. Both descriptive and inferential statistical analyses were computed using SPSS to assess the bibliometric and federal follow-on funding results. Results A total of 143 KL2 scholars were included in the sample with relatively equal groups across the 3 CTSA institutions. The included KL2 scholars produced more publications and citation counts per year on average at the 8-year time point (3.75 publications and 26.44 citation counts) than the 5-year time point (3.4 publications vs 26.16 citation counts). Overall, the KL2 publications from all 3 institutions were cited twice as much as others in their fields based on the relative citation ratio. KL2 scholars published work with researchers from other US institutions over 2 times (5-year time point) or 3.5 times (8-year time point) more than others in their research fields. Within 5 years and 8 years postmatriculation, 44.1% (63/143) and 51.7% (74/143) of KL2 scholars achieved federal funding, respectively. The KL2-scholars of Institution C had a significantly higher citation rate per publication than the other institutions (P<.001). Institution A had a significantly lower rate of nationally field-weighted collaboration than did the other institutions (P<.001). Institution B scholars were more likely to have received federal funding than scholars at Institution A or C (P<.001). Conclusions Multi-institutional data showed a high level of scholarly productivity, impact, collaboration, and federal follow-on funding achieved by KL2 scholars. This study provides insights on the use of bibliometric and federal follow-on funding data to evaluate CTR training success across institutions. CTSA KL2 programs and other CTR career training programs can benefit from these findings in terms of understanding metrics of career success and using that knowledge to develop highly targeted strategies to support early-stage career development of CTR investigators.
Collapse
Affiliation(s)
- Kelli Qua
- Clinical and Translational Science Collaborative, School of Medicine, Case Western Reserve University, Cleveland, OH, United States
| | - Fei Yu
- Health Sciences Library, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States.,School of Information and Library Science, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States
| | - Tanha Patel
- North Carolina Translational and Clinical Sciences Institute, School of Medicine, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States
| | - Gaurav Dave
- North Carolina Translational and Clinical Sciences Institute, School of Medicine, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States.,Division of General Medicine and Clinical Epidemiology, School of Medicine, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States
| | - Katherine Cornelius
- Center for Clinical and Translational Science, Mayo Clinic, Rochester, MN, United States
| | - Clara M Pelfrey
- Clinical and Translational Science Collaborative, School of Medicine, Case Western Reserve University, Cleveland, OH, United States
| |
Collapse
|
9
|
Myers BA, Kahn KL. Practical publication metrics for academics. Clin Transl Sci 2021; 14:1705-1712. [PMID: 33982433 PMCID: PMC8504821 DOI: 10.1111/cts.13067] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2021] [Revised: 04/09/2021] [Accepted: 04/15/2021] [Indexed: 02/06/2023] Open
Abstract
Research organizations are becoming more reliant on quantitative approaches to determine how to recruit and promote researchers, allocate funding, and evaluate the impact of prior allocations. Many of these quantitative metrics are based on research publications. Publication metrics are not only important for individual careers, but also affect the progress of science as a whole via their role in the funding award process. Understanding the origin and intended use of popular publication metrics can inform an evaluative strategy that balances the usefulness of publication metrics with the limitations of what they can convey about the productivity and quality of an author, a publication, or a journal. This paper serves as a brief introduction to citation networks like Google Scholar, Web of Science Core Collection, Scopus, Microsoft Academic, and Dimensions. It also explains two of the most popular publication metrics: the h-index and the journal impact factor. The purpose of this paper is to provide practical information on using citation networks to generate publication metrics, and to discuss ideas for contextualizing and juxtaposing metrics, in order to help researchers in translational science and other disciplines document their impact in as favorable a light as may be justified.
Collapse
Affiliation(s)
- Bethany A Myers
- Louise M. Darling Biomedical Library, University of California, Los Angeles, California, USA
| | - Katherine L Kahn
- Division of General Internal Medicine and Health Services Research, David Geffen School of Medicine, University of California, Los Angeles, California, USA
| |
Collapse
|
10
|
KIM YH, LEVINE AD, NEHL EJ, WALSH JP. A Bibliometric Measure of Translational Science. Scientometrics 2020; 125:2349-2382. [PMID: 33746311 PMCID: PMC7968388 DOI: 10.1007/s11192-020-03668-2] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2020] [Indexed: 12/18/2022]
Abstract
Science funders are increasingly requiring evidence of the broader impacts of even basic research. Initiatives such as NIH's CTSA program are designed to shift the research focus toward more translational research. However, tracking the effectiveness of such programs depends on developing indicators that can track the degree to which basic research is influencing clinical research. We propose a new bibliometric indicator, the TS score, that is relatively simple to calculate, can be implemented at scale, is easy to replicate, and has good reliability and validity properties. This indicator is broadly applicable in settings where the goal is to estimate the degree to which basic research is used in more applied downstream research, relative to use in basic research. The TS score should be of use for a variety of policy analysis and research evaluation purposes.
Collapse
Affiliation(s)
- Yeon Hak KIM
- Ministry of Science and ICT, Sejong, Republic of Korea
| | - Aaron D. LEVINE
- School of Public Policy, Georgia Institute of Technology, Atlanta, GA USA
| | - Eric J. NEHL
- Rollins School of Public Health, Emory University, Atlanta, GA USA
| | - John P. WALSH
- School of Public Policy, Georgia Institute of Technology, Atlanta, GA USA)
| |
Collapse
|
11
|
Llewellyn N, Carter DR, DiazGranados D, Pelfrey C, Rollins L, Nehl EJ. Scope, Influence, and Interdisciplinary Collaboration: The Publication Portfolio of the NIH Clinical and Translational Science Awards (CTSA) Program From 2006 Through 2017. Eval Health Prof 2020; 43:169-179. [PMID: 30917690 PMCID: PMC7781230 DOI: 10.1177/0163278719839435] [Citation(s) in RCA: 16] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
The Clinical and Translational Science Awards (CTSA) program sponsors an array of innovative, collaborative research. This study uses complementary bibliometric approaches to assess the scope, influence, and interdisciplinary collaboration of publications supported by single CTSA hubs and those supported by multiple hubs. Authors identified articles acknowledging CTSA support and assessed the disciplinary scope of research areas represented in that publication portfolio, their citation influence, interdisciplinary overlap among research categories, and characteristics of publications supported by multihub collaborations. Since 2006, CTSA hubs supported 69,436 articles published in 4,927 journals and 189 research areas. The portfolio is well distributed across diverse research areas with above-average citation influence. Most supported publications involved clinical/health sciences, for example, neurology and pediatrics; life sciences, for example, neuroscience and immunology; or a combination of the two. Publications supported by multihub collaborations had distinct content emphasis, stronger citation influence, and greater interdisciplinary overlap. This study characterizes the CTSA consortium's contributions to clinical and translational science, identifies content areas of strength, and provides evidence for the success of multihub collaborations. These methods lay the foundation for future investigation of the best policies and priorities for fostering translational science and allow hubs to understand their progress benchmarked against the larger consortium.
Collapse
Affiliation(s)
- Nicole Llewellyn
- Georgia Clinical & Translational Science Alliance, Emory University School of Medicine, Atlanta, GA, USA
| | | | - Deborah DiazGranados
- School of Medicine, Virginia Commonwealth University, Richmond, VA, USA
- Wright Center for Clinical and Translational Research, Virginia Commonwealth University, Richmond, VA, USA
| | - Clara Pelfrey
- Department of Medical Education, Case Western Reserve University, Cleveland, OH, USA
| | - Latrice Rollins
- Prevention Research Center, Morehouse School of Medicine, Atlanta, GA, USA
| | - Eric J. Nehl
- Rollins School of Public Health, Emory University, Atlanta, GA, USA
| |
Collapse
|
12
|
Using bibliometrics to evaluate translational science training: evidence for early career success of KL2 scholars. J Clin Transl Sci 2020; 5:e24. [PMID: 33948247 PMCID: PMC8057473 DOI: 10.1017/cts.2020.516] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/26/2022] Open
Abstract
Introduction: Evaluating clinical and translational research (CTR) mentored training programs is challenging because no two programs are alike. Careful selection of appropriate metrics is required to make valid comparisons between individuals and between programs. The KL2 program provides mentored-training for early-stage CTR investigators. Clinical and Translational Awards across the country have unique KL2 programs. The evaluation of KL2 programs has begun to incorporate bibliometrics to measure KL2 scholar and program impact. Methods: This study investigated demographic differences in bibliometric performance and post-K award funding of KL2 scholars and compared the bibliometric performance and post-K award federal funding of KL2 scholars and other mentored-K awardees at the same institution. Data for this study included SciVal and iCite bibliometrics and National Institutions of Health RePORTER grant information for mentored-K awardees (K08, K23, and KL2) at Case Western Reserve University between 2005 and 2013. Results: Results showed no demographics differences within the KL2 program scholars. Bibliometric differences between KL2 and other mentored-K awardee indicated an initial KL2 advantage for the number of publications at 5 years’ post-matriculation (i.e., the start of the K award). Regression analyses indicated the number of initial publications was a significant predictor of federal grant funding at the same time point. Analysis beyond the 5-year post-matriculation point did not result in a sustained, significant KL2 advantage. Conclusions: Factors that contributed to the grant funding advantage need to be determined. Additionally, differences between translational and clinical bibliometrics must be interpreted with caution, and appropriate metrics for translational science must be established.
Collapse
|
13
|
Scientometric analysis of the Puerto Rico Clinical and Translational Research Consortium (PRCTRC) research publications, 2010-2018. J Clin Transl Sci 2020; 5:e4. [PMID: 33948234 PMCID: PMC8057457 DOI: 10.1017/cts.2020.43] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022] Open
Abstract
We analyzed the publication productivity supported by the Puerto Rico Consortium for Clinical and Translational Research (PRCTRC) using the structured process of scientometrics. The objective of this study was to evaluate the impact of the research and collaborations as presented in publications. Manuscripts published from 2010 to 2018 and that had the PRCTRC award number and a PMCID number were retrieved from the Science Citation Index database. Scientometric indicators included h-index (HI), average citation (AC), collaboration coefficient (CC), collaboration index (CI), and degree of collaboration (DC) analysis, and relative citation ratio (RCR) was done with Web of Science Platform, iCite, and Stata software. Joinpoint Trend Analysis Software was used to calculate the annual percent change (APC). From 2010 to 2018, 341 publications were identified with an average of 38 publications per year and a total of 3569 citations excluding self-citations. A significant growth (APC: 17.76%, P < 0.05) of scientific production was observed. The overall HI was 31, and the AC per item was 11.04. The overall CC was 0.82, the CI was 8.59, and the DC was 99.1%. This study demonstrates a statistically significant increase in the PRCTRC scientific production. Results allow for the assessment of the progress resulting from the provided support and to plan further strategies accordingly.
Collapse
|
14
|
Yu F, Van AA, Patel T, Mani N, Carnegie A, Corbie-Smith GM, Carey T, Buse J, Dave G. Bibliometrics approach to evaluating the research impact of CTSAs: A pilot study. J Clin Transl Sci 2020; 4:336-344. [PMID: 33244415 PMCID: PMC7681148 DOI: 10.1017/cts.2020.29] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/14/2020] [Revised: 02/25/2020] [Accepted: 03/24/2020] [Indexed: 12/24/2022] Open
Abstract
INTRODUCTION To enhance the performance evaluation of Clinical and Translational Science Award (CTSA) hubs, we examined the utility of advanced bibliometric measures that go beyond simple publication counts to demonstrate the impact of translational research output. METHODS The sampled data included North Carolina Translational and Clinical Science Institute (NC TraCS)-supported publications produced between September 2008 and March 2017. We adopted advanced bibliometric measures and a state-of-the-art bibliometric network analysis tool to assess research productivity, citation impact, the scope of research collaboration, and the clusters of research topics. RESULTS Totally, 754 NC TraCS-supported publications generated over 24,000 citation counts by April 2017 with an average of 33 cites per article. NC TraCS-supported research papers received more than twice as many cites per year as the average National Institute of Health-funded research publications from the same field and time. We identified the top productive researchers and their networks within the CTSA hub. Findings demonstrated the impact of NC TraCS in facilitating interdisciplinary collaborations within the CTSA hub and across the CTSA consortium and connecting researchers with right peers and organizations. CONCLUSION Both improved bibliometrics measures and bibliometric network analysis can bring new perspectives to CTSA evaluation via citation influence and the scope of research collaborations.
Collapse
Affiliation(s)
- Fei Yu
- Health Sciences Library, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Allison Alicia Van
- North Carolina Translational and Clinical Institute, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Tanha Patel
- North Carolina Translational and Clinical Institute, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
- School of Medicine, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Nandita Mani
- Health Sciences Library, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Andrea Carnegie
- North Carolina Translational and Clinical Institute, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
- School of Medicine, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Giselle M. Corbie-Smith
- North Carolina Translational and Clinical Institute, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
- School of Medicine, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Timothy Carey
- North Carolina Translational and Clinical Institute, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
- School of Medicine, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - John Buse
- North Carolina Translational and Clinical Institute, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
- School of Medicine, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Gaurav Dave
- North Carolina Translational and Clinical Institute, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
- School of Medicine, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| |
Collapse
|
15
|
Bibliometric Analysis of Electronic Cigarette Publications: 2003⁻2018. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2019; 16:ijerph16030320. [PMID: 30682767 PMCID: PMC6388121 DOI: 10.3390/ijerph16030320] [Citation(s) in RCA: 21] [Impact Index Per Article: 4.2] [Reference Citation Analysis] [Abstract] [Key Words] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 10/09/2018] [Revised: 01/18/2019] [Accepted: 01/20/2019] [Indexed: 12/15/2022]
Abstract
Electronic cigarettes are at the center of a public health policy debate which leverages scientific publications. This study characterizes e-cigarette publication trends over the past 15 years via a bibliometric analysis. Scopus was searched for “electronic cigarette”, “e-cig”, “e-cigarette”, “vape”, “vaping”, “juul”, or “electronic nicotine delivery system” between 2003–2018. Data included Hirsch index, document type and frequency, and publications by institution, journal, and country. VOSviewer was used to visualize authorship network maps. A total of 4490 e-cigarette publications were identified, most (62.8%) being articles. After 2009, the annual growth rate for e-cigarette publications was the largest in 2014. The annual growth rate was nearly flat in 2017 but increased in 2018. The U.S. produced 51.6% of publications. Annual National Institutes of Health NIH funding for tobacco research mapped closely with the annual volume of e-cigarette publications. Author network analyses illustrated investigator collaborative patterns. The frequency of e-cigarette publications increased significantly in the past decade. A strong relationship of NIH funding for tobacco research and e-cigarette publications demonstrates the importance of e-cigarettes in tobacco research.
Collapse
|
16
|
Surkis A, Spore S. The relative citation ratio: what is it and why should medical librarians care? J Med Libr Assoc 2018; 106:508-513. [PMID: 30271298 PMCID: PMC6148595 DOI: 10.5195/jmla.2018.499] [Citation(s) in RCA: 25] [Impact Index Per Article: 4.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/01/2018] [Accepted: 05/01/2018] [Indexed: 12/05/2022] Open
Abstract
Bibliometrics is becoming increasingly prominent in the world of medical libraries. The number of presentations related to research impact at the Medical Library Association (MLA) annual meeting has been increasing in past years. Medical centers have been using institutional dashboards to track clinical performance for over a decade, and more recently, these institutional dashboards have included measures of academic performance. This commentary reviews current practices and considers the role for a newer metric, the relative citation ratio.
Collapse
Affiliation(s)
- Alisa Surkis
- Assistant Director for Research Data and Metrics, Health Sciences Library, NYU, 577 1st Avenue, New York, NY 10016
| | - Stuart Spore
- Lead, Scholarly Output Assessment, Health Sciences Library, NYU, 577 1st Avenue, New York, NY 10016
| |
Collapse
|
17
|
Llewellyn N, Carter DR, Rollins L, Nehl EJ. Charting the Publication and Citation Impact of the NIH Clinical and Translational Science Awards (CTSA) Program From 2006 Through 2016. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2018; 93:1162-1170. [PMID: 29298181 PMCID: PMC6028299 DOI: 10.1097/acm.0000000000002119] [Citation(s) in RCA: 21] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/07/2023]
Abstract
PURPOSE The authors evaluated publication and citation patterns for articles supported by Clinical and Translational Science Awards (CTSA) hub investment over the first decade of the CTSA program. The aim was to elucidate a pivotal step in the translational process by providing an account of how time, hub maturity, and hub attributes were related to productivity and influence in the academic literature. METHOD In 2017, the authors collected bibliometric data from PubMed, Web of Science InCites, and National Institutes of Health (NIH) iCite for articles citing any CTSA hub grants published from hub inception through 2016. They compiled data on publication and citation rates and indices of relative citation impact aggregated by hub funding year cohort. They compared hub-level bibliometric activity by multi- versus single-institution structure and total monetary award sums, compiled from NIH RePORTER. RESULTS From 2006-2016, CTSA hubs supported over 66,000 publications, with publication rates accelerating as hubs matured. These publications accumulated over 1.2 million citations, with some articles cited over 1,000 times. Indices of relative citation impact indicated CTSA-supported publications were cited more than twice as often as expected for articles of their publication years and disciplines. Multi-institutional hubs and those awarded higher grant sums exhibited significantly higher publication and citation activity. CONCLUSIONS The CTSA program is yielding a robust and growing body of influential research findings with consistently high indices of relative citation impact. Preliminary evidence suggests multi-institutional collaborations and more monetary resources are associated with elevated bibliometric activity and, therefore, may be worth their investment.
Collapse
Affiliation(s)
- Nicole Llewellyn
- N. Llewellyn is manager of research projects, Evaluation and Continuous Improvement Program, Georgia Clinical & Translational Science Alliance, Emory University School of Medicine, Atlanta, Georgia; ORCID: https://orcid.org/0000-0003-1267-2720. D.R. Carter is assistant professor, Department of Psychology, University of Georgia, Athens, Georgia, and member, Evaluation and Continuous Improvement Program, Georgia Clinical & Translational Science Alliance, Emory University School of Medicine, Atlanta, Georgia; ORCID: https://orcid.org/0000-0001-8480-1996. L. Rollins is assistant director of evaluation and institutional assessment, Prevention Research Center, Morehouse School of Medicine, and member, Evaluation and Continuous Improvement Program, Georgia Clinical & Translational Science Alliance, Emory University School of Medicine, Atlanta, Georgia. E.J. Nehl is assistant research professor, Emory University Rollins School of Public Health, and director, Evaluation and Continuous Improvement Program, Georgia Clinical & Translational Science Alliance, Emory University School of Medicine, Atlanta, Georgia
| | | | | | | |
Collapse
|
18
|
Han X, Williams SR, Zuckerman BL. A snapshot of translational research funded by the National Institutes of Health (NIH): A case study using behavioral and social science research awards and Clinical and Translational Science Awards funded publications. PLoS One 2018; 13:e0196545. [PMID: 29742129 PMCID: PMC5942790 DOI: 10.1371/journal.pone.0196545] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/23/2018] [Accepted: 04/14/2018] [Indexed: 11/24/2022] Open
Abstract
The translation of biomedical research from basic knowledge to application has been a priority at the National Institute of Health (NIH) for many years. Tracking the progress of scientific research and knowledge through the translational process is difficult due to variation in the definition of translational research as well as the identification of benchmarks for the spread and application of biomedical research; quantitatively tracking this process is even more difficult. Using a simple and reproducible method to assess whether publications are translational, we examined NIH R01 behavioral and social science research (BSSR) awards funded between 2008 and 2014 to determine whether there are differences in the percent of translational research publications produced by basic and applied research awards. We also assessed the percent of translational research publications produced by the Clinical and Translational Science Awards (CTSA) program to evaluate whether targeted translational research awards result in increased translational research. We found that 3.9% of publications produced by basic research awards were translational; that the percent of translational research publications produced by applied research awards is approximately double that of basic research awards (7.4%); and that targeted translational research awards from the CTSA program produced the highest percentage of translational research publications (13.4%). In addition, we assessed differences in time to first publication, time to first citation, and publication quality by award type (basic vs. applied), and whether an award (or publication) is translational.
Collapse
Affiliation(s)
- Xueying Han
- Science and Technology Policy Institute, Washington DC, United States of America
- * E-mail:
| | - Sharon R. Williams
- Science and Technology Policy Institute, Washington DC, United States of America
| | - Brian L. Zuckerman
- Science and Technology Policy Institute, Washington DC, United States of America
| |
Collapse
|
19
|
Using publication data to evaluate a Clinical and Translational Science Award (CTSA) career development program: Early outcomes from KL2 scholars. J Clin Transl Sci 2018; 1:352-360. [PMID: 29707257 PMCID: PMC5915801 DOI: 10.1017/cts.2018.1] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022] Open
Abstract
Introduction This study uses KL2 scholars' publications to evaluate the types of research the KL2 program supports and to assess the initial productivity and impact of its scholars. Methods We illustrate the feasibility of 3 different approaches to bibliometrics, one viable method for determining the types of research a program or hub supports, and demonstrate how these data can be further combined with internal data records. Results Gender differences were observed in the types of research scholars undertake. Overall KL2 scholars are performing well, with their publications being cited more than the norm for National Institutes of Health publications. Favorable results were also observed in scholars' continued engagement in research. Conclusion This study illustrates that linking bibliometric data and data categorizing publications along the translational spectrum with a Clinical and Translational Science Award hub's internal data records is feasible and offers a number of innovative possibilities for the evaluation of a Clinical and Translational Science Award hub's programs and investigators.
Collapse
|
20
|
|