Predicting translational progress in biomedical research

What predicts citation counts and translational impact in headache research? A machine learning analysis

BACKGROUND

We aimed to develop the first machine learning models to predict citation counts and the translational impact, defined as inclusion in guidelines or policy documents, of headache research, and assess which factors are most predictive.

METHODS

Bibliometric data and the titles, abstracts, and keywords from 8600 publications in three headache-oriented journals from their inception to 31 December 2017 were used. A series of machine learning models were implemented to predict three classes of 5-year citation count intervals (0-5, 6-14 and, >14 citations); and the translational impact of a publication. Models were evaluated out-of-sample with area under the receiver operating characteristics curve (AUC).

RESULTS

The top performing gradient boosting model predicted correct citation count class with an out-of-sample AUC of 0.81. Bibliometric data such as page count, number of references, first and last author citation counts and h-index were among the most important predictors. Prediction of translational impact worked optimally when including both bibliometric data and information from the title, abstract and keywords, reaching an out-of-sample AUC of 0.71 for the top performing random forest model.

CONCLUSION

Citation counts are best predicted by bibliometric data, while models incorporating both bibliometric data and publication content identifies the translational impact of headache research.

Illuminating the druggable genome: Pathways to progress

There are ∼4500 genes within the 'druggable genome', the subset of the human genome that expresses proteins able to bind drug-like molecules, yet existing drugs only target a few hundred. A substantial subset of druggable proteins are largely uncharacterized or understudied, with many falling within G protein-coupled receptor (GPCR), ion channel, and kinase protein families. To improve scientific understanding of these three understudied protein families, the US National Institutes of Health launched the Illuminating the Druggable Genome Program. Now, as the program draws to a close, this review will lay out resources developed by the program that are intended to equip the scientific community with the tools necessary to explore previously understudied biology with the potential to rapidly impact human health.

Translation in action: Influence, collaboration, and evolution of COVID-19 research with Clinical and Translational Science Awards consortium support

The National Institutes of Health (NIH)'s Clinical and Translational Science Awards (CTSA) consortium aims to accelerate translational processes that move discoveries from bench to bedside. The coronavirus disease 2019 (COVID-19) pandemic presented unmatched challenges and applications for CTSA hubs nationwide. Our study used bibliometrics to assess features of COVID-19 publications supported by the national CTSA program to characterize the consortium's response to the pandemic. Our goal was to understand relative scientific influence, collaboration across hubs, and trends in research emphasis over time. We identified publications from NIH's curated iSearch COVID-19 Publication Portfolio from February 2020 to February 2023; 3234 peer-reviewed articles relevant to COVID-19 cited a CTSA grant. All 66 CTSA hubs were represented, with large-size and longstanding hubs contributing more publications. Most publications cited UL1 grants, 457 cited KL2/TL1 training grants, and 164 cited multiple hub grants. Compared to a random sample of non-CTSA-supported COVID-19 publications, the CTSA portfolio exhibited greater clinical relevance, more human research, and higher altmetric and citation influence. Results were similar for multi-hub publications involving networked initiatives like multi-site clinical trials or the National COVID-19 Cohort Collaborative. Shifts from molecular/cellular-oriented research toward human-oriented research over time were evident, demonstrating translation in action. Results illuminate how the CTSA consortium confronted the pandemic through high-quality projects oriented toward human research, working across hubs on high-value collaborations, advancing along the translational spectrum over time. Findings validate CTSA hubs as critical support structures during health emergencies.

Current landscape of treating different cancers using nanomedicines: Trends and perspectives

The efforts to use novel nanotechnologies in medicine and cancer have been widespread. In order to understand better the focus areas of cancer nanomedicine research to date, we conducted a survey of nanomedicine developmental and clinical research in conjunction with treatment of various cancers. The survey has been performed based on number of publications, rate of citations, entry into clinical trials, and funding rates by the National Cancer Institute. Our survey indicates that breast and brain cancers are the most and one of the least studied by nanotechnology researchers, respectively. Breast cancer nano-therapies seem to also be most likely to achieve clinical translation as the number of publications produced, amount of funding, total citations, and clinical trials (active and completed) are the highest when compared with research in other cancers. Brain cancer, despite its low survival, has capture much less attention of nanomedicine research community as survey indicated, although nanotechnology can offer novel approaches which can address brain cancer challenges. This article is categorized under: Therapeutic Approaches and Drug Discovery > Nanomedicine for Oncologic Disease.

Analysis of cognitive framework and biomedical translation of tissue engineering in otolaryngology

Tissue engineering is a relatively recent research area aimed at developing artificial tissues that can restore, maintain, or even improve the anatomical and/or functional integrity of injured tissues. Otolaryngology, as a leading surgical specialty in head and neck surgery, is a candidate for the use of these advanced therapies and medicinal products developed. Nevertheless, a knowledge-based analysis of both areas together is still needed. The dataset was retrieved from the Web of Science database from 1900 to 2020. SciMAT software was used to perform the science mapping analysis and the data for the biomedical translation identification was obtained from the iCite platform. Regarding the analysis of the cognitive structure, we find consolidated research lines, such as the generation of cartilage for use as a graft in reconstructive surgery, reconstruction of microtia, or the closure of perforations of the tympanic membrane. This last research area occupies the most relevant clinical translation with the rest of the areas presenting a lower translational level. In conclusion, Tissue engineering is still in an early translational stage in otolaryngology, otology being the field where most advances have been achieved. Therefore, although otolaryngologists should play an active role in translational research in tissue engineering, greater multidisciplinary efforts are required to promote and encourage the translation of potential clinical applications of tissue engineering for routine clinical use.

The positive impact on translational research of Fondazione italiana di ricerca per la Sclerosi Laterale Amiotrofica (AriSLA), a non-profit foundation focused on amyotrophic lateral sclerosis. Convergence of ex-ante evaluation and ex-post outcomes when goals are set upfront

Charities investing on rare disease research greatly contribute to generate ground-breaking knowledge with the clear goal of finding a cure for their condition of interest. Although the amount of their investments may be relatively small compared to major funders, the advocacy groups' clear mission promotes innovative research and aggregates highly motivated and mission-oriented scientists. Here, we illustrate the case of Fondazione italiana di ricerca per la Sclerosi Laterale Amiotrofica (AriSLA), the main Italian funding agency entirely dedicated to amyotrophic lateral sclerosis research. An international benchmark analysis of publications derived from AriSLA-funded projects indicated that their mean relative citation ratio values (iCite dashboard, National Institutes of Health, U.S.) were very high, suggesting a strong influence on the referring international scientific community. An interesting trend of research toward translation based on the "triangle of biomedicine" and paper citations (iCite) was also observed. Qualitative analysis on researchers' accomplishments was convergent with the bibliometric data, indicating a high level of performance of several working groups, lines of research that speak of progression toward clinical translation, and one study that has progressed from the investigation of cellular mechanisms to a Phase 2 international clinical trial. The key elements of the success of the AriSLA investment lie in: (i) the clear definition of the objectives (research with potential impact on patients, no matter how far), (ii) a rigorous peer-review process entrusted to an international panel of experts, (iii) diversification of the portfolio with ad hoc selection criteria, which also contributed to bringing new experts and younger scientists to the field, and (iv) a close interaction of AriSLA stakeholders with scientists, who developed a strong sense of belonging. Periodic review of the portfolio of investments is a vital practice for funding agencies. Sharing information between funding agencies about their own policies and research assessment methods and outcomes help guide the international debate on funding strategies and research directions to be undertaken, particularly in the field of rare diseases, where synergy is a relevant enabling factor.

Translating Scientific Discovery Into Health Policy Impact: Innovative Bibliometrics Bridge Translational Research Publications to Policy Literature

To understand how translational science efforts lead to outcomes, it is common to examine publications as a key step in the translational process. The National Institutes of Health's Clinical and Translational Science Awards (CTSA) program aims to accelerate that process by providing support to investigators. Although it is challenging to measure the impact of such support on translational outcomes, CTSA-supported research that arises in research publications can advance translation through use of these publications in public policy and guideline documents from government health agencies, intergovernmental organizations, and other outlets. Using cutting-edge bibliometric tools, the authors evaluated how CTSA-supported research has extended its impact beyond academic silos to influence public policy literature. The authors identified approximately 118,490 publications that acknowledged receiving support from a CTSA hub, from the inception of the program in 2006 through 2021. Articles were queried in the Overton policy database, which indexes references to publications in global policy literature. The search revealed 13% of CTSA-supported articles were referenced in policy documents, significantly more than the expected proportion (10%) calculated by Overton. References came from 576 policy source outlets across 87 countries, predominantly the United States and Europe. The most frequent sources included guidelines in PubMed Central, the World Health Organization, and the Centers for Disease Control and Prevention. The authors illustrate the bridge from translational research to public policy with case studies of 6 articles based on CTSA-supported research and having notable policy impact. They found articles with greater clinical relevance, altmetric attention (i.e., nonacademic community/public attention), and academic citation influence were more likely to be referenced in policy literature. Study findings help to characterize the kinds of research that have influenced and may be expected to influence health policy in the future.

Impact of National Institutes of Health and Food and Drug Administration Tobacco Research Funding: A Bibliometrics Analyses

INTRODUCTION

Conduct bibliometric analyses documenting the output of National Institutes of Health (NIH) tobacco-related and Food and Drug Administration (FDA) tobacco regulatory science (FDA-TRS) research portfolios.

AIMS AND METHODS

PubMed identifiers for publications between 2015 and 2020 citing tobacco funding by NIH and/or FDA were imported into NIH iCite generating measures of productivity and influence, including number of citations, journal, relative citation ratios (RCR), and comparison of research influence across Web of Science (WoS) disciplines. Coauthorship and measures of centrality among and between NIH and FDA-supported investigators gauged collaboration.

RESULTS

Between FY 2015 and 2020, 8160 publications cited funding from NIH tobacco-related grants, 1776 cited FDA-TRS grants and 496 cited Common funding (ie, both NIH and FDA-TRS funding). The proportion of publications citing NIH grants declined while those citing FDA-TRS or Common funding rose significantly. Publications citing Common funding showed the highest influence (mean RCR = 2.52). Publications citing FDA-TRS funding displayed higher median RCRs than publications citing NIH funding in most WoS categories. Higher translational progress was estimated over time for FDA-TRS and Common publications compared to NIH publications. Authors citing Common funding scored highest across all collaboration measures.

CONCLUSIONS

This study demonstrates the high bibliometric output of tobacco research overall. The rise in publications citing FDA-TRS and Common likely reflects increased funding for TRS research. Higher RCRs across WoS subject categories and trends towards human translation among FDA-TRS and Common publications indicate focus on research to inform regulation. This analysis suggests that FDA support for TRS has expanded the field of tobacco control resulting in sustained productivity, influence, and collaboration.

IMPLICATIONS

This paper is the first effort to better describe the impact of tobacco research resulting from the addition of FDA funding for TRS in the past decade. The analysis provides impetus for further investigation into the publication topics and their focus which would offer insight into the specific evidence generated on tobacco control and regulation.

Guidelines to Analyze Preclinical Studies Using Perinatal Derivatives

The last 18 years have brought an increasing interest in the therapeutic use of perinatal derivatives (PnD). Preclinical studies used to assess the potential of PnD therapy include a broad range of study designs. The COST SPRINT Action (CA17116) aims to provide systematic and comprehensive reviews of preclinical studies for the understanding of the therapeutic potential and mechanisms of PnD in diseases and injuries that benefit from PnD therapy. Here we describe the publication search and data mining, extraction, and synthesis strategies employed to collect and prepare the published data selected for meta-analyses and reviews of the efficacy of PnD therapies for different diseases and injuries. A coordinated effort was made to prepare the data suitable to make statements for the treatment efficacy of the different types of PnD, routes, time points, and frequencies of administration, and the dosage based on clinically relevant effects resulting in clear increase, recovery or amelioration of the specific tissue or organ function. According to recently proposed guidelines, the harmonization of the nomenclature of PnD types will allow for the assessment of the most efficient treatments in various disease models. Experts within the COST SPRINT Action (CA17116), together with external collaborators, are doing the meta-analyses and reviews using the data prepared with the strategies presented here in the relevant disease or research fields. Our final aim is to provide standards to assess the safety and clinical benefit of PnD and to minimize redundancy in the use of animal models following the 3R principles for animal experimentation.

Translational Bioinformatics Applied to the Study of Complex Diseases

Translational Bioinformatics (TBI) is defined as the union of translational medicine and bioinformatics. It emerges as a major advance in science and technology by covering everything, from the most basic database discoveries, to the development of algorithms for molecular and cellular analysis, as well as their clinical applications. This technology makes it possible to access the knowledge of scientific evidence and apply it to clinical practice. This manuscript aims to highlight the role of TBI in the study of complex diseases, as well as its application to the understanding and treatment of cancer. An integrative literature review was carried out, obtaining articles through several websites, among them: PUBMED, Science Direct, NCBI-PMC, Scientific Electronic Library Online (SciELO), and Google Academic, published in English, Spanish, and Portuguese, indexed in the referred databases and answering the following guiding question: "How does TBI provide a scientific understanding of complex diseases?" An additional effort is aimed at the dissemination, inclusion, and perpetuation of TBI knowledge from the academic environment to society, helping the study, understanding, and elucidating of complex disease mechanics and their treatment.

Epithelial-to-mesenchymal transition as a learning paradigm of cell biology

Epithelial-to-mesenchymal transition (EMT) is a complex biological process that occurs during normal embryogenesis and in certain pathological conditions, particularly in cancer. EMT can be viewed as a cell biology-based process, since it involves all the cellular components, including the plasma membrane, cytoskeleton and extracellular matrix, endoplasmic reticulum, Golgi apparatus, lysosomes, and mitochondria, as well as cellular processes, such as regulation of gene expression and cell cycle, adhesion, migration, signaling, differentiation, and death. Therefore, we propose that EMT could be used to motivate undergraduate medical students to learn and understand cell biology. Here, we describe and discuss the involvement of each cellular component and process during EMT. To investigate the density with which different cell biology concepts are used in EMT research, we apply a bibliometric approach. The most frequent cell biology topics in EMT studies were regulation of gene expression, cell signaling, cell cycle, cell adhesion, cell death, cell differentiation, and cell migration. Finally, we suggest that the study of EMT could be incorporated into undergraduate disciplines to improve cell biology understanding among premedical, medical and biomedical students.

Exploring the clinical translation intensity of papers published by the world's top scientists in basic medicine

The extent to which basic medical research is translated into clinical practice is a topic of interest to all stakeholders. In this study, we assessed the clinical translation intensity of papers published by scientists who have made outstanding contributions to the field of basic medicine (Lasker Prize winners for Basic Medical Research). Approximate Potential for Translation (APT), Translational science scores (TS), and Citations by clinical research (Cited by Clin.) were analyzed as dependent variables. A traditional citation indicator was used as a reference (relative citation ratio, RCR). In order to examine the correlation between these different indicators and the characteristics of the paper, the author, and the institution. we used nonparametric tests, Spearman correlations, ordinal least squares regressions (OLS), quantile regressions, and zero-inflated negative binomial regression methods. We found that among the basic medical research papers published by Lasker Basic Medicine Award winners, (1) 20% are cited by clinical research; 11.6% of the papers were more valuable for clinical research than basic research; 12.8% have a probability of more than 50% to be cited in future clinical studies; (2) Spearman correlations were conducted among APT, TS, Cited by Clin., RCR, and all of the other continuous variables. There is a significant, positive, low to moderate correlation between APT, TS, and Cited by Clin (APT and TS: r = 0.549, p < 0.01; APT and Cited by Clin: r = 0.530, p < 0.01; TS and Cited by Clin: r = 0.383, p < 0.01). However, the relationship between RCR and the three indicators of clinical translation intensity was not consistent. APT was positively correlated with RCR (r = 0.553, p < 0.01). Cited by Clin. is weakly positively correlated with RCR (r = 0.381, p < 0.01). There is almost no correlation between TS and RCR (r = 0.184, p < 0.01). (3) Publication age, primary research paper, multidisciplinary science, number of disciplines, authors, institutions, funded projects, references, length of the title, length of paper, physical age, gender, nationality, institutional type, Nobel Prize have a significant relationship with 1 to 3 types of clinical translation intensity measures. In a sample of basic medical research papers published by the world's top scientists in basic medicine, we came to the following conclusions: the three indicators, APT, TS and Cited by Clin., measured the clinical translation intensity of the papers from different perspectives. They are both related to each other and have their own characteristics. In a sample of basic medical research papers published by the world's top scientists in basic medicine, characteristics at the paper, winner, and institution level significantly correlated with the measures of clinical translation intensity. Gender effect on the clinical translation intensity of papers was confirmed. Traditional citation-based indicators and translational-focused indicators measure academic impact and clinical impact respectively. There is a certain degree of disconnect between them. Two types of indicators should be used in combination in future assessments of basic medical research.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11192-023-04634-4.

Sharing GWAS summary statistics results in more citations

A review of citation rates from genomic studies in the GWAS Catalog suggests that sharing summary statistics results, on average, in ~81.8% more citations, highlighting a benefit of publicly sharing GWAS summary statistics.

Fifty years of the Journal of Perinatal Medicine: an altmetric and bibliometric study

OBJECTIVES

To apply scientometric methodology to characterize influential articles in the Journal of Perinatal Medicine (JPM).

METHODS

We performed a cross-sectional study of all JPM articles indexed in Clarivate Web of Science (WOS), NIH Open Citation Collection, and Altmetric Explorer databases (1973-2022). We identified articles cited ≥100 times in WOS and articles with highest Relative Citation Ratios (RCR, a metric of influence based on citations) and highest Altmetric Attention Scores (AAS, a metric of engagement with social media and public platforms). We performed descriptive analysis to characterize influential articles based on citation rates vs. highest AAS, and quantile regression with bootstrapping to estimate the median differences (95% confidence intervals).

RESULTS

We identified 4095 JPM articles that were indexed in the WOS, of which 3,959 (96.7%) had RCRs and 939 (22.9%) had AASs. The study cohort included 34 articles cited ≥100 times and the 34 top-RCR and 34 top-AAS articles, representing 83 unique articles. These influential articles had median 67 citations (IQR 17-114), median RCR 3.4 (IQR 1.7-5.0), and median AAS 14 (IQR 3-28). The majority were observational studies and reviews. Compared to top-AAS articles, top-cited articles had higher median citations (117 [IQR 111-147] vs. 13 [IQR 5-62]; median difference 104.0, 95% CI 86.6-121.4) and citations per year (7.3 [IQR 4.9-10.6] vs. 2.3 [0.7-4.6]; median difference 5.5 [95% CI 3.1-7.9]). Results were similar for top-RCR vs. top-AAS articles.

CONCLUSIONS

We identified influential articles during 50 years of JPM, providing insight into the impact of the journal and providing a template for future studies of academic journals.

The rocky road to translational science: An analysis of Clinical and Translational Science Awards

Studies point out that the productivity decline in biomedicine is in significant part due to difficulties in translating basic science into clinical application. To promote translational research, the US NIH launched the Clinical and Translational Science Awards (CTSA) program in 2006. Prior evaluations of the CTSA program often assumed that the key to translation is inter-organizational collaboration or multidisciplinarity; hence, changes in either of these were measured as evidence of translational science. However, using novel measures of translational science, this study examines the relationship between CTSAs and translational science per se. We define ‘translational science’ as basic science that has translational features, and we employ two distinct, complementary measures of translational science based on publication data. Using 115 Carnegie R1 universities and their translational science publications, we find that receiving a CTSA does not obviously cause receiving institutions to conduct more translational science. Furthermore, our Principal Investigator-level analysis shows that those who received direct support from the CTSA program had already generated more translational science than others and that their research behavior did not change significantly after receiving a CTSA. Future evaluation research can adopt our novel measures of translational science and evaluation research design in the assessment of translational research activities. Finally, we conclude with a discussion of the implications of our findings for science governance.

Tracking biomedical articles along the translational continuum: a measure based on biomedical knowledge representation

Keeping track of translational research is essential to evaluating the performance of programs on translational medicine. Despite several indicators in previous studies, a consensus measure is still needed to represent the translational features of biomedical research at the article level. In this study, we first trained semantic representations of biomedical entities and documents (i.e., bio-entity2vec and bio-doc2vec) based on over 30 million PubMed articles. With these vectors, we then developed a new measure called Translational Progression (TP) for tracking biomedical articles along the translational continuum. We validated the effectiveness of TP from two perspectives (Clinical trial phase identification and ACH classification), which showed excellent consistency between TP and other indicators. Meanwhile, TP has several advantages. First, it can track the degree of translation of biomedical research dynamically and in real-time. Second, it is straightforward to interpret and operationalize. Third, it doesn't require labor-intensive MeSH labeling and it is suitable for big scholarly data as well as papers that are not indexed in PubMed. In addition, we examined the translational progressions of biomedical research from three dimensions (including overall distribution, time, and research topic), which revealed three significant findings. The proposed measure in this study could be used by policymakers to monitor biomedical research with high translational potential in real-time and make better decisions. It can also be adopted and improved for other domains, such as physics or computer science, to assess the application value of scientific discoveries.

Robustness of evidence reported in preprints during peer review

Scientists have expressed concern that the risk of flawed decision making is increased through the use of preprint data that might change after undergoing peer review. This Health Policy paper assesses how COVID-19 evidence presented in preprints changes after review. We quantified attrition dynamics of more than 1000 epidemiological estimates first reported in 100 preprints matched to their subsequent peer-reviewed journal publication. Point estimate values changed an average of 6% during review; the correlation between estimate values before and after review was high (0·99) and there was no systematic trend. Expert peer-review scores of preprint quality were not related to eventual publication in a peer-reviewed journal. Uncertainty was reduced during peer review, with CIs reducing by 7% on average. These results support the use of preprints, a component of biomedical research literature, in decision making. These results can also help inform the use of preprints during the ongoing COVID-19 pandemic and future disease outbreaks.

Diminishing Basic Science Research Experience Among United States Cardiothoracic Surgery Trainees

INTRODUCTION

There is growing concern regarding the attrition of surgeon-scientists. To understand the decline of basic science research (BSR), it is essential to examine trends in research conducted by trainees. We hypothesized that, over recent decades, cardiothoracic (CT) surgery trainees have published fewer BSR articles.

MATERIALS AND METHODS

CT surgeons at United States training institutions in 2020 who completed training in the past three decades, excluding international trainees, were analyzed (1991-2000: n = 148; 2001-2010: n = 228; 2011-2020: n = 247). Publication records were obtained from Scopus. Articles with medical subject heading terms involving molecular/cellular or animal research were classified as BSR using the National Institutes of Health iCite Translation module. Data were analyzed using Fisher's exact test or the Wilcoxon rank-sum test.

RESULTS

While the proportion of surgeons who published a first-author paper during training remained stable over the past two decades (178/228 [78.1%] versus 189/247 [76.5%], P = 0.7427), the proportion who published a first-author BSR paper decreased significantly (135/228 [59.2%] versus 96/247 [38.9%], P < 0.0001). Among surgeons who published a first-author paper in training, the total papers published by each trainee did not change over the past two decades (3.5 versus 3.3 first-author papers per 10 y of training, P = 0.8819). However, the number of BSR papers published during training decreased significantly (1.7 versus 0.8 first-author papers per 10 y of training, P < 0.0001).

CONCLUSIONS

CT surgery trainees are publishing fewer BSR papers. Additional efforts are needed to increase exposure of trainees to BSR and reaffirm that BSR is a valuable and worthwhile pursuit for academic surgeons.

The ripple effects of funding on researchers and output

Using unique, new, matched UMETRICS data on people employed on research projects and Author-ity data on biomedical publications, this paper shows that National Institutes of Health funding stimulates research by supporting the teams that conduct it. While faculty-both principal investigators (PIs) and other faculty-and their productivity are heavily affected by funding, so are trainees and staff. The largest effects of funding on research output are ripple effects on publications that do not include PIs. While funders focus on research output from projects, they would be well advised to consider how funding ripples through the wide range of people, including trainees and staff, employed on projects.

Methods for identifying biomedical translation: a systematic review

Translational medicine is an important area of biomedicine, and has significantly facilitated the development of biomedical research. Despite its relevance, there is no consensus on how to evaluate its progress and impact. A systematic review was carried out to identify all the methods to evaluate translational research. Seven methods were found according to the established criteria to analyze their characteristics, advantages, and limitations. They allow us to perform this type of evaluation in different ways. No relevant advantages were found between them; each one presented its specific limitations that need to be considered. Nevertheless, the Triangle of Biomedicine could be considered the most relevant method, concerning the time since its publication and usefulness. In conclusion, there is still a lack of a gold-standard method for evaluating biomedical translational research.

Identification of human gene research articles with wrongly identified nucleotide sequences

Nucleotide sequence reagents underpin molecular techniques that have been applied across hundreds of thousands of publications. We have previously reported wrongly identified nucleotide sequence reagents in human research publications and described a semi-automated screening tool Seek & Blastn to fact-check their claimed status. We applied Seek & Blastn to screen >11,700 publications across five literature corpora, including all original publications in Gene from 2007 to 2018 and all original open-access publications in Oncology Reports from 2014 to 2018. After manually checking Seek & Blastn outputs for >3,400 human research articles, we identified 712 articles across 78 journals that described at least one wrongly identified nucleotide sequence. Verifying the claimed identities of >13,700 sequences highlighted 1,535 wrongly identified sequences, most of which were claimed targeting reagents for the analysis of 365 human protein-coding genes and 120 non-coding RNAs. The 712 problematic articles have received >17,000 citations, including citations by human clinical trials. Given our estimate that approximately one-quarter of problematic articles may misinform the future development of human therapies, urgent measures are required to address unreliable gene research articles.

Big splashes & ripple effects: a narrative review of the short- & long-term impact of publications supported by an NIH CTSA pediatrics program

BACKGROUND AND OBJECTIVE

This review examines a promising new framework for analyzing outputs of pediatric research in the context of translational advancement. We demonstrate a method for evaluating the impact of an NIH Clinical and Translational Science Award's (CTSA) Pediatrics Program through publications that have emerged from supported research. The Georgia CTSA Pediatrics Program provides training, funding, and infrastructure to ensure that researchers have the resources to advance pediatric health. Internal evaluations found that research supported by this program is exceptionally impactful within the academic community and commands high interest within the lay community. Therefore, we examined the impact of this research in both traditional academic and broader community spheres using bibliometrics-the study of supported publications. Bibliometrics describe a pivotal stage in the translational process of bringing scientific discoveries to clinical/community use and include both academic citations and 'altmetric' or non-academic attention. These complementary approaches combine to shed light on the short- and long-term impact of the research on segments of the translational pipeline, including academic literature, community discourse, technological advancement, and public health policy.

METHODS

The authors identified a portfolio of 250 articles supported by the Georgia CTSA Pediatrics Program from 2007-2020. We utilized various bibliometrics to analyze both short-term attention, or 'splash' made by articles, and long-term influence, or 'ripples' made across both academic and public spheres.

KEY CONTENT AND FINDINGS

The short-term splash of the portfolio was indicated through publication in high-impact factor journals, peer faculty recommendations, and Mendeley readership, as well as by early altmetric attention in news stories, blogs, and Twitter posts. The portfolio's long-term ripples were demonstrated by high absolute and relative rates of academic citation and by downstream altmetric influence in public-facing documents, including Wikipedia articles, patent applications, and policy documents.

CONCLUSIONS

This article reviews a useful bibliometric methodology for illustrating the waves of impact made by pediatric research. Whereas splash provides a picture of early interest in a publication, a preliminary indicator of eventual utility and impact, ripples provide a measure of the cumulative influence of an article over time. Both reflect opportunities for a line of research to advance along the translational spectrum.

Translatability Analysis of National Institutes of Health-Funded Biomedical Research That Applies Artificial Intelligence

IMPORTANCE

Despite the rapid growth of interest and diversity in applications of artificial intelligence (AI) to biomedical research, there are limited objective ways to characterize the potential for use of AI in clinical practice.

OBJECTIVE

To examine what types of medical AI have the greatest estimated translational impact (ie, ability to lead to development that has measurable value for human health) potential.

DESIGN, SETTING, AND PARTICIPANTS

In this cohort study, research grants related to AI awarded between January 1, 1985, and December 31, 2020, were identified from a National Institutes of Health (NIH) award database. The text content for each award was entered into a Natural Language Processing (NLP) clustering algorithm. An NIH database was also used to extract citation data, including the number of citations and approximate potential to translate (APT) score for published articles associated with the granted awards to create proxies for translatability.

EXPOSURES

Unsupervised assignment of AI-related research awards to application topics using NLP.

MAIN OUTCOMES AND MEASURES

Annualized citations per $1 million funding (ACOF) and average APT score for award-associated articles, grouped by application topic. The APT score is a machine-learning based metric created by the NIH Office of Portfolio Analysis that quantifies the likelihood of future citation by a clinical article.

RESULTS

A total of 16 629 NIH awards related to AI were included in the analysis, and 75 applications of AI were identified. Total annual funding for AI grew from $17.4 million in 1985 to $1.43 billion in 2020. By average APT, interpersonal communication technologies (0.488; 95% CI, 0.472-0.504) and population genetics (0.463; 95% CI, 0.453-0.472) had the highest translatability; environmental health (ACOF, 1038) and applications focused on the electronic health record (ACOF, 489) also had high translatability. The category of applications related to biochemical analysis was found to have low translatability by both metrics (average APT, 0.393; 95% CI, 0.388-0.398; ACOF, 246).

CONCLUSIONS AND RELEVANCE

Based on this study's findings, data on grants from the NIH can apparently be used to identify and characterize medical applications of AI to understand changes in academic productivity, funding support, and potential for translational impact. This method may be extended to characterize other research domains.

An analysis of the Clinical and Translational Science Award pilot project portfolio using data from Research Performance Progress Reports

Introduction:

Pilot projects (“pilots”) are important for testing hypotheses in advance of investing more funds for full research studies. For some programs, such as Clinical and Translational Science Awards (CTSAs) supported by the National Center for Translational Sciences, pilots also make up a significant proportion of the research projects conducted with direct CTSA support. Unfortunately, administrative data on pilots are not typically captured in accessible databases. Though data on pilots are included in Research Performance Progress Reports, it is often difficult to extract, especially for large programs like the CTSAs where more than 600 pilots may be reported across all awardees annually. Data extraction challenges preclude analyses that could provide valuable information about pilots to researchers and administrators.

Methods:

To address those challenges, we describe a script that partially automates extraction of pilot data from CTSA research progress reports. After extraction of the pilot data, we use an established machine learning (ML) model to determine the scientific content of pilots for subsequent analysis. Analysis of ML-assigned scientific categories reveals the scientific diversity of the CTSA pilot portfolio and relationships among individual pilots and institutions.

Results:

The CTSA pilots are widely distributed across a number of scientific areas. Content analysis identifies similar projects and the degree of overlap for scientific interests among hubs.

Conclusion:

Our results demonstrate that pilot data remain challenging to extract but can provide useful information for communicating with stakeholders, administering pilot portfolios, and facilitating collaboration among researchers and hubs.

The characteristics of early-stage research into human genes are substantially different from subsequent research

Throughout the last 2 decades, several scholars observed that present day research into human genes rarely turns toward genes that had not already been extensively investigated in the past. Guided by hypotheses derived from studies of science and innovation, we present here a literature-wide data-driven meta-analysis to identify the specific scientific and organizational contexts that coincided with early-stage research into human genes throughout the past half century. We demonstrate that early-stage research into human genes differs in team size, citation impact, funding mechanisms, and publication outlet, but that generalized insights derived from studies of science and innovation only partially apply to early-stage research into human genes. Further, we demonstrate that, presently, genome biology accounts for most of the initial early-stage research, while subsequent early-stage research can engage other life sciences fields. We therefore anticipate that the specificity of our findings will enable scientists and policymakers to better promote early-stage research into human genes and increase overall innovation within the life sciences.

Measuring the Interactions Between Health Demand, Informatics Supply, and Technological Applications in Digital Medical Innovation for China: Content Mapping and Analysis

BACKGROUND

There were 2 major incentives introduced by the Chinese government to promote medical informatics in 2009 and 2016. As new drugs are the major source of medical innovation, informatics-related concepts and techniques are a major source of digital medical innovation. However, it is unclear whether the research efforts of medical informatics in China have met the health needs, such as disease management and population health.

OBJECTIVE

We proposed an approach to mapping the interplay between different knowledge entities by using the tree structure of Medical Subject Headings (MeSH) to gain insights into the interactions between informatics supply, health demand, and technological applications in digital medical innovation in China.

METHODS

All terms under the MeSH tree parent node "Diseases [C]" or node "Health [N01.400]" or "Public Health [N06.850]" were labelled as H. All terms under the node "Information Science [L]" were labelled as I, and all terms under node "Analytical, Diagnostic and Therapeutic Techniques, and Equipment [E]" were labelled as T. The H-I-T interactions can be measured by using their co-occurrences in a given publication.

RESULTS

The H-I-T interactions in China are showing significant growth and a more concentrated interplay were observed. Computing methodologies, informatics, and communications media (such as social media and the internet) constitute the majority of I-related concepts and techniques used for resolving the health promotion and diseases management problems in China. Generally there is a positive correlation between the burden and informatics research efforts for diseases in China. We think it is not contradictory that informatics research should be focused on the greatest burden of diseases or where it can have the most impact. Artificial intelligence is a competing field of medical informatics research in China, with a notable focus on diagnostic deep learning algorithms for medical imaging.

CONCLUSIONS

It is suggested that technological transfers, namely the functionality to be realized by medical/health informatics (eg, diagnosis, therapeutics, surgical procedures, laboratory testing techniques, and equipment and supplies) should be strengthened. Research on natural language processing and electronic health records should also be strengthened to improve the real-world applications of health information technologies and big data in the future.

pmparser and PMDB: resources for large-scale, open studies of the biomedical literature

PubMed is an invaluable resource for the biomedical community. Although PubMed is freely available, the existing API is not designed for large-scale analyses and the XML structure of the underlying data is inconvenient for complex queries. We developed an R package called pmparser to convert the data in PubMed to a relational database. Our implementation of the database, called PMDB, currently contains data on over 31 million PubMed Identifiers (PMIDs) and is updated regularly. Together, pmparser and PMDB can enable large-scale, reproducible, and transparent analyses of the biomedical literature. pmparser is licensed under GPL-2 and available at https://pmparser.hugheylab.org. PMDB is available in both PostgreSQL (DOI 10.5281/zenodo.4008109) and Google BigQuery (https://console.cloud.google.com/bigquery?project=pmdb-bq&d=pmdb).

Research productivity and collaboration of the NIH-funded HIV vaccine trials network: A bibliometric analysis

The HIV Vaccine Trials Network (HVTN) is the world's largest publicly funded, multi-disciplinary international collaboration facilitating the development of vaccines to prevent HIV/AIDS and has conducted the vast majority of HIV/AIDS clinical trials since its inception in 1999. Although scientific findings from the program have been published in scholarly journals, the impact of a large scientific research network such as the HVTN on the HIV/AIDS vaccine field has not been assessed. This paper describes and elucidates the productivity, influence, and collaboration among HVTN researchers over the last two decades. Our analyses indicate that the HVTN has funded a large number of HIV/AIDS vaccine safety and efficacy clinical trials through a strong global network of clinical sites. In addition, several metrics indicate HVTN researchers also published original research articles that are influential in the HIV vaccine field. Scientific research collaboration is critically important in a complex and multidisciplinary field such as HIV vaccine development as it allows improved sharing of knowledge and expertise as well as the pooling of resources and data. We found that collaboration in the HIV vaccine field increased during this time period and collaboration among HVTN authors increased even more. Combining these productivity, influence, and collaboration metrics with research outcomes can provide a comprehensive assessment of large complex programs such as the HVTN.

A Bibliometric Measure of Translational Science

Science funders are increasingly requiring evidence of the broader impacts of even basic research. Initiatives such as NIH's CTSA program are designed to shift the research focus toward more translational research. However, tracking the effectiveness of such programs depends on developing indicators that can track the degree to which basic research is influencing clinical research. We propose a new bibliometric indicator, the TS score, that is relatively simple to calculate, can be implemented at scale, is easy to replicate, and has good reliability and validity properties. This indicator is broadly applicable in settings where the goal is to estimate the degree to which basic research is used in more applied downstream research, relative to use in basic research. The TS score should be of use for a variety of policy analysis and research evaluation purposes.

A detailed open access model of the PubMed literature

Portfolio analysis is a fundamental practice of organizational leadership and is a necessary precursor of strategic planning. Successful application requires a highly detailed model of research options. We have constructed a model, the first of its kind, that accurately characterizes these options for the biomedical literature. The model comprises over 18 million PubMed documents from 1996-2019. Document relatedness was measured using a hybrid citation analysis + text similarity approach. The resulting 606.6 million document-to-document links were used to create 28,743 document clusters and an associated visual map. Clusters are characterized using metadata (e.g., phrases, MeSH) and over 20 indicators (e.g., funding, patent activity). The map and cluster-level data are embedded in Tableau to provide an interactive model enabling in-depth exploration of a research portfolio. Two example usage cases are provided, one to identify specific research opportunities related to coronavirus, and the second to identify research strengths of a large cohort of African American and Native American researchers at the University of Michigan Medical School.

A protocol for retrospective translational science case studies of health interventions

The critical processes driving successful research translation remain understudied. We describe a mixed-method case study protocol for analyzing translational research that has led to the successful development and implementation of innovative health interventions. An overarching goal of these case studies is to describe systematically the chain of events between basic, fundamental scientific discoveries and the adoption of evidence-based health applications, including description of varied, long-term impacts. The case study approach isolates many of the key factors that enable the successful translation of research into practice and provides compelling evidence connecting the intervention to measurable changes in health and medical practice, public health outcomes, and other broader societal impacts. The goal of disseminating this protocol is to systematize a rigorous approach, which can enhance reproducibility, promote the development of a large collection of comparable studies, and enable cross-case analyses. This approach, an application of the "science of translational science," will lead to a better understanding of key research process markers, timelines, and potential points of leverage for intervention that may help facilitate decisions, processes, and policies to speed the sustainable translational process. Case studies are effective communication vehicles to demonstrate both accountability and the impacts of the public's investment in research.

Alzheimer's Disease, and Breast and Prostate Cancer Research: Translational Failures and the Importance to Monitor Outputs and Impact of Funded Research

Dementia and cancer are becoming increasingly prevalent in Western countries. In the last two decades, research focused on Alzheimer's disease (AD) and cancer, in particular, breast cancer (BC) and prostate cancer (PC), has been substantially funded both in Europe and worldwide. While scientific research outcomes have contributed to increase our understanding of the disease etiopathology, still the prevalence of these chronic degenerative conditions remains very high across the globe. By definition, no model is perfect. In particular, animal models of AD, BC, and PC have been and still are traditionally used in basic/fundamental, translational, and preclinical research to study human disease mechanisms, identify new therapeutic targets, and develop new drugs. However, animals do not adequately model some essential features of human disease; therefore, they are often unable to pave the way to the development of drugs effective in human patients. The rise of new technological tools and models in life science, and the increasing need for multidisciplinary approaches have encouraged many interdisciplinary research initiatives. With considerable funds being invested in biomedical research, it is becoming pivotal to define and apply indicators to monitor the contribution to innovation and impact of funded research. Here, we discuss some of the issues underlying translational failure in AD, BC, and PC research, and describe how indicators could be applied to retrospectively measure outputs and impact of funded biomedical research.

Filtration Selection and Data Consilience: Distinguishing Signal from Artefact with Mechanical Impact Simulator Data

A large variety of data filtration techniques exist in biomechanics literature. Data filtration is both an 'art' and a 'science' to eliminate noise and retain true signal to draw conclusions that will direct future hypotheses, experimentation, and technology development. Thus, data consilience is paramount, but is dependent on filtration methodologies. In this study, we utilized ligament strain, vertical ground reaction force, and kinetic data from cadaveric impact simulations to assess data from four different filters (12 vs. 50 Hz low-pass; forward vs. zero lag). We hypothesized that 50 Hz filtered data would demonstrate larger peak magnitudes, but exhibit consilience of waveforms and statistical significance as compared to 12 Hz filtered data. Results demonstrated high data consilience for matched pair t test correlations of peak ACL strain (≥ 0.97), MCL strain (≥ 0.93) and vertical ground reaction force (≥ 0.98). Kinetics had a larger range of correlation (0.06-0.96) that was dependent on both external load application and direction of motion monitored. Coefficients of multiple correlation demonstrated high data consilience for zero lag filtered data. With respect to in vitro mechanical data, selection of low-pass filter cutoff frequency will influence both the magnitudes of discrete and waveform data. Dependent on the data type (i.e., strain and ground reaction forces), this will not likely significantly alter conclusions of statistical significance previously reported in the literature with high consilience of matched pair t-test correlations and coefficients of multiple correlation demonstrated. However, rotational kinetics are more sensitive to filtration selection and could be suspect to errors, especially at lower magnitudes.

Identifying and understanding factors that affect the translation of therapies from the laboratory to patients: a study protocol

Background: The process of translating preclinical findings into a clinical setting takes decades. Previous studies have suggested that only 5-10% of the most promising preclinical studies are successfully translated into viable clinical applications. The underlying determinants of this low success rate (e.g. poor experimental design, suboptimal animal models, poor reporting) have not been examined in an empirical manner. Our study aims to determine the contemporary success rate of preclinical-to-clinical translation, and subsequently determine if an association between preclinical study design and translational success/failure exists. Methods: Established systematic review methodology will be used with regards to the literature search, article screening and study selection process. Preclinical, basic science studies published in high impact basic science journals between 1995 and 2015 will be included. Included studies will focus on publicly available interventions with potential clinical promise. The primary outcome will be successful clinical translation of promising therapies - defined as the conduct of at least one Phase II trial (or greater) with a positive finding. A case-control study will then be performed to evaluate the association between elements of preclinical study design and reporting and the likelihood of successful translation. Discussion: This study will provide a comprehensive analysis of the therapeutic translation from the laboratory bench to the bedside. Importantly, any association between factors of study design and the success of translation will be identified. These findings may inform future research teams attempting preclinical-to-clinical translation. Results will be disseminated to identified knowledge users that fund/support preclinical research.

National Institutes of Health Research Plan on Rehabilitation: Analysis and Progress

OBJECTIVE

To summarize the progress toward the National Institutes of Health (NIH) Research Plan on Rehabilitation goals and the methods by which tracking occurred.

DESIGN

Each grant award was manually coded by NIH staff for research plan goals, type of science categories (eg, basic, applied, infrastructure, etc), and if applicable, training, and then validated by NIH institute and center (IC) experts. Data for years 2015 through 2017 were used to develop a coding algorithm to automatically code grants in 2018 for validation by NIH IC experts. Additional data for all years (2015-2018) were also analyzed to track changes and progress.

SETTING

The research utilized administrative data from NIH Reporter and internal NIH databases.

PARTICIPANTS

The data sample included research grants and programs funded from fiscal years 2015 through 2018. The year 2015 was considered a baseline year as the research plan was published in 2016.

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURES

The primary outcome measures were substantial growth in NIH funding and numbers of awards for rehabilitation research, across most research plan goals and types of science, as well as validation of an automatic algorithm for coding grants.

RESULTS

Number of grants, funding dollars, funding mechanisms, patent data, scientific influence and translational science, research plan goals, and type of science categories were tracked across years (2015-2018). Algorithm validation is presented for 2018 data.

CONCLUSIONS

NIH advanced the goals stated in the Research Plan on Rehabilitation, but gap areas remain. Though funding in this portfolio is growing, continued focus and participation by the field is needed to advance rehabilitation science.

The NIH Open Citation Collection: A public access, broad coverage resource

Citation data have remained hidden behind proprietary, restrictive licensing agreements, which raises barriers to entry for analysts wishing to use the data, increases the expense of performing large-scale analyses, and reduces the robustness and reproducibility of the conclusions. For the past several years, the National Institutes of Health (NIH) Office of Portfolio Analysis (OPA) has been aggregating and enhancing citation data that can be shared publicly. Here, we describe the NIH Open Citation Collection (NIH-OCC), a public access database for biomedical research that is made freely available to the community. This dataset, which has been carefully generated from unrestricted data sources such as MedLine, PubMed Central (PMC), and CrossRef, now underlies the citation statistics delivered in the NIH iCite analytic platform. We have also included data from a machine learning pipeline that identifies, extracts, resolves, and disambiguates references from full-text articles available on the internet. Open citation links are available to the public in a major update of iCite (https://icite.od.nih.gov).