Technological advances in preclinical meta-research

A Systematic Online Living Evidence Summary of experimental Alzheimer's disease research

BACKGROUND

Despite extensive investment, the development of effective treatments for Alzheimer's disease (AD) has been largely unsuccessful. To improve translation, it is crucial to ensure the quality and reproducibility of foundational evidence generated from laboratory models. Systematic reviews play a key role in providing an unbiased overview of the evidence, assessing rigour and reporting, and identifying factors that influence reproducibility. However, the sheer pace of evidence generation is prohibitive to evidence synthesis and assessment.

NEW METHOD

To address these challenges, we have developed AD-SOLES, an integrated workflow of automated tools that collect, curate, and visualise the totality of evidence from in vivo experiments.

RESULTS

AD-SOLES is a publicly accessible interactive dashboard aiming to surface and expose data from in vivo experiments. It summarises the latest evidence, tracks reporting quality and transparency, and allows research users to easily locate evidence relevant to their specific research question.

COMPARISON WITH EXISTING METHODS

Using automated screening methodologies within AD-SOLES, systematic reviews can begin at an accelerated starting point compared to traditional approaches. Furthermore, through text-mining approaches within the full-text of publications, users can identify research of interest using specific models, outcomes, or interventions without relying on details in the title and/or abstract.

CONCLUSIONS

By automating the collection, curation, and visualisation of evidence from in vivo experiments, AD-SOLES addresses the challenges posed by the rapid pace of evidence generation. AD-SOLES aims to offer guidance for research improvement, reduce research waste, highlight knowledge gaps, and support informed decision making for researchers, funders, patients, and the public.

The Automated Systematic Search Deduplicator (ASySD): a rapid, open-source, interoperable tool to remove duplicate citations in biomedical systematic reviews

BACKGROUND

Researchers performing high-quality systematic reviews search across multiple databases to identify relevant evidence. However, the same publication is often retrieved from several databases. Identifying and removing such duplicates ("deduplication") can be extremely time-consuming, but failure to remove these citations can lead to the wrongful inclusion of duplicate data. Many existing tools are not sensitive enough, lack interoperability with other tools, are not freely accessible, or are difficult to use without programming knowledge. Here, we report the performance of our Automated Systematic Search Deduplicator (ASySD), a novel tool to perform automated deduplication of systematic searches for biomedical reviews.

METHODS

We evaluated ASySD's performance on 5 unseen biomedical systematic search datasets of various sizes (1845-79,880 citations). We compared the performance of ASySD with EndNote's automated deduplication option and with the Systematic Review Assistant Deduplication Module (SRA-DM).

RESULTS

ASySD identified more duplicates than either SRA-DM or EndNote, with a sensitivity in different datasets of 0.95 to 0.99. The false-positive rate was comparable to human performance, with a specificity of > 0.99. The tool took less than 1 h to identify and remove duplicates within each dataset.

CONCLUSIONS

For duplicate removal in biomedical systematic reviews, ASySD is a highly sensitive, reliable, and time-saving tool. It is open source and freely available online as both an R package and a user-friendly web application.

Advanced methods and implementations for the meta-analyses of animal models: Current practices and future recommendations

Meta-analytic techniques have been widely used to synthesize data from animal models of human diseases and conditions, but these analyses often face two statistical challenges due to complex nature of animal data (e.g., multiple effect sizes and multiple species): statistical dependency and confounding heterogeneity. These challenges can lead to unreliable and less informative evidence, which hinders the translation of findings from animal to human studies. We present a literature survey of meta-analysis using animal models (animal meta-analysis), showing that these issues are not adequately addressed in current practice. To address these challenges, we propose a meta-analytic framework based on multilevel (linear mixed-effects) models. Through conceptualization, formulations, and worked examples, we illustrate how this framework can appropriately address these issues while allowing for testing new questions. Additionally, we introduce other advanced techniques such as multivariate models, robust variance estimation, and meta-analysis of emergent effect sizes, which can deliver robust inferences and novel biological insights. We also provide a tutorial with annotated R code to demonstrate the implementation of these techniques.

Auto-STEED: A data mining tool for automated extraction of experimental parameters and risk of bias items from in vivo publications

Background

Systematic reviews, i.e., research summaries that address focused questions in a structured and reproducible manner, are a cornerstone of evidence-based medicine and research. However, certain systematic review steps such as data extraction are labour-intensive which hampers their applicability, not least with the rapidly expanding body of biomedical literature.

Objective

To bridge this gap, we aimed at developing a data mining tool in the R programming environment to automate data extraction from neuroscience in vivo publications. The function was trained on a literature corpus (n=45 publications) of animal motor neuron disease studies and tested in two validation corpora (motor neuron diseases, n=31 publications; multiple sclerosis, n=244 publications).

Results

Our data mining tool Auto-STEED (Automated and STructured Extraction of Experimental Data) was able to extract key experimental parameters such as animal models and species as well as risk of bias items such as randomization or blinding from in vivo studies. Sensitivity and specificity were over 85 and 80%, respectively, for most items in both validation corpora. Accuracy and F-scores were above 90% and 0.9 for most items in the validation corpora. Time savings were above 99%.

Conclusions

Our developed text mining tool Auto-STEED is able to extract key experimental parameters and risk of bias items from the neuroscience in vivo literature. With this, the tool can be deployed to probe a field in a research improvement context or to replace one human reader during data extraction resulting in substantial time-savings and contribute towards automation of systematic reviews. The function is available on Github.

Reducing systematic review burden using Deduklick: a novel, automated, reliable, and explainable deduplication algorithm to foster medical research

BACKGROUND

Identifying and removing reference duplicates when conducting systematic reviews (SRs) remain a major, time-consuming issue for authors who manually check for duplicates using built-in features in citation managers. To address issues related to manual deduplication, we developed an automated, efficient, and rapid artificial intelligence-based algorithm named Deduklick. Deduklick combines natural language processing algorithms with a set of rules created by expert information specialists.

METHODS

Deduklick's deduplication uses a multistep algorithm of data normalization, calculates a similarity score, and identifies unique and duplicate references based on metadata fields, such as title, authors, journal, DOI, year, issue, volume, and page number range. We measured and compared Deduklick's capacity to accurately detect duplicates with the information specialists' standard, manual duplicate removal process using EndNote on eight existing heterogeneous datasets. Using a sensitivity analysis, we manually cross-compared the efficiency and noise of both methods.

DISCUSSION

Deduklick achieved average recall of 99.51%, average precision of 100.00%, and average F1 score of 99.75%. In contrast, the manual deduplication process achieved average recall of 88.65%, average precision of 99.95%, and average F1 score of 91.98%. Deduklick achieved equal to higher expert-level performance on duplicate removal. It also preserved high metadata quality and drastically reduced time spent on analysis. Deduklick represents an efficient, transparent, ergonomic, and time-saving solution for identifying and removing duplicates in SRs searches. Deduklick could therefore simplify SRs production and represent important advantages for scientists, including saving time, increasing accuracy, reducing costs, and contributing to quality SRs.

The Case for Modernizing Biomedical Research in Ireland through the Creation of an Irish 3Rs Centre

Simple Summary

The 3Rs principle indicates the measures to refine, reduce, and replace the use of animals in research. Thirteen out of the twenty-seven European countries already have 3Rs Centres. These support the development, acceptance, and/or implementation of methods that address the 3Rs principle at a research and educational level. To date, Ireland has no 3Rs centre. In this commentary, we present the reasons for Ireland to embrace the creation of a national Irish 3Rs Centre. We believe the centre will enable Ireland to be ready for the paradigm shift that is internationally occurring in biomedical research, towards the modernisation and replacement of tests carried out in animals when it is scientifically possible to do so.

Abstract

Since its publication, the 3Rs principle has provided a cornerstone for more ethical and humane biomedical and regulatory research. In Europe, the 3Rs principle has been incorporated into the European Directive 63/2010/EU, with the ultimate aim of fully replacing the procedures on live animals for scientific and educational purposes as soon as it is scientifically possible to do so. Thus, a critical shift in the discussion on animal use in biomedical and regulatory research is undergoing in Europe, a discussion where satisfying the “replacement” principle is becoming more and more defined as a scientific rather than ethical need. 3Rs Centres have been established in recent years across Europe. To date, Ireland has no 3Rs Centre, and the uptake of the 3Rs principle, and in particular of the “replacement” aspect, has been slow. In this Commentary, we present the Irish context of the use of animal models in biomedical and regulatory research, and urge for what, in the authors’ opinion, are the most critical actions that Ireland must undertake to align its biomedical (basic, applied and translational) research with the European 3Rs strategy.