1
|
Hair K, Wilson E, Maksym O, Macleod MR, Sena ES. A Systematic Online Living Evidence Summary of experimental Alzheimer's disease research. J Neurosci Methods 2024; 409:110209. [PMID: 38964475 DOI: 10.1016/j.jneumeth.2024.110209] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2023] [Revised: 05/02/2024] [Accepted: 06/28/2024] [Indexed: 07/06/2024]
Abstract
BACKGROUND Despite extensive investment, the development of effective treatments for Alzheimer's disease (AD) has been largely unsuccessful. To improve translation, it is crucial to ensure the quality and reproducibility of foundational evidence generated from laboratory models. Systematic reviews play a key role in providing an unbiased overview of the evidence, assessing rigour and reporting, and identifying factors that influence reproducibility. However, the sheer pace of evidence generation is prohibitive to evidence synthesis and assessment. NEW METHOD To address these challenges, we have developed AD-SOLES, an integrated workflow of automated tools that collect, curate, and visualise the totality of evidence from in vivo experiments. RESULTS AD-SOLES is a publicly accessible interactive dashboard aiming to surface and expose data from in vivo experiments. It summarises the latest evidence, tracks reporting quality and transparency, and allows research users to easily locate evidence relevant to their specific research question. COMPARISON WITH EXISTING METHODS Using automated screening methodologies within AD-SOLES, systematic reviews can begin at an accelerated starting point compared to traditional approaches. Furthermore, through text-mining approaches within the full-text of publications, users can identify research of interest using specific models, outcomes, or interventions without relying on details in the title and/or abstract. CONCLUSIONS By automating the collection, curation, and visualisation of evidence from in vivo experiments, AD-SOLES addresses the challenges posed by the rapid pace of evidence generation. AD-SOLES aims to offer guidance for research improvement, reduce research waste, highlight knowledge gaps, and support informed decision making for researchers, funders, patients, and the public.
Collapse
Affiliation(s)
- Kaitlyn Hair
- Centre for Clinical Brain Sciences, The University of Edinburgh, Edinburgh, UK
| | - Emma Wilson
- Centre for Clinical Brain Sciences, The University of Edinburgh, Edinburgh, UK
| | - Olena Maksym
- Centre for Clinical Brain Sciences, The University of Edinburgh, Edinburgh, UK
| | - Malcolm R Macleod
- Centre for Clinical Brain Sciences, The University of Edinburgh, Edinburgh, UK
| | - Emily S Sena
- Centre for Clinical Brain Sciences, The University of Edinburgh, Edinburgh, UK.
| |
Collapse
|
2
|
Tóth B, Berek L, Gulácsi L, Péntek M, Zrubka Z. Automation of systematic reviews of biomedical literature: a scoping review of studies indexed in PubMed. Syst Rev 2024; 13:174. [PMID: 38978132 PMCID: PMC11229257 DOI: 10.1186/s13643-024-02592-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/27/2023] [Accepted: 06/20/2024] [Indexed: 07/10/2024] Open
Abstract
BACKGROUND The demand for high-quality systematic literature reviews (SRs) for evidence-based medical decision-making is growing. SRs are costly and require the scarce resource of highly skilled reviewers. Automation technology has been proposed to save workload and expedite the SR workflow. We aimed to provide a comprehensive overview of SR automation studies indexed in PubMed, focusing on the applicability of these technologies in real world practice. METHODS In November 2022, we extracted, combined, and ran an integrated PubMed search for SRs on SR automation. Full-text English peer-reviewed articles were included if they reported studies on SR automation methods (SSAM), or automated SRs (ASR). Bibliographic analyses and knowledge-discovery studies were excluded. Record screening was performed by single reviewers, and the selection of full text papers was performed in duplicate. We summarized the publication details, automated review stages, automation goals, applied tools, data sources, methods, results, and Google Scholar citations of SR automation studies. RESULTS From 5321 records screened by title and abstract, we included 123 full text articles, of which 108 were SSAM and 15 ASR. Automation was applied for search (19/123, 15.4%), record screening (89/123, 72.4%), full-text selection (6/123, 4.9%), data extraction (13/123, 10.6%), risk of bias assessment (9/123, 7.3%), evidence synthesis (2/123, 1.6%), assessment of evidence quality (2/123, 1.6%), and reporting (2/123, 1.6%). Multiple SR stages were automated by 11 (8.9%) studies. The performance of automated record screening varied largely across SR topics. In published ASR, we found examples of automated search, record screening, full-text selection, and data extraction. In some ASRs, automation fully complemented manual reviews to increase sensitivity rather than to save workload. Reporting of automation details was often incomplete in ASRs. CONCLUSIONS Automation techniques are being developed for all SR stages, but with limited real-world adoption. Most SR automation tools target single SR stages, with modest time savings for the entire SR process and varying sensitivity and specificity across studies. Therefore, the real-world benefits of SR automation remain uncertain. Standardizing the terminology, reporting, and metrics of study reports could enhance the adoption of SR automation techniques in real-world practice.
Collapse
Affiliation(s)
- Barbara Tóth
- Doctoral School of Innovation Management, Óbuda University, Bécsi út 96/B, Budapest, 1034, Hungary
| | - László Berek
- Doctoral School for Safety and Security, Óbuda University, Bécsi út 96/B, Budapest, 1034, Hungary
- University Library, Óbuda University, Bécsi út 96/B, Budapest, 1034, Hungary
| | - László Gulácsi
- HECON Health Economics Research Center, University Research, and Innovation Center, Óbuda University, Bécsi út 96/B, Budapest, 1034, Hungary
| | - Márta Péntek
- HECON Health Economics Research Center, University Research, and Innovation Center, Óbuda University, Bécsi út 96/B, Budapest, 1034, Hungary
| | - Zsombor Zrubka
- HECON Health Economics Research Center, University Research, and Innovation Center, Óbuda University, Bécsi út 96/B, Budapest, 1034, Hungary.
| |
Collapse
|
3
|
Buedo P, Prieto E, Perek-Białas J, Odziemczyk-Stawarz I, Waligora M. More ethics in the laboratory, please! Scientists' perspectives on ethics in the preclinical phase. Account Res 2024:1-16. [PMID: 38235967 DOI: 10.1080/08989621.2023.2294996] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/04/2023] [Accepted: 12/11/2023] [Indexed: 01/19/2024]
Abstract
In recent years there have been calls to improve ethics in preclinical research. Promoting ethics in preclinical research should consider the perspectives of scientists. Our study aims to explore researchers' perspectives on ethics in the preclinical phase. Using interviews and focus groups, we collected views on ethical issues in preclinical research from experienced (n = 11) and early-stage researchers (ESRs) (n = 14) working in a gene therapy and regenerative medicine consortium. A recurring theme among ESRs was the impact of health-related preclinical research on climate change. They highlighted the importance of strengthening ethics in relations within the scientific community. Experienced researchers were focused on technicalities of methods used in preclinical research. They stressed the need for more safeguards to protect the sensitive personal data they work with. Both groups drew attention to the importance of the social context of research and its social impact. They agreed that it is important to be socially responsible - to be aware of and be sensitive to the needs and views of society. This study helps to identify key ethical challenges and, when combined with more data, can ultimately lead to informed and evidence-based improvements to existing regulations.
Collapse
Affiliation(s)
- Paola Buedo
- Research Ethics in Medicine Study Group (REMEDY), Jagiellonian University Medical College, Krakow, Poland
| | - Eugenia Prieto
- Instituto de Diversidad y Evolución Austral (IDEAus), Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET), Puerto Madryn, Argentina
| | - Jolanta Perek-Białas
- Institute of Sociology and Center of Evaluation and Public Policy Analysis, Jagiellonian University, Poland and Warsaw School of Economics, Warsaw, Poland
| | | | - Marcin Waligora
- Research Ethics in Medicine Study Group (REMEDY), Jagiellonian University Medical College, Krakow, Poland
| |
Collapse
|
4
|
Ineichen BV, Rosso M, Macleod MR. From data deluge to publomics: How AI can transform animal research. Lab Anim (NY) 2023; 52:213-214. [PMID: 37758917 DOI: 10.1038/s41684-023-01256-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/29/2023]
Affiliation(s)
- Benjamin V Ineichen
- Center for Reproducible Science, University of Zurich, Zurich, Switzerland.
- Department of Neuroradiology, Clinical Neuroscience Center, University Hospital Zurich, University of Zurich, Zurich, Switzerland.
| | - Marianna Rosso
- Center for Reproducible Science, University of Zurich, Zurich, Switzerland
| | - Malcolm R Macleod
- Centre for Clinical Brain Sciences, Edinburgh Medical School, The University of Edinburgh, Edinburgh, Scotland, UK
| |
Collapse
|
5
|
Hair K, Wilson E, Wong C, Tsang A, Macleod M, Bannach-Brown A. Systematic online living evidence summaries: emerging tools to accelerate evidence synthesis. Clin Sci (Lond) 2023; 137:773-784. [PMID: 37219941 PMCID: PMC10220429 DOI: 10.1042/cs20220494] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/19/2022] [Revised: 01/30/2023] [Accepted: 03/06/2023] [Indexed: 05/24/2023]
Abstract
Systematic reviews and meta-analysis are the cornerstones of evidence-based decision making and priority setting. However, traditional systematic reviews are time and labour intensive, limiting their feasibility to comprehensively evaluate the latest evidence in research-intensive areas. Recent developments in automation, machine learning and systematic review technologies have enabled efficiency gains. Building upon these advances, we developed Systematic Online Living Evidence Summaries (SOLES) to accelerate evidence synthesis. In this approach, we integrate automated processes to continuously gather, synthesise and summarise all existing evidence from a research domain, and report the resulting current curated content as interrogatable databases via interactive web applications. SOLES can benefit various stakeholders by (i) providing a systematic overview of current evidence to identify knowledge gaps, (ii) providing an accelerated starting point for a more detailed systematic review, and (iii) facilitating collaboration and coordination in evidence synthesis.
Collapse
Affiliation(s)
- Kaitlyn Hair
- Centre for Clinical Brain Sciences, The University of Edinburgh, Edinburgh, U.K
| | - Emma Wilson
- Centre for Clinical Brain Sciences, The University of Edinburgh, Edinburgh, U.K
| | - Charis Wong
- Anne Rowling Regenerative Neurology Clinic, University of Edinburgh, Edinburgh, U.K
- Euan Macdonald Centre for Motor Neuron Disease Research, University of Edinburgh, Edinburgh, U.K
| | - Anthony Tsang
- King’s Technology Evaluation Centre, King’s College London, U.K
| | - Malcolm Macleod
- Centre for Clinical Brain Sciences, The University of Edinburgh, Edinburgh, U.K
| | - Alexandra Bannach-Brown
- Charité Universitaetsmedizin Berlin, Berlin Institute of Health – QUEST Center, Berlin, Germany
| |
Collapse
|
6
|
Wilson E, Cruz F, Maclean D, Ghanawi J, McCann S, Brennan P, Liao J, Sena E, Macleod M. Screening for in vitro systematic reviews: a comparison of screening methods and training of a machine learning classifier. Clin Sci (Lond) 2023; 137:181-193. [PMID: 36630537 PMCID: PMC9885807 DOI: 10.1042/cs20220594] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2022] [Revised: 12/15/2022] [Accepted: 01/11/2023] [Indexed: 01/13/2023]
Abstract
OBJECTIVE Existing strategies to identify relevant studies for systematic review may not perform equally well across research domains. We compare four approaches based on either human or automated screening of either title and abstract or full text, and report the training of a machine learning algorithm to identify in vitro studies from bibliographic records. METHODS We used a systematic review of oxygen-glucose deprivation (OGD) in PC-12 cells to compare approaches. For human screening, two reviewers independently screened studies based on title and abstract or full text, with disagreements reconciled by a third. For automated screening, we applied text mining to either title and abstract or full text. We trained a machine learning algorithm with decisions from 2000 randomly selected PubMed Central records enriched with a dataset of known in vitro studies. RESULTS Full-text approaches performed best, with human (sensitivity: 0.990, specificity: 1.000 and precision: 0.994) outperforming text mining (sensitivity: 0.972, specificity: 0.980 and precision: 0.764). For title and abstract, text mining (sensitivity: 0.890, specificity: 0.995 and precision: 0.922) outperformed human screening (sensitivity: 0.862, specificity: 0.998 and precision: 0.975). At our target sensitivity of 95% the algorithm performed with specificity of 0.850 and precision of 0.700. CONCLUSION In this in vitro systematic review, human screening based on title and abstract erroneously excluded 14% of relevant studies, perhaps because title and abstract provide an incomplete description of methods used. Our algorithm might be used as a first selection phase in in vitro systematic reviews to limit the extent of full text screening required.
Collapse
Affiliation(s)
- Emma Wilson
- Centre for Clinical Brain Sciences, The University of Edinburgh, Edinburgh, U.K
| | - Florenz Cruz
- Berlin Institute of Health at Charité-Universitätsmedizin Berlin, QUEST Center, Berlin, Germany
| | - Duncan Maclean
- University of Edinburgh Medical School, University of Edinburgh, Edinburgh, U.K
| | | | - Sarah K. McCann
- Berlin Institute of Health at Charité-Universitätsmedizin Berlin, QUEST Center, Berlin, Germany
| | - Paul M. Brennan
- Centre for Clinical Brain Sciences, The University of Edinburgh, Edinburgh, U.K
| | - Jing Liao
- Centre for Clinical Brain Sciences, The University of Edinburgh, Edinburgh, U.K
| | - Emily S. Sena
- Centre for Clinical Brain Sciences, The University of Edinburgh, Edinburgh, U.K
| | - Malcolm Macleod
- Centre for Clinical Brain Sciences, The University of Edinburgh, Edinburgh, U.K
| |
Collapse
|
7
|
Bannach-Brown A, Hair K, Bahor Z, Soliman N, Macleod M, Liao J. Technological advances in preclinical meta-research. BMJ OPEN SCIENCE 2021; 5:e100131. [PMID: 35047701 PMCID: PMC8647618 DOI: 10.1136/bmjos-2020-100131] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/16/2023] Open
Affiliation(s)
- Alexandra Bannach-Brown
- Berlin Institute of Health, QUEST Center, Charité Universitätsmedizin Berlin, Berlin, Germany
- Institute for Evidence-Based Practice, Bond University, Robina, Queensland, Australia
| | - Kaitlyn Hair
- Centre for Clinical Brain Sciences, The University of Edinburgh Edinburgh Medical School, Edinburgh, Scotland, UK
| | - Zsanett Bahor
- Centre for Clinical Brain Sciences, The University of Edinburgh Edinburgh Medical School, Edinburgh, Scotland, UK
| | - Nadia Soliman
- Pain Research; Faculty of Medicine, Department of Surgery and Cancer, Imperial College London, London, Greater London, UK
| | - Malcolm Macleod
- Centre for Clinical Brain Sciences, The University of Edinburgh Edinburgh Medical School, Edinburgh, Scotland, UK
| | - Jing Liao
- Centre for Clinical Brain Sciences, The University of Edinburgh Edinburgh Medical School, Edinburgh, Scotland, UK
| |
Collapse
|