1
|
Tóth B, Berek L, Gulácsi L, Péntek M, Zrubka Z. Automation of systematic reviews of biomedical literature: a scoping review of studies indexed in PubMed. Syst Rev 2024; 13:174. [PMID: 38978132 PMCID: PMC11229257 DOI: 10.1186/s13643-024-02592-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/27/2023] [Accepted: 06/20/2024] [Indexed: 07/10/2024] Open
Abstract
BACKGROUND The demand for high-quality systematic literature reviews (SRs) for evidence-based medical decision-making is growing. SRs are costly and require the scarce resource of highly skilled reviewers. Automation technology has been proposed to save workload and expedite the SR workflow. We aimed to provide a comprehensive overview of SR automation studies indexed in PubMed, focusing on the applicability of these technologies in real world practice. METHODS In November 2022, we extracted, combined, and ran an integrated PubMed search for SRs on SR automation. Full-text English peer-reviewed articles were included if they reported studies on SR automation methods (SSAM), or automated SRs (ASR). Bibliographic analyses and knowledge-discovery studies were excluded. Record screening was performed by single reviewers, and the selection of full text papers was performed in duplicate. We summarized the publication details, automated review stages, automation goals, applied tools, data sources, methods, results, and Google Scholar citations of SR automation studies. RESULTS From 5321 records screened by title and abstract, we included 123 full text articles, of which 108 were SSAM and 15 ASR. Automation was applied for search (19/123, 15.4%), record screening (89/123, 72.4%), full-text selection (6/123, 4.9%), data extraction (13/123, 10.6%), risk of bias assessment (9/123, 7.3%), evidence synthesis (2/123, 1.6%), assessment of evidence quality (2/123, 1.6%), and reporting (2/123, 1.6%). Multiple SR stages were automated by 11 (8.9%) studies. The performance of automated record screening varied largely across SR topics. In published ASR, we found examples of automated search, record screening, full-text selection, and data extraction. In some ASRs, automation fully complemented manual reviews to increase sensitivity rather than to save workload. Reporting of automation details was often incomplete in ASRs. CONCLUSIONS Automation techniques are being developed for all SR stages, but with limited real-world adoption. Most SR automation tools target single SR stages, with modest time savings for the entire SR process and varying sensitivity and specificity across studies. Therefore, the real-world benefits of SR automation remain uncertain. Standardizing the terminology, reporting, and metrics of study reports could enhance the adoption of SR automation techniques in real-world practice.
Collapse
Affiliation(s)
- Barbara Tóth
- Doctoral School of Innovation Management, Óbuda University, Bécsi út 96/B, Budapest, 1034, Hungary
| | - László Berek
- Doctoral School for Safety and Security, Óbuda University, Bécsi út 96/B, Budapest, 1034, Hungary
- University Library, Óbuda University, Bécsi út 96/B, Budapest, 1034, Hungary
| | - László Gulácsi
- HECON Health Economics Research Center, University Research, and Innovation Center, Óbuda University, Bécsi út 96/B, Budapest, 1034, Hungary
| | - Márta Péntek
- HECON Health Economics Research Center, University Research, and Innovation Center, Óbuda University, Bécsi út 96/B, Budapest, 1034, Hungary
| | - Zsombor Zrubka
- HECON Health Economics Research Center, University Research, and Innovation Center, Óbuda University, Bécsi út 96/B, Budapest, 1034, Hungary.
| |
Collapse
|
2
|
Stevens ER, Laynor G. Enhancing the quality and efficiency of regulatory science literature reviews through innovation and collaboration with library and information science experts. Front Med (Lausanne) 2024; 11:1434427. [PMID: 39021816 PMCID: PMC11251899 DOI: 10.3389/fmed.2024.1434427] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/17/2024] [Accepted: 06/24/2024] [Indexed: 07/20/2024] Open
Affiliation(s)
- Elizabeth R. Stevens
- Department of Population Health, New York University Grossman School of Medicine, New York, NY, United States
| | - Gregory Laynor
- Health Sciences Library, New York University Grossman School of Medicine, New York, NY, United States
| |
Collapse
|
3
|
Khraisha Q, Put S, Kappenberg J, Warraitch A, Hadfield K. Can large language models replace humans in systematic reviews? Evaluating GPT-4's efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages. Res Synth Methods 2024; 15:616-626. [PMID: 38484744 DOI: 10.1002/jrsm.1715] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/10/2023] [Revised: 02/06/2024] [Accepted: 02/25/2024] [Indexed: 07/13/2024]
Abstract
Systematic reviews are vital for guiding practice, research and policy, although they are often slow and labour-intensive. Large language models (LLMs) could speed up and automate systematic reviews, but their performance in such tasks has yet to be comprehensively evaluated against humans, and no study has tested Generative Pre-Trained Transformer (GPT)-4, the biggest LLM so far. This pre-registered study uses a "human-out-of-the-loop" approach to evaluate GPT-4's capability in title/abstract screening, full-text review and data extraction across various literature types and languages. Although GPT-4 had accuracy on par with human performance in some tasks, results were skewed by chance agreement and dataset imbalance. Adjusting for these caused performance scores to drop across all stages: for data extraction, performance was moderate, and for screening, it ranged from none in highly balanced literature datasets (~1:1) to moderate in those datasets where the ratio of inclusion to exclusion in studies was imbalanced (~1:3). When screening full-text literature using highly reliable prompts, GPT-4's performance was more robust, reaching "human-like" levels. Although our findings indicate that, currently, substantial caution should be exercised if LLMs are being used to conduct systematic reviews, they also offer preliminary evidence that, for certain review tasks delivered under specific conditions, LLMs can rival human performance.
Collapse
Affiliation(s)
- Qusai Khraisha
- Trinity Centre for Global Health, Trinity College Dublin, Dublin, Ireland
- School of Psychology, Trinity College Dublin, Dublin, Ireland
| | - Sophie Put
- Department of Education, York University, York, UK
| | | | - Azza Warraitch
- Trinity Centre for Global Health, Trinity College Dublin, Dublin, Ireland
- School of Psychology, Trinity College Dublin, Dublin, Ireland
| | - Kristin Hadfield
- Trinity Centre for Global Health, Trinity College Dublin, Dublin, Ireland
- School of Psychology, Trinity College Dublin, Dublin, Ireland
| |
Collapse
|
4
|
Khalil H, Pollock D, McInerney P, Evans C, Moraes EB, Godfrey CM, Alexander L, Tricco A, Peters MDJ, Pieper D, Saran A, Ameen D, Taneri PE, Munn Z. Automation tools to support undertaking scoping reviews. Res Synth Methods 2024. [PMID: 38885942 DOI: 10.1002/jrsm.1731] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2023] [Revised: 05/15/2024] [Accepted: 06/02/2024] [Indexed: 06/20/2024]
Abstract
OBJECTIVE This paper describes several automation tools and software that can be considered during evidence synthesis projects and provides guidance for their integration in the conduct of scoping reviews. STUDY DESIGN AND SETTING The guidance presented in this work is adapted from the results of a scoping review and consultations with the JBI Scoping Review Methodology group. RESULTS This paper describes several reliable, validated automation tools and software that can be used to enhance the conduct of scoping reviews. Developments in the automation of systematic reviews, and more recently scoping reviews, are continuously evolving. We detail several helpful tools in order of the key steps recommended by the JBI's methodological guidance for undertaking scoping reviews including team establishment, protocol development, searching, de-duplication, screening titles and abstracts, data extraction, data charting, and report writing. While we include several reliable tools and software that can be used for the automation of scoping reviews, there are some limitations to the tools mentioned. For example, some are available in English only and their lack of integration with other tools results in limited interoperability. CONCLUSION This paper highlighted several useful automation tools and software programs to use in undertaking each step of a scoping review. This guidance has the potential to inform collaborative efforts aiming at the development of evidence informed, integrated automation tools and software packages for enhancing the conduct of high-quality scoping reviews.
Collapse
Affiliation(s)
- Hanan Khalil
- School of Psychology and Public Health, Department of Public Health, La Trobe University, Melbourne, Australia
- The Queensland Centre of Evidence Based Nursing and Midwifery: A JBI Centre of Excellence, Brisbane, Queensland, Australia
| | - Danielle Pollock
- JBI, University of Adelaide, Adelaide, Australia
- Health Evidence Synthesis, Recommendations and Impact (HESRI), School of Public Health, University of Adelaide, Adelaide, Australia
| | - Patricia McInerney
- The Wits JBI Centre for Evidence-Based Practice: A JBI Centre of Excellence, Faculty of Health Sciences, University of the Witwatersrand, South Africa
| | - Catrin Evans
- The Nottingham Centre for Evidence Based Healthcare: A JBI Centre of Excellence, University of Nottingham, UK
| | - Erica B Moraes
- Nursing School, Department of Nursing Fundamentals and Administration, Federal Fluminense University, Rio de Janeiro, Brazil
- The Brazilian Centre of Evidence-based Healthcare: A JBI Centre of Excellence - JBI, Brazil
| | - Christina M Godfrey
- Queen's Collaboration for Health Care Quality: A JBI Centre of Excellence, Queen's University School of Nursing, Kingston, Ontario, Canada
| | - Lyndsay Alexander
- The Scottish Centre for Evidence-based, Multi-Professional Practice: A JBI Centre of Excellence, Aberdeen, UK
- School of Health Sciences, Robert Gordon University, Aberdeen, UK
| | - Andrea Tricco
- Queen's Collaboration for Health Care Quality: A JBI Centre of Excellence, Queen's University School of Nursing, Kingston, Ontario, Canada
- Epidemiology Division and Institute for Health, Management, and Evaluation, Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, Toronto, Ontario, Canada
| | - Micah D J Peters
- Health Evidence Synthesis, Recommendations and Impact (HESRI), School of Public Health, University of Adelaide, Adelaide, Australia
- University of South Australia, Clinical and Health Sciences, Rosemary Bryant AO Research Centre, Adelaide, South Australia, Australia
- University of Adelaide, Faculty of Health and Medical Sciences, Adelaide Nursing School, Adelaide, Australia
| | - Dawid Pieper
- Faculty of Health Sciences Brandenburg, Brandenburg Medical School (Theodor Fontane), Institute for Health Services and Health System Research, Rüdersdorf, Germany
- Center for Health Services Research, Brandenburg Medical School (Theodor Fontane), Rüdersdorf, Germany
| | | | - Daniel Ameen
- Faculty of Medicine, Nursing and Health Sciences, School of Medicine, Monash University, Australia
| | - Petek Eylul Taneri
- HRB-Trials Methodology Research Network, College of Medicine, Nursing and Health Sciences, University of Galway, Galway, Ireland
| | - Zachary Munn
- JBI, University of Adelaide, Adelaide, Australia
- Health Evidence Synthesis, Recommendations and Impact (HESRI), School of Public Health, University of Adelaide, Adelaide, Australia
| |
Collapse
|
5
|
Antoniou M, Melagraki G, Lynch I, Afantitis A. In Vitro Toxicological Insights from the Biomedical Applications of Iron Carbide Nanoparticles in Tumor Theranostics: A Systematic Review and Meta-Analysis. NANOMATERIALS (BASEL, SWITZERLAND) 2024; 14:734. [PMID: 38727328 PMCID: PMC11085367 DOI: 10.3390/nano14090734] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/27/2024] [Revised: 04/13/2024] [Accepted: 04/14/2024] [Indexed: 05/12/2024]
Abstract
(1) Background: Despite the encouraging indications regarding the suitability (biocompatibility) of iron carbide nanoparticles (ICNPs) in various biomedical applications, the published evidence of their biosafety is dispersed and relatively sparse. The present review synthesizes the existing nanotoxicological data from in vitro studies relevant to the diagnosis and treatment of cancer. (2) Methods: A systematic review was performed in electronic databases (PubMed, Scopus, and Wiley Online Library) on December 2023, searching for toxicity assessments of ICNPs of different sizes, coatings, and surface modifications investigated in immortalized human and murine cell lines. The risk of bias in the studies was assessed using the ToxRTool for in vitro studies. (3) Results: Among the selected studies (n = 22), cell viability emerged as the most frequently assessed cellular-level toxicity endpoint. The results of the meta-analysis showed that cell models treated with ICNPs had a reduced cell viability (SMD = -2.531; 95% CI: -2.959 to -2.109) compared to untreated samples. A subgroup analysis was performed due to the high magnitude of heterogeneity (I2 = 77.1%), revealing that ICNP concentration and conjugated ligands are the factors that largely influence toxicity (p < 0.001). (4) Conclusions: A dose-dependent cytotoxicity of ICNP exposure was observed, regardless of the health status of the cell, tested organism, and NP size. Inconsistent reporting of ICNP physicochemical properties was noted, which hinders comparability among the studies. A comprehensive exploration of the available in vivo studies is required in future research to assess the safety of ICNPs' use in bioimaging and cancer treatment.
Collapse
Affiliation(s)
- Maria Antoniou
- Department of Nanoinformatics, NovaMechanics Ltd., Nicosia 1046, Cyprus;
- Entelos Institute, Larnaca 6059, Cyprus;
- The Cyprus Institute, Nicosia 2121, Cyprus
| | - Georgia Melagraki
- Division of Physical Sciences & Applications, Hellenic Military Academy, 16672 Vari, Greece;
| | - Iseult Lynch
- Entelos Institute, Larnaca 6059, Cyprus;
- School of Geography, Earth and Environmental Sciences, University of Birmingham Edgbaston, Birmingham B15 2TT, UK
| | - Antreas Afantitis
- Department of Nanoinformatics, NovaMechanics Ltd., Nicosia 1046, Cyprus;
- Entelos Institute, Larnaca 6059, Cyprus;
| |
Collapse
|
6
|
Malik S, Muhammad K, Waheed Y. Emerging Applications of Nanotechnology in Healthcare and Medicine. Molecules 2023; 28:6624. [PMID: 37764400 PMCID: PMC10536529 DOI: 10.3390/molecules28186624] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/15/2023] [Revised: 09/05/2023] [Accepted: 09/12/2023] [Indexed: 09/29/2023] Open
Abstract
Knowing the beneficial aspects of nanomedicine, scientists are trying to harness the applications of nanotechnology in diagnosis, treatment, and prevention of diseases. There are also potential uses in designing medical tools and processes for the new generation of medical scientists. The main objective for conducting this research review is to gather the widespread aspects of nanomedicine under one heading and to highlight standard research practices in the medical field. Comprehensive research has been conducted to incorporate the latest data related to nanotechnology in medicine and therapeutics derived from acknowledged scientific platforms. Nanotechnology is used to conduct sensitive medical procedures. Nanotechnology is showing successful and beneficial uses in the fields of diagnostics, disease treatment, regenerative medicine, gene therapy, dentistry, oncology, aesthetics industry, drug delivery, and therapeutics. A thorough association of and cooperation between physicians, clinicians, researchers, and technologies will bring forward a future where there is a more calculated, outlined, and technically programed field of nanomedicine. Advances are being made to overcome challenges associated with the application of nanotechnology in the medical field due to the pathophysiological basis of diseases. This review highlights the multipronged aspects of nanomedicine and how nanotechnology is proving beneficial for the health industry. There is a need to minimize the health, environmental, and ethical concerns linked to nanotechnology.
Collapse
Affiliation(s)
- Shiza Malik
- Bridging Health Foundation, Rawalpindi 46000, Pakistan
| | - Khalid Muhammad
- Department of Biology, College of Science, UAE University, Al Ain 15551, United Arab Emirates
| | - Yasir Waheed
- Office of Research, Innovation and Commercialization, Shaheed Zulfiqar Ali Bhutto Medical University, Islamabad 44000, Pakistan
- Gilbert and Rose-Marie Chagoury School of Medicine, Lebanese American University, Byblos 1401, Lebanon
| |
Collapse
|
7
|
Future perspectives for advancing regulatory science of nanotechnology-enabled health products. Drug Deliv Transl Res 2022; 12:2145-2156. [PMID: 35691982 PMCID: PMC9360093 DOI: 10.1007/s13346-022-01165-y] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 04/06/2022] [Indexed: 02/01/2023]
Abstract
The identification of regulatory challenges for nanotechnology-enabled health products, followed by discussions with the involved stakeholders, is the first step towards a strategic planning of how such challenges can be successfully addressed in the future. In order to better understand whether the identified regulatory needs are sector-specific for health products or might also hinder the progress in other domains, the REFINE consortium reached out to communities representing other sectors that also exploit the potential of nanotechnology, i.e. industrial chemicals, food and cosmetics. Through a series of trans-sectorial workshops, REFINE partners identified common as well as sector-specific challenges and discussed possible ways forward. Potential solutions lie in a more strengthen collaboration between regulatory and research communities resulting in a targeted production and exploitation of academic data for the regulatory decision-making. Furthermore, a coordinated use of knowledge sharing platforms and databases, trans-sectorial standardisation activities and harmonisation of regulatory activities between geographical regions are possible ways forward, in line with the upcoming European political initiatives such as the Chemical Strategy for Sustainability (CSS). Finally, we also discuss the perspectives for further development and sustainability of methods and tools developed in the REFINE project.
Collapse
|