1
|
Affengruber L, van der Maten MM, Spiero I, Nussbaumer-Streit B, Mahmić-Kaknjo M, Ellen ME, Goossen K, Kantorova L, Hooft L, Riva N, Poulentzas G, Lalagkas PN, Silva AG, Sassano M, Sfetcu R, Marqués ME, Friessova T, Baladia E, Pezzullo AM, Martinez P, Gartlehner G, Spijker R. An exploration of available methods and tools to improve the efficiency of systematic review production: a scoping review. BMC Med Res Methodol 2024; 24:210. [PMID: 39294580 PMCID: PMC11409535 DOI: 10.1186/s12874-024-02320-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/17/2024] [Accepted: 08/26/2024] [Indexed: 09/20/2024] Open
Abstract
BACKGROUND Systematic reviews (SRs) are time-consuming and labor-intensive to perform. With the growing number of scientific publications, the SR development process becomes even more laborious. This is problematic because timely SR evidence is essential for decision-making in evidence-based healthcare and policymaking. Numerous methods and tools that accelerate SR development have recently emerged. To date, no scoping review has been conducted to provide a comprehensive summary of methods and ready-to-use tools to improve efficiency in SR production. OBJECTIVE To present an overview of primary studies that evaluated the use of ready-to-use applications of tools or review methods to improve efficiency in the review process. METHODS We conducted a scoping review. An information specialist performed a systematic literature search in four databases, supplemented with citation-based and grey literature searching. We included studies reporting the performance of methods and ready-to-use tools for improving efficiency when producing or updating a SR in the health field. We performed dual, independent title and abstract screening, full-text selection, and data extraction. The results were analyzed descriptively and presented narratively. RESULTS We included 103 studies: 51 studies reported on methods, 54 studies on tools, and 2 studies reported on both methods and tools to make SR production more efficient. A total of 72 studies evaluated the validity (n = 69) or usability (n = 3) of one method (n = 33) or tool (n = 39), and 31 studies performed comparative analyses of different methods (n = 15) or tools (n = 16). 20 studies conducted prospective evaluations in real-time workflows. Most studies evaluated methods or tools that aimed at screening titles and abstracts (n = 42) and literature searching (n = 24), while for other steps of the SR process, only a few studies were found. Regarding the outcomes included, most studies reported on validity outcomes (n = 84), while outcomes such as impact on results (n = 23), time-saving (n = 24), usability (n = 13), and cost-saving (n = 3) were less often evaluated. CONCLUSION For title and abstract screening and literature searching, various evaluated methods and tools are available that aim at improving the efficiency of SR production. However, only few studies have addressed the influence of these methods and tools in real-world workflows. Few studies exist that evaluate methods or tools supporting the remaining tasks. Additionally, while validity outcomes are frequently reported, there is a lack of evaluation regarding other outcomes.
Collapse
Affiliation(s)
- Lisa Affengruber
- Cochrane Austria, Department for Evidence-Based Medicine and Clinical Epidemiology, University for Continuing Education Krems, Krems an der Donau, Austria.
- School for Public Health and Primary Care (CAPHRI), Maastricht University, Maastricht, the Netherlands.
| | - Miriam M van der Maten
- Knowledge Institute of Federation of Medical Specialists, Utrecht, The Netherlands
- Cochrane Netherlands, Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands
| | - Isa Spiero
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands
| | - Barbara Nussbaumer-Streit
- Cochrane Austria, Department for Evidence-Based Medicine and Clinical Epidemiology, University for Continuing Education Krems, Krems an der Donau, Austria
| | - Mersiha Mahmić-Kaknjo
- Zenica Cantonal Hospital, Department for Clinical Pharmacology, Zenica, Bosnia and Herzegovina
| | - Moriah E Ellen
- Department of Health Policy and Management, Guilford Glazer Faculty of Business and Management and Faculty of Health Sciences, Ben-Gurion University of the Negev, Beer-Sheva, Israel
- Institute of Health Policy Management and Evaluation, Dalla Lana School Of Public Health, University of Toronto, Toronto, Canada
- McMaster Health Forum, McMaster University, Hamilton, Canada
| | - Käthe Goossen
- Witten/Herdecke University, Institute for Research in Operative Medicine (IFOM), Cologne, Germany
| | - Lucia Kantorova
- Czech National Centre for Evidence-Based Healthcare and Knowledge Translation (Cochrane Czech Republic, Czech CEBHC: JBI Centre of Excellence, Masaryk University GRADE Centre), Institute of Biostatistics and Analyses, Faculty of Medicine, Masaryk University, Brno, Czech Republic
| | - Lotty Hooft
- Cochrane Netherlands, Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands
| | - Nicoletta Riva
- Department of Pathology, Faculty of Medicine and Surgery, University of Malta, Msida, Malta
| | - Georgios Poulentzas
- Laboratory of Hygiene and Environmental Protection, Department of Medicine, Democritus University of Thrace, Alexandroupolis, Greece
| | - Panagiotis Nikolaos Lalagkas
- Laboratory of Hygiene and Environmental Protection, Department of Medicine, Democritus University of Thrace, Alexandroupolis, Greece
| | - Anabela G Silva
- CINTESIS.RISE@UA, University of Aveiro, Campus Universitário de Santiago, Aveiro, Portugal
| | - Michele Sassano
- Section of Hygiene, University Department of Life Sciences and Public Health, Università Cattolica del Sacro Cuore, Rome, Italy
- Department of Medical and Surgical Sciences, University of Bologna, Bologna, Italy
| | - Raluca Sfetcu
- National Institute for Health Services Management, Bucharest, Romania
- Spiru Haret University, Faculty of Psychology and Educational Sciences, Bucharest, Romania
| | - María E Marqués
- Red de Nutrición Basada en La Evidencia, Academia Española de Nutrición y Dietética, Pamplona, Spain
| | - Tereza Friessova
- Department of Health Sciences, Faculty of Medicine, Masaryk University, Brno, Czech Republic
| | - Eduard Baladia
- Red de Nutrición Basada en La Evidencia, Academia Española de Nutrición y Dietética, Pamplona, Spain
| | - Angelo Maria Pezzullo
- Section of Hygiene, University Department of Life Sciences and Public Health, Università Cattolica del Sacro Cuore, Rome, Italy
| | - Patricia Martinez
- Red de Nutrición Basada en La Evidencia, Academia Española de Nutrición y Dietética, Pamplona, Spain
- Techné Research Group, Department of Knowledge Engineering of the Faculty of Science, University of Granada, Granada, Spain
| | - Gerald Gartlehner
- Cochrane Austria, Department for Evidence-Based Medicine and Clinical Epidemiology, University for Continuing Education Krems, Krems an der Donau, Austria
- RTI International, Center for Public Health Methods, Research Triangle Park, Durham, NC, USA
| | - René Spijker
- Cochrane Netherlands, Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands
- Amsterdam UMC, University of Amsterdam, Medical Library, Amsterdam Public Health, Amsterdam, the Netherlands
| |
Collapse
|
2
|
Forbes C, Greenwood H, Carter M, Clark J. Automation of duplicate record detection for systematic reviews: Deduplicator. Syst Rev 2024; 13:206. [PMID: 39095913 PMCID: PMC11295717 DOI: 10.1186/s13643-024-02619-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/22/2023] [Accepted: 07/18/2024] [Indexed: 08/04/2024] Open
Abstract
BACKGROUND To describe the algorithm and investigate the efficacy of a novel systematic review automation tool "the Deduplicator" to remove duplicate records from a multi-database systematic review search. METHODS We constructed and tested the efficacy of the Deduplicator tool by using 10 previous Cochrane systematic review search results to compare the Deduplicator's 'balanced' algorithm to a semi-manual EndNote method. Two researchers each performed deduplication on the 10 libraries of search results. For five of those libraries, one researcher used the Deduplicator, while the other performed semi-manual deduplication with EndNote. They then switched methods for the remaining five libraries. In addition to this analysis, comparison between the three different Deduplicator algorithms ('balanced', 'focused' and 'relaxed') was performed on two datasets of previously deduplicated search results. RESULTS Before deduplication, the mean library size for the 10 systematic reviews was 1962 records. When using the Deduplicator, the mean time to deduplicate was 5 min per 1000 records compared to 15 min with EndNote. The mean error rate with Deduplicator was 1.8 errors per 1000 records in comparison to 3.1 with EndNote. Evaluation of the different Deduplicator algorithms found that the 'balanced' algorithm had the highest mean F1 score of 0.9647. The 'focused' algorithm had the highest mean accuracy of 0.9798 and the highest recall of 0.9757. The 'relaxed' algorithm had the highest mean precision of 0.9896. CONCLUSIONS This demonstrates that using the Deduplicator for duplicate record detection reduces the time taken to deduplicate, while maintaining or improving accuracy compared to using a semi-manual EndNote method. However, further research should be performed comparing more deduplication methods to establish relative performance of the Deduplicator against other deduplication methods.
Collapse
Affiliation(s)
- Connor Forbes
- Institute for Evidence-Based Healthcare, Bond University, Gold Coast, Australia.
| | - Hannah Greenwood
- Institute for Evidence-Based Healthcare, Bond University, Gold Coast, Australia
| | - Matt Carter
- Institute for Evidence-Based Healthcare, Bond University, Gold Coast, Australia
| | - Justin Clark
- Institute for Evidence-Based Healthcare, Bond University, Gold Coast, Australia
| |
Collapse
|
3
|
Yu T, Yang X, Clark J, Lin L, Furuya-Kanamori L, Xu C. Accelerating evidence synthesis for safety assessment through ClinicalTrials.gov platform: a feasibility study. BMC Med Res Methodol 2024; 24:165. [PMID: 39080524 PMCID: PMC11290241 DOI: 10.1186/s12874-024-02225-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/08/2023] [Accepted: 04/18/2024] [Indexed: 08/02/2024] Open
Abstract
BACKGROUND Standard systematic review can be labor-intensive and time-consuming meaning that it can be difficult to provide timely evidence when there is an urgent public health emergency such as a pandemic. The ClinicalTrials.gov provides a promising way to accelerate evidence production. METHODS We conducted a search on PubMed to gather systematic reviews containing a minimum of 5 studies focused on safety aspects derived from randomized controlled trials (RCTs) of pharmacological interventions, aiming to establish a real-world dataset. The registration information of each trial from eligible reviews was further collected and verified. The meta-analytic data were then re-analyzed by using 1) the full meta-analytic data with all trials and 2) emulated rapid data with trials that had been registered and posted results on ClinicalTrials.gov, under the same synthesis methods. The effect estimates of the full meta-analysis and rapid meta-analysis were then compared. RESULTS The real-world dataset comprises 558 meta-analyses. Among them, 56 (10.0%) meta-analyses included RCTs that were not registered in ClinicalTrials.gov. For the remaining 502 meta-analyses, the median percentage of RCTs registered within each meta-analysis is 70.1% (interquartile range: 33.3% to 88.9%). Under a 20% bias threshold, rapid meta-analyses conducted through ClinicalTrials.gov achieved accurate point estimates ranging from 77.4% (using the MH model) to 83.1% (using the GLMM model); 91.0% to 95.3% of these analyses accurately predicted the direction of effects. CONCLUSIONS Utilizing the ClinicalTrials.gov platform for safety assessment with a minimum of 5 RCTs holds significant potential for accelerating evidence synthesis to support urgent decision-making.
Collapse
Affiliation(s)
- Tianqi Yu
- Center of Research in Epidemiology and Statistics, Université Paris Cité, Inserm, Paris, France
| | - Xi Yang
- Department of Maternal, Child and Adolescent Health, School of Public Health, Anhui Medical University, Hefei, Anhui, China
| | - Justin Clark
- Institute for Evidence-Based Healthcare, Faculty of Health Sciences and Medicine, Bond University, Gold Coast, QLD, Australia
| | - Lifeng Lin
- Department of Epidemiology and Biostatistics, University of Arizona, Tucson, AZ, USA
| | - Luis Furuya-Kanamori
- UQ Centre for Clinical Research, The University of Queensland, Herston, Australia
| | - Chang Xu
- Proof of Concept Center, Eastern Hepatobiliary Surgery Hospital, Third Affiliated Hospital, Second Military Medical University, Naval Medical University, Shanghai, China.
| |
Collapse
|
4
|
Affengruber L, Nussbaumer-Streit B, Hamel C, Van der Maten M, Thomas J, Mavergames C, Spijker R, Gartlehner G. Rapid review methods series: Guidance on the use of supportive software. BMJ Evid Based Med 2024; 29:264-271. [PMID: 38242566 PMCID: PMC11287527 DOI: 10.1136/bmjebm-2023-112530] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 12/22/2023] [Indexed: 01/21/2024]
Abstract
This paper is part of a series of methodological guidance from the Cochrane Rapid Reviews Methods Group. Rapid reviews (RRs) use modified systematic review methods to accelerate the review process while maintaining systematic, transparent and reproducible methods. This paper guides how to use supportive software for RRs.We strongly encourage the use of supportive software throughout RR production. Specifically, we recommend (1) using collaborative online platforms that enable working in parallel, allow for real-time project management and centralise review details; (2) using automation software to support, but not entirely replace a human reviewer and human judgement and (3) being transparent in reporting the methodology and potential risk for bias due to the use of supportive software.
Collapse
Affiliation(s)
- Lisa Affengruber
- Department for Evidence-based Medicine and Evaluation, Cochrane Austria, University for Continuing Education Krems, Krems, Austria
- Department of Family Medicine, Maastricht University, Maastricht, The Netherlands
| | - Barbara Nussbaumer-Streit
- Department for Evidence-based Medicine and Evaluation, Cochrane Austria, University for Continuing Education Krems, Krems, Austria
| | - Candyce Hamel
- Canadian Association of Radiologists, Ottawa, Ontario, Canada
- School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada
| | - Miriam Van der Maten
- Knowledge Institute, Dutch Association of Medical Specialists, Utrecht, The Netherlands
| | - James Thomas
- University College London, UCL Social Research Institute, London, UK
| | | | - Rene Spijker
- Cochrane Netherlands, Julius Center for Health Sciences and Primary Care, UMC Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Gerald Gartlehner
- Department for Evidence-based Medicine and Evaluation, Cochrane Austria, University for Continuing Education Krems, Krems, Austria
- Center for Public Health Methods, RTI International, Research Triangle Park, North Carolina, USA
| |
Collapse
|
5
|
Chaboyer W, Latimer S, Priyadarshani U, Harbeck E, Patton D, Sim J, Moore Z, Deakin J, Carlini J, Lovegrove J, Jahandideh S, Gillespie BM. The effect of pressure injury prevention care bundles on pressure injuries in hospital patients: A complex intervention systematic review and meta-analysis. Int J Nurs Stud 2024; 155:104768. [PMID: 38642429 DOI: 10.1016/j.ijnurstu.2024.104768] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2023] [Revised: 03/25/2024] [Accepted: 03/27/2024] [Indexed: 04/22/2024]
Abstract
BACKGROUND Numerous interventions for pressure injury prevention have been developed, including care bundles. OBJECTIVE To systematically review the effectiveness of pressure injury prevention care bundles on pressure injury prevalence, incidence, and hospital-acquired pressure injury rate in hospitalised patients. DATA SOURCES The Medical Literature Analysis and Retrieval System Online (via PubMed), the Cumulative Index to Nursing and Allied Health Literature, EMBASE, Scopus, the Cochrane Library and two registries were searched (from 2009 to September 2023). STUDY ELIGIBILITY CRITERIA Randomised controlled trials and non-randomised studies with a comparison group published in English after 2008 were included. Studies reporting on the frequency of pressure injuries where the number of patients was not the numerator or denominator, or where the denominator was not reported, and single subgroups of hospitalised patients were excluded. Educational programmes targeting healthcare professionals and bundles targeting specific types of pressure injuries were excluded. PARTICIPANTS AND INTERVENTIONS Bundles with ≥3 components directed towards patients and implemented in ≥2 hospital services were included. STUDY APPRAISAL AND SYNTHESIS METHODS Screening, data extraction and risk of bias assessments were undertaken independently by two researchers. Random effects meta-analyses were conducted. The certainty of the body of evidence was assessed using Grading of Recommendations, Assessment, Development and Evaluation. RESULTS Nine studies (seven non-randomised with historical controls; two randomised) conducted in eight countries were included. There were four to eight bundle components; most were core, and only a few were discretionary. Various strategies were used prior to (six studies), during (five studies) and after (two studies) implementation to embed the bundles. The pooled risk ratio for pressure injury prevalence (five non-randomised studies) was 0.55 (95 % confidence intervals 0.29-1.03), and for hospital-acquired pressure injury rate (five non-randomised studies) it was 0.31 (95 % confidence intervals 0.12-0.83). All non-randomised studies were at high risk of bias, with very low certainty of evidence. In the two randomised studies, the care bundles had non-significant effects on hospital-acquired pressure injury incidence density, but data could not be pooled. CONCLUSIONS AND IMPLICATIONS OF KEY FINDINGS Whilst some studies showed decreases in pressure injuries, this evidence was very low certainty. The potential benefits of adding emerging evidence-based components to bundles should be considered. Future effectiveness studies should include contemporaneous controls and the development of a comprehensive, theory and evidence-informed implementation plan. SYSTEMATIC REVIEW REGISTRATION NUMBER PROSPERO CRD42023423058. TWEETABLE ABSTRACT Pressure injury prevention care bundles decrease hospital-acquired pressure injuries, but the certainty of this evidence is very low.
Collapse
Affiliation(s)
- Wendy Chaboyer
- NHMRC Centre of Research Excellence in Wiser Wound Care, Griffith University, Gold Coast Campus, Queensland 4222, Australia; School of Nursing and Midwifery, Griffith University, Gold Coast Campus, Queensland 4222, Australia.
| | - Sharon Latimer
- NHMRC Centre of Research Excellence in Wiser Wound Care, Griffith University, Gold Coast Campus, Queensland 4222, Australia; School of Nursing and Midwifery, Griffith University, Gold Coast Campus, Queensland 4222, Australia. https://twitter.com/SharonLLatimer
| | - Udeshika Priyadarshani
- NHMRC Centre of Research Excellence in Wiser Wound Care, Griffith University, Gold Coast Campus, Queensland 4222, Australia; School of Nursing and Midwifery, Griffith University, Gold Coast Campus, Queensland 4222, Australia; Department of Nursing and Midwifery, Faculty of Allied Health Sciences, General Sir John Kotelawala Defence University, Sri Lanka
| | - Emma Harbeck
- NHMRC Centre of Research Excellence in Wiser Wound Care, Griffith University, Gold Coast Campus, Queensland 4222, Australia
| | - Declan Patton
- School of Nursing & Midwifery, Royal College of Surgeons Ireland, University of Medicine and Health Sciences, 123 St Stephens's Green, Dublin, 2, Ireland
| | - Jenny Sim
- Faculty of Health, University of Technology Sydney, 235 Jones Street, Ultimo, NSW 2007, Australia; School of Nursing, Midwifery & Paramedicine, Australian Catholic University, North Sydney Australia
| | - Zena Moore
- School of Nursing & Midwifery, Royal College of Surgeons Ireland, University of Medicine and Health Sciences, 123 St Stephens's Green, Dublin, 2, Ireland
| | - Jodie Deakin
- NHMRC Centre of Research Excellence in Wiser Wound Care, Griffith University, Gold Coast Campus, Queensland 4222, Australia; School of Nursing and Midwifery, Griffith University, Gold Coast Campus, Queensland 4222, Australia. https://twitter.com/jodie_deakin3
| | - Joan Carlini
- NHMRC Centre of Research Excellence in Wiser Wound Care, Griffith University, Gold Coast Campus, Queensland 4222, Australia; Health Consumer and Department of Marketing, Griffith Business School, Griffith University, Gold Coast Campus, Queensland 4222, Australia
| | - Josephine Lovegrove
- NHMRC Centre of Research Excellence in Wiser Wound Care, Griffith University, Gold Coast Campus, Queensland 4222, Australia
| | - Sepideh Jahandideh
- NHMRC Centre of Research Excellence in Wiser Wound Care, Griffith University, Gold Coast Campus, Queensland 4222, Australia
| | - Brigid M Gillespie
- NHMRC Centre of Research Excellence in Wiser Wound Care, Griffith University, Gold Coast Campus, Queensland 4222, Australia; School of Nursing and Midwifery, Griffith University, Gold Coast Campus, Queensland 4222, Australia; Gold Coast University Hospital and Health Service, Gold Coast, QLD, Australia. https://twitter.com/bgillespie6
| |
Collapse
|
6
|
Khalil H, Pollock D, McInerney P, Evans C, Moraes EB, Godfrey CM, Alexander L, Tricco A, Peters MDJ, Pieper D, Saran A, Ameen D, Taneri PE, Munn Z. Automation tools to support undertaking scoping reviews. Res Synth Methods 2024. [PMID: 38885942 DOI: 10.1002/jrsm.1731] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2023] [Revised: 05/15/2024] [Accepted: 06/02/2024] [Indexed: 06/20/2024]
Abstract
OBJECTIVE This paper describes several automation tools and software that can be considered during evidence synthesis projects and provides guidance for their integration in the conduct of scoping reviews. STUDY DESIGN AND SETTING The guidance presented in this work is adapted from the results of a scoping review and consultations with the JBI Scoping Review Methodology group. RESULTS This paper describes several reliable, validated automation tools and software that can be used to enhance the conduct of scoping reviews. Developments in the automation of systematic reviews, and more recently scoping reviews, are continuously evolving. We detail several helpful tools in order of the key steps recommended by the JBI's methodological guidance for undertaking scoping reviews including team establishment, protocol development, searching, de-duplication, screening titles and abstracts, data extraction, data charting, and report writing. While we include several reliable tools and software that can be used for the automation of scoping reviews, there are some limitations to the tools mentioned. For example, some are available in English only and their lack of integration with other tools results in limited interoperability. CONCLUSION This paper highlighted several useful automation tools and software programs to use in undertaking each step of a scoping review. This guidance has the potential to inform collaborative efforts aiming at the development of evidence informed, integrated automation tools and software packages for enhancing the conduct of high-quality scoping reviews.
Collapse
Affiliation(s)
- Hanan Khalil
- School of Psychology and Public Health, Department of Public Health, La Trobe University, Melbourne, Australia
- The Queensland Centre of Evidence Based Nursing and Midwifery: A JBI Centre of Excellence, Brisbane, Queensland, Australia
| | - Danielle Pollock
- JBI, University of Adelaide, Adelaide, Australia
- Health Evidence Synthesis, Recommendations and Impact (HESRI), School of Public Health, University of Adelaide, Adelaide, Australia
| | - Patricia McInerney
- The Wits JBI Centre for Evidence-Based Practice: A JBI Centre of Excellence, Faculty of Health Sciences, University of the Witwatersrand, South Africa
| | - Catrin Evans
- The Nottingham Centre for Evidence Based Healthcare: A JBI Centre of Excellence, University of Nottingham, UK
| | - Erica B Moraes
- Nursing School, Department of Nursing Fundamentals and Administration, Federal Fluminense University, Rio de Janeiro, Brazil
- The Brazilian Centre of Evidence-based Healthcare: A JBI Centre of Excellence - JBI, Brazil
| | - Christina M Godfrey
- Queen's Collaboration for Health Care Quality: A JBI Centre of Excellence, Queen's University School of Nursing, Kingston, Ontario, Canada
| | - Lyndsay Alexander
- The Scottish Centre for Evidence-based, Multi-Professional Practice: A JBI Centre of Excellence, Aberdeen, UK
- School of Health Sciences, Robert Gordon University, Aberdeen, UK
| | - Andrea Tricco
- Queen's Collaboration for Health Care Quality: A JBI Centre of Excellence, Queen's University School of Nursing, Kingston, Ontario, Canada
- Epidemiology Division and Institute for Health, Management, and Evaluation, Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, Toronto, Ontario, Canada
| | - Micah D J Peters
- Health Evidence Synthesis, Recommendations and Impact (HESRI), School of Public Health, University of Adelaide, Adelaide, Australia
- University of South Australia, Clinical and Health Sciences, Rosemary Bryant AO Research Centre, Adelaide, South Australia, Australia
- University of Adelaide, Faculty of Health and Medical Sciences, Adelaide Nursing School, Adelaide, Australia
| | - Dawid Pieper
- Faculty of Health Sciences Brandenburg, Brandenburg Medical School (Theodor Fontane), Institute for Health Services and Health System Research, Rüdersdorf, Germany
- Center for Health Services Research, Brandenburg Medical School (Theodor Fontane), Rüdersdorf, Germany
| | | | - Daniel Ameen
- Faculty of Medicine, Nursing and Health Sciences, School of Medicine, Monash University, Australia
| | - Petek Eylul Taneri
- HRB-Trials Methodology Research Network, College of Medicine, Nursing and Health Sciences, University of Galway, Galway, Ireland
| | - Zachary Munn
- JBI, University of Adelaide, Adelaide, Australia
- Health Evidence Synthesis, Recommendations and Impact (HESRI), School of Public Health, University of Adelaide, Adelaide, Australia
| |
Collapse
|
7
|
Witte C, Schmidt DM, Cimiano P. Comparing generative and extractive approaches to information extraction from abstracts describing randomized clinical trials. J Biomed Semantics 2024; 15:3. [PMID: 38654304 PMCID: PMC11036632 DOI: 10.1186/s13326-024-00305-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2024] [Accepted: 04/05/2024] [Indexed: 04/25/2024] Open
Abstract
BACKGROUND Systematic reviews of Randomized Controlled Trials (RCTs) are an important part of the evidence-based medicine paradigm. However, the creation of such systematic reviews by clinical experts is costly as well as time-consuming, and results can get quickly outdated after publication. Most RCTs are structured based on the Patient, Intervention, Comparison, Outcomes (PICO) framework and there exist many approaches which aim to extract PICO elements automatically. The automatic extraction of PICO information from RCTs has the potential to significantly speed up the creation process of systematic reviews and this way also benefit the field of evidence-based medicine. RESULTS Previous work has addressed the extraction of PICO elements as the task of identifying relevant text spans or sentences, but without populating a structured representation of a trial. In contrast, in this work, we treat PICO elements as structured templates with slots to do justice to the complex nature of the information they represent. We present two different approaches to extract this structured information from the abstracts of RCTs. The first approach is an extractive approach based on our previous work that is extended to capture full document representations as well as by a clustering step to infer the number of instances of each template type. The second approach is a generative approach based on a seq2seq model that encodes the abstract describing the RCT and uses a decoder to infer a structured representation of a trial including its arms, treatments, endpoints and outcomes. Both approaches are evaluated with different base models on a manually annotated dataset consisting of RCT abstracts on an existing dataset comprising 211 annotated clinical trial abstracts for Type 2 Diabetes and Glaucoma. For both diseases, the extractive approach (with flan-t5-base) reached the best F 1 score, i.e. 0.547 ( ± 0.006 ) for type 2 diabetes and 0.636 ( ± 0.006 ) for glaucoma. Generally, the F 1 scores were higher for glaucoma than for type 2 diabetes and the standard deviation was higher for the generative approach. CONCLUSION In our experiments, both approaches show promising performance extracting structured PICO information from RCTs, especially considering that most related work focuses on the far easier task of predicting less structured objects. In our experimental results, the extractive approach performs best in both cases, although the lead is greater for glaucoma than for type 2 diabetes. For future work, it remains to be investigated how the base model size affects the performance of both approaches in comparison. Although the extractive approach currently leaves more room for direct improvements, the generative approach might benefit from larger models.
Collapse
Affiliation(s)
- Christian Witte
- Semantic Computing Group, Center for Cognitive Interaction Technology, Bielefeld University, Inspiration 1, Bielefeld, 33619, NRW, Germany
| | - David M Schmidt
- Semantic Computing Group, Center for Cognitive Interaction Technology, Bielefeld University, Inspiration 1, Bielefeld, 33619, NRW, Germany.
| | - Philipp Cimiano
- Semantic Computing Group, Center for Cognitive Interaction Technology, Bielefeld University, Inspiration 1, Bielefeld, 33619, NRW, Germany
| |
Collapse
|
8
|
McSween-Cadieux E, Lane J, Hong QN, Houle AA, Lauzier-Jobin F, Saint-Pierre Mousset E, Prigent O, Ziam S, Poder T, Lesage A, Dagenais P. Production and use of rapid responses during the COVID-19 pandemic in Quebec (Canada): perspectives from evidence synthesis producers and decision makers. Health Res Policy Syst 2024; 22:22. [PMID: 38351054 PMCID: PMC10863098 DOI: 10.1186/s12961-024-01105-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2023] [Accepted: 01/08/2024] [Indexed: 02/16/2024] Open
Abstract
BACKGROUND The COVID-19 pandemic has required evidence to be made available more rapidly than usual, in order to meet the needs of decision makers in a timely manner. These exceptional circumstances have caused significant challenges for organizations and teams responsible for evidence synthesis. They had to adapt to provide rapid responses to support decision-making. This study aimed to document (1) the challenges and adaptations made to produce rapid responses during the pandemic, (2) their perceived usefulness, reported use and factors influencing their use and (3) the methodological adaptations made to produce rapid responses. METHODS A qualitative study was conducted in 2021 with eight organizations in the health and social services system in Quebec (Canada), including three institutes with a provincial mandate. Data collection included focus groups (n = 9 groups in 8 organizations with 64 participants), interviews with decision makers (n = 12), and a document analysis of COVID-19 rapid responses (n = 128). A thematic analysis of qualitative data (objectives 1 and 2) and a descriptive analysis of documents (objective 3) were conducted. RESULTS The results highlight the teams and organizations' agility to deal with the many challenges encountered during the pandemic (e.g., increased their workloads, adoption of new technological tools or work processes, improved collaboration, development of scientific monitoring, adaptation of evidence synthesis methodologies and products). The challenge of balancing rigor and speed was reported by teams and organizations. When available at the right time, rapid responses have been reported as a useful tool for informing or justifying decisions in a context of uncertainty. Several factors that may influence their use were identified (e.g., clearly identify needs, interactions with producers, perceived rigor and credibility, precise and feasible recommendations). Certain trends in the methodological approaches used to speed up the evidence synthesis process were identified. CONCLUSIONS This study documented rapid responses producers' experiences during the COVID-19 pandemic in Quebec, and decision makers who requested, consulted, or used these products. Potential areas of improvements are identified such as reinforce coordination, improve communication loops, clarify guidelines or methodological benchmarks, and enhance utility of rapid response products for decision makers.
Collapse
Affiliation(s)
- Esther McSween-Cadieux
- Department of School and Social Adaptation Studies, Faculty of Education, Université de Sherbrooke, Sherbrooke, Canada.
- Centre RBC d'expertise Universitaire en Santé Mentale, Université de Sherbrooke, Sherbrooke, Canada.
| | - Julie Lane
- Department of School and Social Adaptation Studies, Faculty of Education, Université de Sherbrooke, Sherbrooke, Canada
- Centre RBC d'expertise Universitaire en Santé Mentale, Université de Sherbrooke, Sherbrooke, Canada
| | - Quan Nha Hong
- School of Rehabilitation, Université de Montréal, Montreal, Canada
| | - Andrée-Anne Houle
- Centre RBC d'expertise Universitaire en Santé Mentale, Université de Sherbrooke, Sherbrooke, Canada
- Department of Psychoeducation, Université de Sherbrooke, Sherbrooke, Canada
| | - François Lauzier-Jobin
- Centre RBC d'expertise Universitaire en Santé Mentale, Université de Sherbrooke, Sherbrooke, Canada
| | - Eliane Saint-Pierre Mousset
- Department of School and Social Adaptation Studies, Faculty of Education, Université de Sherbrooke, Sherbrooke, Canada
- Centre RBC d'expertise Universitaire en Santé Mentale, Université de Sherbrooke, Sherbrooke, Canada
| | - Ollivier Prigent
- Department of School and Social Adaptation Studies, Faculty of Education, Université de Sherbrooke, Sherbrooke, Canada
| | - Saliha Ziam
- School of Business Administration, Université TÉLUQ, Montreal, Canada
| | - Thomas Poder
- Centre de Recherche de l'Institut Universitaire en Santé Mentale de Montréal (CR-IUSMM), CIUSSS-de-l'Est-de-l'île-de-Montréal, Montreal, Canada
- Department of Management, Evaluation and Health Policy, School of Public Health, University of Montreal, Montreal, Canada
| | - Alain Lesage
- Centre de Recherche de l'Institut Universitaire en Santé Mentale de Montréal (CR-IUSMM), CIUSSS-de-l'Est-de-l'île-de-Montréal, Montreal, Canada
| | - Pierre Dagenais
- Department of Medicine, Faculty of Medicine and Health Science, University of Sherbrooke, Sherbrooke, Canada
| |
Collapse
|
9
|
Chiu SUF, Huang RY, Chiu CC. A commentary on 'Meta-research studies in surgery: a field that should be encouraged to assess and improve the quality of surgical evidence'. Int J Surg 2024; 110:604-605. [PMID: 37800556 PMCID: PMC10793774 DOI: 10.1097/js9.0000000000000800] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2023] [Accepted: 09/17/2023] [Indexed: 10/07/2023]
Affiliation(s)
- Si-Un Frank Chiu
- Department of Computer Science
- Department of Economics, Brown University, Providence, Rhode Island, USA
| | - Ru-Yi Huang
- Department of Family Medicine
- School of Medicine, College of Medicine, I-Shou University, Kaohsiung, Taiwan
| | - Chong-Chi Chiu
- Department of Medical Education and Research
- Department of General Surgery, E-Da Cancer Hospital
- School of Medicine, College of Medicine, I-Shou University, Kaohsiung, Taiwan
| |
Collapse
|
10
|
Liu S, Bourgeois FT, Narang C, Dunn AG. A comparison of machine learning methods to find clinical trials for inclusion in new systematic reviews from their PROSPERO registrations prior to searching and screening. Res Synth Methods 2024; 15:73-85. [PMID: 37749068 PMCID: PMC10872991 DOI: 10.1002/jrsm.1672] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2022] [Revised: 08/13/2023] [Accepted: 09/08/2023] [Indexed: 09/27/2023]
Abstract
Searching for trials is a key task in systematic reviews and a focus of automation. Previous approaches required knowing examples of relevant trials in advance, and most methods are focused on published trial articles. To complement existing tools, we compared methods for finding relevant trial registrations given a International Prospective Register of Systematic Reviews (PROSPERO) entry and where no relevant trials have been screened for inclusion in advance. We compared SciBERT-based (extension of Bidirectional Encoder Representations from Transformers) PICO extraction, MetaMap, and term-based representations using an imperfect dataset mined from 3632 PROSPERO entries connected to a subset of 65,662 trial registrations and 65,834 trial articles known to be included in systematic reviews. Performance was measured by the median rank and recall by rank of trials that were eventually included in the published systematic reviews. When ranking trial registrations relative to PROSPERO entries, 296 trial registrations needed to be screened to identify half of the relevant trials, and the best performing approach used a basic term-based representation. When ranking trial articles relative to PROSPERO entries, 162 trial articles needed to be screened to identify half of the relevant trials, and the best-performing approach used a term-based representation. The results show that MetaMap and term-based representations outperformed approaches that included PICO extraction for this use case. The results suggest that when starting with a PROSPERO entry and where no trials have been screened for inclusion, automated methods can reduce workload, but additional processes are still needed to efficiently identify trial registrations or trial articles that meet the inclusion criteria of a systematic review.
Collapse
Affiliation(s)
- Shifeng Liu
- Biomedical Informatics and Digital Health, Faculty of Medicine and Health, The University of Sydney, Sydney, New South Wales, Australia
| | - Florence T Bourgeois
- Computational Health Informatics Program, Boston Children's Hospital, Boston, Massachusetts, USA
- Department of Pediatrics, Harvard Medical School, Boston, Massachusetts, USA
| | - Claire Narang
- Computational Health Informatics Program, Boston Children's Hospital, Boston, Massachusetts, USA
| | - Adam G Dunn
- Biomedical Informatics and Digital Health, Faculty of Medicine and Health, The University of Sydney, Sydney, New South Wales, Australia
- Computational Health Informatics Program, Boston Children's Hospital, Boston, Massachusetts, USA
| |
Collapse
|
11
|
Sarri G, Forsythe A, Elvidge J, Dawoud D. Living health technology assessments: how close to living reality? BMJ Evid Based Med 2023; 28:369-371. [PMID: 36797052 DOI: 10.1136/bmjebm-2022-112152] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 02/04/2023] [Indexed: 02/18/2023]
Affiliation(s)
- Grammati Sarri
- Real World and Advanced Analytics, Cytel Inc, London, UK
| | - Anna Forsythe
- Real World and Advanced Analytics, Cytel Inc, Miami, Florida, USA
| | - Jamie Elvidge
- Science, Evidence and Analytics Directorate, National Institute for Health and Care Excellence, London, UK
| | - Dalia Dawoud
- Science, Evidence and Analytics Directorate, National Institute for Health and Care Excellence, London, UK
| |
Collapse
|
12
|
Hocking L, Parkinson S, Adams A, Molding Nielsen E, Ang C, de Carvalho Gomes H. Overcoming the challenges of using automated technologies for public health evidence synthesis. Euro Surveill 2023; 28:2300183. [PMID: 37943502 PMCID: PMC10636742 DOI: 10.2807/1560-7917.es.2023.28.45.2300183] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2023] [Accepted: 08/10/2023] [Indexed: 11/10/2023] Open
Abstract
Many organisations struggle to keep pace with public health evidence due to the volume of published literature and length of time it takes to conduct literature reviews. New technologies that help automate parts of the evidence synthesis process can help conduct reviews more quickly and efficiently to better provide up-to-date evidence for public health decision making. To date, automated approaches have seldom been used in public health due to significant barriers to their adoption. In this Perspective, we reflect on the findings of a study exploring experiences of adopting automated technologies to conduct evidence reviews within the public health sector. The study, funded by the European Centre for Disease Prevention and Control, consisted of a literature review and qualitative data collection from public health organisations and researchers in the field. We specifically focus on outlining the challenges associated with the adoption of automated approaches and potential solutions and actions that can be taken to mitigate these. We explore these in relation to actions that can be taken by tool developers (e.g. improving tool performance and transparency), public health organisations (e.g. developing staff skills, encouraging collaboration) and funding bodies/the wider research system (e.g. researchers, funding bodies, academic publishers and scholarly journals).
Collapse
|
13
|
Sutton A, O'Keefe H, Johnson EE, Marshall C. A mapping exercise using automated techniques to develop a search strategy to identify systematic review tools. Res Synth Methods 2023; 14:874-881. [PMID: 37669905 DOI: 10.1002/jrsm.1665] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/10/2022] [Revised: 07/31/2023] [Accepted: 08/04/2023] [Indexed: 09/07/2023]
Abstract
The Systematic Review Toolbox aims provide a web-based catalogue of tools that support various tasks within the systematic review and wider evidence synthesis process. Identifying publications surrounding specific systematic review tools is currently challenging, leading to a high screening burden for few eligible records. We aimed to develop a search strategy that could be regularly and automatically run to identify eligible records for the SR Toolbox, thus reducing time on task and burden for those involved. We undertook a mapping exercise to identify the PubMed IDs of papers indexed within the SR Toolbox. We then used the Yale MeSH Analyser and Visualisation of Similarities (VOS) Viewer text-mining software to identify the most commonly used MeSH terms and text words within the eligible records. These MeSH terms and text words were combined using Boolean Operators into a search strategy for Ovid MEDLINE. Prior to the mapping exercise and search strategy development, 81 software tools and 55 'Other' tools were included within the SR Toolbox. Since implementation of the search strategy, 146 tools have been added. There has been an increase in tools added to the toolbox since the search was developed and its corresponding auto-alert in MEDLINE was originally set up. Developing a search strategy based on a mapping exercise is an effective way of identifying new tools to support the systematic review process. Further research could be conducted to help prioritise records for screening to reduce reviewer burden further and to adapt the strategy for disciplines beyond healthcare.
Collapse
Affiliation(s)
- Anthea Sutton
- Sheffield Centre for Health and Related Research, School of Medicine and Population Health, The University of Sheffield, Sheffield, UK
| | - Hannah O'Keefe
- NIHR Innovation Observatory, Newcastle University, Newcastle, UK
| | - Eugenie Evelynne Johnson
- NIHR Innovation Observatory, Newcastle University, Newcastle, UK
- Population Health Sciences Institute, Newcastle University, Newcastle, UK
| | | |
Collapse
|
14
|
Orel E, Ciglenecki I, Thiabaud A, Temerev A, Calmy A, Keiser O, Merzouki A. An Automated Literature Review Tool (LiteRev) for Streamlining and Accelerating Research Using Natural Language Processing and Machine Learning: Descriptive Performance Evaluation Study. J Med Internet Res 2023; 25:e39736. [PMID: 37713261 PMCID: PMC10541641 DOI: 10.2196/39736] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2022] [Revised: 01/08/2023] [Accepted: 06/26/2023] [Indexed: 09/16/2023] Open
Abstract
BACKGROUND Literature reviews (LRs) identify, evaluate, and synthesize relevant papers to a particular research question to advance understanding and support decision-making. However, LRs, especially traditional systematic reviews, are slow, resource-intensive, and become outdated quickly. OBJECTIVE LiteRev is an advanced and enhanced version of an existing automation tool designed to assist researchers in conducting LRs through the implementation of cutting-edge technologies such as natural language processing and machine learning techniques. In this paper, we present a comprehensive explanation of LiteRev's capabilities, its methodology, and an evaluation of its accuracy and efficiency to a manual LR, highlighting the benefits of using LiteRev. METHODS Based on the user's query, LiteRev performs an automated search on a wide range of open-access databases and retrieves relevant metadata on the resulting papers, including abstracts or full texts when available. These abstracts (or full texts) are text processed and represented as a term frequency-inverse document frequency matrix. Using dimensionality reduction (pairwise controlled manifold approximation) and clustering (hierarchical density-based spatial clustering of applications with noise) techniques, the corpus is divided into different topics described by a list of the most important keywords. The user can then select one or several topics of interest, enter additional keywords to refine its search, or provide key papers to the research question. Based on these inputs, LiteRev performs a k-nearest neighbor (k-NN) search and suggests a list of potentially interesting papers. By tagging the relevant ones, the user triggers new k-NN searches until no additional paper is suggested for screening. To assess the performance of LiteRev, we ran it in parallel to a manual LR on the burden and care for acute and early HIV infection in sub-Saharan Africa. We assessed the performance of LiteRev using true and false predictive values, recall, and work saved over sampling. RESULTS LiteRev extracted, processed, and transformed text into a term frequency-inverse document frequency matrix of 631 unique papers from PubMed. The topic modeling module identified 16 topics and highlighted 2 topics of interest to the research question. Based on 18 key papers, the k-NNs module suggested 193 papers for screening out of 613 papers in total (31.5% of the whole corpus) and correctly identified 64 relevant papers out of the 87 papers found by the manual abstract screening (recall rate of 73.6%). Compared to the manual full text screening, LiteRev identified 42 relevant papers out of the 48 papers found manually (recall rate of 87.5%). This represents a total work saved over sampling of 56%. CONCLUSIONS We presented the features and functionalities of LiteRev, an automation tool that uses natural language processing and machine learning methods to streamline and accelerate LRs and support researchers in getting quick and in-depth overviews on any topic of interest.
Collapse
Affiliation(s)
- Erol Orel
- Institute of Global Health, University of Geneva, Geneva, Switzerland
| | | | - Amaury Thiabaud
- Institute of Global Health, University of Geneva, Geneva, Switzerland
| | - Alexander Temerev
- Institute of Global Health, University of Geneva, Geneva, Switzerland
| | - Alexandra Calmy
- HIV/AIDS Unit, Division of Infectious Diseases, Geneva University Hospital, Geneva, Switzerland
| | - Olivia Keiser
- Institute of Global Health, University of Geneva, Geneva, Switzerland
| | - Aziza Merzouki
- Institute of Global Health, University of Geneva, Geneva, Switzerland
| |
Collapse
|
15
|
Oliveira Dos Santos Á, Sergio da Silva E, Machado Couto L, Valadares Labanca Reis G, Silva Belo V. The use of artificial intelligence for automating or semi-automating biomedical literature analyses: a scoping review. J Biomed Inform 2023; 142:104389. [PMID: 37187321 DOI: 10.1016/j.jbi.2023.104389] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2023] [Revised: 04/11/2023] [Accepted: 05/08/2023] [Indexed: 05/17/2023]
Abstract
OBJECTIVE Evidence-based medicine (EBM) is a decision-making process based on the conscious and judicious use of the best available scientific evidence. However, the exponential increase in the amount of information currently available likely exceeds the capacity of human-only analysis. In this context, artificial intelligence (AI) and its branches such as machine learning (ML) can be used to facilitate human efforts in analyzing the literature to foster EBM. The present scoping review aimed to examine the use of AI in the automation of biomedical literature survey and analysis with a view to establishing the state-of-the-art and identifying knowledge gaps. MATERIALS AND METHODS Comprehensive searches of the main databases were performed for articles published up to June 2022 and studies were selected according to inclusion and exclusion criteria. Data were extracted from the included articles and the findings categorized. RESULTS The total number of records retrieved from the databases was 12,145, of which 273 were included in the review. Classification of the studies according to the use of AI in evaluating the biomedical literature revealed three main application groups, namely assembly of scientific evidence (n=127; 47%), mining the biomedical literature (n=112; 41%) and quality analysis (n=34; 12%). Most studies addressed the preparation of systematic reviews, while articles focusing on the development of guidelines and evidence synthesis were the least frequent. The biggest knowledge gap was identified within the quality analysis group, particularly regarding methods and tools that assess the strength of recommendation and consistency of evidence. CONCLUSION Our review shows that, despite significant progress in the automation of biomedical literature surveys and analyses in recent years, intense research is needed to fill knowledge gaps on more difficult aspects of ML, deep learning and natural language processing, and to consolidate the use of automation by end-users (biomedical researchers and healthcare professionals).
Collapse
Affiliation(s)
| | - Eduardo Sergio da Silva
- Federal University of São João del-Rei, Campus Centro-Oeste Dona Lindu, Divinópolis, Minas Gerais, Brazil.
| | - Letícia Machado Couto
- Federal University of São João del-Rei, Campus Centro-Oeste Dona Lindu, Divinópolis, Minas Gerais, Brazil.
| | | | - Vinícius Silva Belo
- Federal University of São João del-Rei, Campus Centro-Oeste Dona Lindu, Divinópolis, Minas Gerais, Brazil.
| |
Collapse
|
16
|
Furuya-Kanamori L, Lin L, Kostoulas P, Clark J, Xu C. Limits in the search date for rapid reviews of diagnostic test accuracy studies. Res Synth Methods 2023; 14:173-179. [PMID: 36054082 PMCID: PMC9922791 DOI: 10.1002/jrsm.1598] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2022] [Revised: 07/12/2022] [Accepted: 08/01/2022] [Indexed: 11/11/2022]
Abstract
Limiting the search date is a common approach utilised in therapeutic/interventional rapid reviews. Yet the accuracy of pooled estimates is unknown when applied to rapid reviews of diagnostic test accuracy studies. Data from all systematic reviews of diagnostic test accuracy studies published in the Cochrane Database of Systematic Reviews, until February 2022 were collected. Meta-analyses with at least five studies were included to emulate rapid reviews by limiting the search to the recent 1, 2, 3, 4, 5, 10, 15, 20, 25, 30, 35 and 40 years. The magnitude of the pooled area under the curve (AUC), sensitivity and specificity for the full meta-analysis and the rapid reviews were compared. A total of 846 diagnostic meta-analyses were included. When the search date was limited to the recent 10 and 15 years, more than 75% and 80% of meta-analyses presented less than 5% difference between the pooled AUC, sensitivity and specificity of the full meta-analysis and the rapid review. There was little gain in the precision of the pooled estimates when the emulated rapid reviews included more than 15 years in the search. Rapid reviews restricted by search date are a valid and reliable approach for diagnostic test accuracy studies. Robust evidence can be achieved by restricting the search date to the recent 10-15 years. Future studies need to examine the reduction in workload and time to finish the rapid reviews under different search date limits.
Collapse
Affiliation(s)
- Luis Furuya-Kanamori
- UQ Centre for Clinical Research, Faculty of Medicine, University of Queensland, Herston, Australia
| | - Lifeng Lin
- Department of Epidemiology and Biostatistics, University of Arizona, Tucson, Arizona, USA
- Department of Statistics, Florida State University, Tallahassee, Florida, USA
| | - Polychronis Kostoulas
- Faculty of Public and One Health, School of Health Sciences, University of Thessaly, Karditsa, Greece
| | - Justin Clark
- Institute for Evidence-Based Healthcare, Bond University, Robina, Australia
| | - Chang Xu
- Ministry of Education Key Laboratory for Population Health Across-Life Cycle, Anhui Medical University, Anhui, China
| |
Collapse
|
17
|
Burgard T, Bittermann A. Reducing Literature Screening Workload With Machine Learning. ZEITSCHRIFT FUR PSYCHOLOGIE-JOURNAL OF PSYCHOLOGY 2023. [DOI: 10.1027/2151-2604/a000509] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/24/2023]
Abstract
Abstract. In our era of accelerated accumulation of knowledge, the manual screening of literature for eligibility is increasingly becoming too labor-intensive for summarizing the current state of knowledge in a timely manner. Recent advances in machine learning and natural language processing promise to reduce the screening workload by automatically detecting unseen references with a high probability of inclusion. As a variety of tools have been developed, the current review provides an overview of their characteristics and performance. A systematic search in various databases yielded 488 eligible reports, revealing 15 tools for screening automation that differed in methodology, features, and accessibility. For the review on the performance of screening tools, 21 studies could be included. In comparison to sampling records randomly, active screening with prioritization approximately halves the screening workload. However, a comparison of tools under equal or at least similar conditions is needed to derive clear recommendations.
Collapse
Affiliation(s)
- Tanja Burgard
- Research Synthesis Methods, Leibniz Institute for Psychology (ZPID), Trier, Germany
| | - André Bittermann
- Big Data, Leibniz Institute for Psychology (ZPID), Trier, Germany
| |
Collapse
|
18
|
Unsupervised title and abstract screening for systematic review: a retrospective case-study using topic modelling methodology. Syst Rev 2023; 12:1. [PMID: 36597132 PMCID: PMC9811792 DOI: 10.1186/s13643-022-02163-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/27/2022] [Accepted: 12/21/2022] [Indexed: 01/04/2023] Open
Abstract
BACKGROUND The importance of systematic reviews in collating and summarising available research output on a particular topic cannot be over-emphasized. However, initial screening of retrieved literature is significantly time and labour intensive. Attempts at automating parts of the systematic review process have been made with varying degree of success partly due to being domain-specific, requiring vendor-specific software or manually labelled training data. Our primary objective was to develop statistical methodology for performing automated title and abstract screening for systematic reviews. Secondary objectives included (1) to retrospectively apply the automated screening methodology to previously manually screened systematic reviews and (2) to characterize the performance of the automated screening methodology scoring algorithm in a simulation study. METHODS We implemented a Latent Dirichlet Allocation-based topic model to derive representative topics from the retrieved documents' title and abstract. The second step involves defining a score threshold for classifying the documents as relevant for full-text review or not. The score is derived based on a set of search keywords (often the database retrieval search terms). Two systematic review studies were retrospectively used to illustrate the methodology. RESULTS In one case study (helminth dataset), [Formula: see text] sensitivity compared to manual title and abstract screening was achieved. This is against a false positive rate of [Formula: see text]. For the second case study (Wilson disease dataset), a sensitivity of [Formula: see text] and specificity of [Formula: see text] were achieved. CONCLUSIONS Unsupervised title and abstract screening has the potential to reduce the workload involved in conducting systematic review. While sensitivity of the methodology on the tested data is low, approximately [Formula: see text] specificity was achieved. Users ought to keep in mind that potentially low sensitivity might occur. One approach to mitigate this might be to incorporate additional targeted search keywords such as the indexing databases terms into the search term copora. Moreover, automated screening can be used as an additional screener to the manual screeners.
Collapse
|
19
|
Tran Mau-Them F, Overs A, Bruel AL, Duquet R, Thareau M, Denommé-Pichon AS, Vitobello A, Sorlin A, Safraou H, Nambot S, Delanne J, Moutton S, Racine C, Engel C, De Giraud d’Agay M, Lehalle D, Goldenberg A, Willems M, Coubes C, Genevieve D, Verloes A, Capri Y, Perrin L, Jacquemont ML, Lambert L, Lacaze E, Thevenon J, Hana N, Van-Gils J, Dubucs C, Bizaoui V, Gerard-Blanluet M, Lespinasse J, Mercier S, Guerrot AM, Maystadt I, Tisserant E, Faivre L, Philippe C, Duffourd Y, Thauvin-Robinet C. Combining globally search for a regular expression and print matching lines with bibliographic monitoring of genomic database improves diagnosis. Front Genet 2023; 14:1122985. [PMID: 37152996 PMCID: PMC10157399 DOI: 10.3389/fgene.2023.1122985] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2022] [Accepted: 02/13/2023] [Indexed: 05/09/2023] Open
Abstract
Introduction: Exome sequencing has a diagnostic yield ranging from 25% to 70% in rare diseases and regularly implicates genes in novel disorders. Retrospective data reanalysis has demonstrated strong efficacy in improving diagnosis, but poses organizational difficulties for clinical laboratories. Patients and methods: We applied a reanalysis strategy based on intensive prospective bibliographic monitoring along with direct application of the GREP command-line tool (to "globally search for a regular expression and print matching lines") in a large ES database. For 18 months, we submitted the same five keywords of interest [(intellectual disability, (neuro)developmental delay, and (neuro)developmental disorder)] to PubMed on a daily basis to identify recently published novel disease-gene associations or new phenotypes in genes already implicated in human pathology. We used the Linux GREP tool and an in-house script to collect all variants of these genes from our 5,459 exome database. Results: After GREP queries and variant filtration, we identified 128 genes of interest and collected 56 candidate variants from 53 individuals. We confirmed causal diagnosis for 19/128 genes (15%) in 21 individuals and identified variants of unknown significance for 19/128 genes (15%) in 23 individuals. Altogether, GREP queries for only 128 genes over a period of 18 months permitted a causal diagnosis to be established in 21/2875 undiagnosed affected probands (0.7%). Conclusion: The GREP query strategy is efficient and less tedious than complete periodic reanalysis. It is an interesting reanalysis strategy to improve diagnosis.
Collapse
Affiliation(s)
- Frédéric Tran Mau-Them
- Unité Fonctionnelle Innovation en Diagnostic Génomique des maladies rares, CHU Dijon, Dijon, France
- INSERM UMR1231 GAD, Dijon, France
- *Correspondence: Frédéric Tran Mau-Them,
| | - Alexis Overs
- Unité Fonctionnelle Innovation en Diagnostic Génomique des maladies rares, CHU Dijon, Dijon, France
| | - Ange-Line Bruel
- Unité Fonctionnelle Innovation en Diagnostic Génomique des maladies rares, CHU Dijon, Dijon, France
- INSERM UMR1231 GAD, Dijon, France
| | - Romain Duquet
- Unité Fonctionnelle Innovation en Diagnostic Génomique des maladies rares, CHU Dijon, Dijon, France
| | - Mylene Thareau
- Unité Fonctionnelle Innovation en Diagnostic Génomique des maladies rares, CHU Dijon, Dijon, France
| | - Anne-Sophie Denommé-Pichon
- Unité Fonctionnelle Innovation en Diagnostic Génomique des maladies rares, CHU Dijon, Dijon, France
- INSERM UMR1231 GAD, Dijon, France
| | - Antonio Vitobello
- Unité Fonctionnelle Innovation en Diagnostic Génomique des maladies rares, CHU Dijon, Dijon, France
- INSERM UMR1231 GAD, Dijon, France
| | - Arthur Sorlin
- Unité Fonctionnelle Innovation en Diagnostic Génomique des maladies rares, CHU Dijon, Dijon, France
- INSERM UMR1231 GAD, Dijon, France
| | - Hana Safraou
- Unité Fonctionnelle Innovation en Diagnostic Génomique des maladies rares, CHU Dijon, Dijon, France
- INSERM UMR1231 GAD, Dijon, France
| | - Sophie Nambot
- Centre de Référence Maladies Rares “Anomalies du développement et syndromes malformatifs”, Centre de Génétique, FHUTRANSLAD et Institut GIMI, CHU Dijon Bourgogne, Dijon, France
| | - Julian Delanne
- Centre de Référence Maladies Rares “Anomalies du développement et syndromes malformatifs”, Centre de Génétique, FHUTRANSLAD et Institut GIMI, CHU Dijon Bourgogne, Dijon, France
| | - Sebastien Moutton
- Centre de Référence Maladies Rares “Anomalies du développement et syndromes malformatifs”, Centre de Génétique, FHUTRANSLAD et Institut GIMI, CHU Dijon Bourgogne, Dijon, France
| | - Caroline Racine
- Centre de Référence Maladies Rares “Anomalies du développement et syndromes malformatifs”, Centre de Génétique, FHUTRANSLAD et Institut GIMI, CHU Dijon Bourgogne, Dijon, France
| | - Camille Engel
- Unité Fonctionnelle Innovation en Diagnostic Génomique des maladies rares, CHU Dijon, Dijon, France
| | | | - Daphne Lehalle
- Centre de Référence Maladies Rares “Anomalies du développement et syndromes malformatifs”, Centre de Génétique, FHUTRANSLAD et Institut GIMI, CHU Dijon Bourgogne, Dijon, France
| | - Alice Goldenberg
- Normandie Univ, UNIROUEN, Inserm U1245 and Rouen University Hospital, Rouen, France
- Department of Genetics and Reference Center for Developmental Disorders, Normandy Center for Genomic and Personalized Medicine, Rouen, France
| | - Marjolaine Willems
- Département de Génétique Médicale Maladies Rares et Médecine Personnalisée, Centre de Référence Maladies Rares Anomalies du Développement, Hôpital Arnaud de Villeneuve, Université Montpellier, Montpellier, France
| | - Christine Coubes
- Département de Génétique Médicale Maladies Rares et Médecine Personnalisée, Centre de Référence Maladies Rares Anomalies du Développement, Hôpital Arnaud de Villeneuve, Université Montpellier, Montpellier, France
| | - David Genevieve
- Département de Génétique Médicale Maladies Rares et Médecine Personnalisée, Centre de Référence Maladies Rares Anomalies du Développement, Hôpital Arnaud de Villeneuve, Université Montpellier, Montpellier, France
| | - Alain Verloes
- Centre de Référence Anomalies du Développement et Syndromes Malformatifs, Department of Medical Genetics, AP-HPNord- Université de Paris, Hôpital Robert Debré, Paris, France
- INSERM UMR 1141, Paris, France
| | - Yline Capri
- Service de Génétique Clinique, CHU Robert Debré, Paris, France
| | - Laurence Perrin
- Service de Génétique Clinique, CHU Robert Debré, Paris, France
| | - Marie-Line Jacquemont
- Unité de Génétique Médicale, Pole Femme-Mère-Enfant, Groupe Hospitalier Sud Réunion, CHU de La Réunion, La Réunion, France
| | | | - Elodie Lacaze
- Unité de Génétique Médicale, Groupe Hospitalier du Havre, Le Havre, France
| | - Julien Thevenon
- Centre de Référence Maladies Rares “Anomalies du développement et syndromes malformatifs”, Centre de Génétique, FHUTRANSLAD et Institut GIMI, CHU Dijon Bourgogne, Dijon, France
| | - Nadine Hana
- Département de Génétique, Assistance Publique-Hôpitaux de Paris, Hôpital Bichat, Paris, France
- INSERM U1148, Laboratory for Vascular Translational Science, Université Paris de Paris, Hôpital Bichat, Paris, France
| | - Julien Van-Gils
- Service de Génétique Médicale, CHU de Bordeaux, Bordeaux, France
| | - Charlotte Dubucs
- Department of Medical Genetics, Toulouse University Hospital, Toulouse, France
| | - Varoona Bizaoui
- Service de Génétique, Centre Hospitalier Universitaire Caen Normandie, Caen, France
| | | | | | - Sandra Mercier
- Service de Génétique Médicale, CHU Nantes, Nantes, France
| | - Anne-Marie Guerrot
- Department of Genetics and Reference Center for Developmental Disorders, Normandie Univ, UNIROUEN, CHU Rouen, Rouen, France
- Inserm U1245, FHU G4 Génomique, Rouen, France
| | - Isabelle Maystadt
- Centre de Génétique Humaine, Institut de Pathologie et de Génétique, Gosselies, Belgium
| | - Emilie Tisserant
- Unité Fonctionnelle Innovation en Diagnostic Génomique des maladies rares, CHU Dijon, Dijon, France
| | - Laurence Faivre
- INSERM UMR1231 GAD, Dijon, France
- Centre de Référence Maladies Rares “Anomalies du développement et syndromes malformatifs”, Centre de Génétique, FHUTRANSLAD et Institut GIMI, CHU Dijon Bourgogne, Dijon, France
| | - Christophe Philippe
- Unité Fonctionnelle Innovation en Diagnostic Génomique des maladies rares, CHU Dijon, Dijon, France
- INSERM UMR1231 GAD, Dijon, France
| | - Yannis Duffourd
- Unité Fonctionnelle Innovation en Diagnostic Génomique des maladies rares, CHU Dijon, Dijon, France
- INSERM UMR1231 GAD, Dijon, France
| | - Christel Thauvin-Robinet
- Unité Fonctionnelle Innovation en Diagnostic Génomique des maladies rares, CHU Dijon, Dijon, France
- INSERM UMR1231 GAD, Dijon, France
- Centre de Référence Maladies Rares “Anomalies du développement et syndromes malformatifs”, Centre de Génétique, FHUTRANSLAD et Institut GIMI, CHU Dijon Bourgogne, Dijon, France
| |
Collapse
|
20
|
Cierco Jimenez R, Lee T, Rosillo N, Cordova R, Cree IA, Gonzalez A, Indave Ruiz BI. Machine learning computational tools to assist the performance of systematic reviews: A mapping review. BMC Med Res Methodol 2022; 22:322. [PMID: 36522637 PMCID: PMC9756658 DOI: 10.1186/s12874-022-01805-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 11/26/2022] [Indexed: 12/23/2022] Open
Abstract
BACKGROUND Within evidence-based practice (EBP), systematic reviews (SR) are considered the highest level of evidence in that they summarize the best available research and describe the progress in a determined field. Due its methodology, SR require significant time and resources to be performed; they also require repetitive steps that may introduce biases and human errors. Machine learning (ML) algorithms therefore present a promising alternative and a potential game changer to speed up and automate the SR process. This review aims to map the current availability of computational tools that use ML techniques to assist in the performance of SR, and to support authors in the selection of the right software for the performance of evidence synthesis. METHODS The mapping review was based on comprehensive searches in electronic databases and software repositories to obtain relevant literature and records, followed by screening for eligibility based on titles, abstracts, and full text by two reviewers. The data extraction consisted of listing and extracting the name and basic characteristics of the included tools, for example a tool's applicability to the various SR stages, pricing options, open-source availability, and type of software. These tools were classified and graphically represented to facilitate the description of our findings. RESULTS A total of 9653 studies and 585 records were obtained from the structured searches performed on selected bibliometric databases and software repositories respectively. After screening, a total of 119 descriptions from publications and records allowed us to identify 63 tools that assist the SR process using ML techniques. CONCLUSIONS This review provides a high-quality map of currently available ML software to assist the performance of SR. ML algorithms are arguably one of the best techniques at present for the automation of SR. The most promising tools were easily accessible and included a high number of user-friendly features permitting the automation of SR and other kinds of evidence synthesis reviews.
Collapse
Affiliation(s)
- Ramon Cierco Jimenez
- International Agency for Research on Cancer (IARC/WHO), Evidence Synthesis and Classification Branch, Lyon, France.
- Laboratori de Medicina Computacional, Unitat de Bioestadística, Facultat de Medicina, Universitat Autònoma de Barcelona, Bellaterra, Spain.
| | - Teresa Lee
- International Agency for Research on Cancer (IARC/WHO), Services to Science and Research Branch, Lyon, France
| | - Nicolás Rosillo
- Servicio de Medicina Preventiva, Hospital Universitario 12 de Octubre, Madrid, Spain
| | - Reynalda Cordova
- International Agency for Research on Cancer (IARC/WHO), Nutrition and Metabolism Branch, Lyon, France
- Department of Nutritional Sciences, University of Vienna, Vienna, Austria
| | - Ian A Cree
- International Agency for Research on Cancer (IARC/WHO), Evidence Synthesis and Classification Branch, Lyon, France
| | - Angel Gonzalez
- Laboratori de Medicina Computacional, Unitat de Bioestadística, Facultat de Medicina, Universitat Autònoma de Barcelona, Bellaterra, Spain
| | - Blanca Iciar Indave Ruiz
- International Agency for Research on Cancer (IARC/WHO), Evidence Synthesis and Classification Branch, Lyon, France
| |
Collapse
|
21
|
Li X, Zhang A, Al-Zaidy R, Rao A, Baral S, Bao L, Giles CL. Automating document classification with distant supervision to increase the efficiency of systematic reviews: A case study on identifying studies with HIV impacts on female sex workers. PLoS One 2022; 17:e0270034. [PMID: 35771807 PMCID: PMC9246134 DOI: 10.1371/journal.pone.0270034] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/28/2021] [Accepted: 06/02/2022] [Indexed: 12/03/2022] Open
Abstract
There remains a limited understanding of the HIV prevention and treatment needs among female sex workers in many parts of the world. Systematic reviews of existing literature can help fill this gap; however, well-done systematic reviews are time-demanding and labor-intensive. Here, we propose an automatic document classification approach to a systematic review to significantly reduce the effort in reviewing documents and optimizing empiric decision making. We first describe a manual document classification procedure that is used to curate a pertinent training dataset and then propose three classifiers: a keyword-guided method, a cluster analysis-based method, and a random forest approach that utilizes a large set of feature tokens. This approach is used to identify documents studying female sex workers that contain content relevant to either HIV or experienced violence. We compare the performance of the three classifiers by cross-validation in terms of area under the curve of the receiver operating characteristic and precision and recall plot, and found random forest approach reduces the amount of manual reading for our example by 80%; in sensitivity analysis, we found that even trained with only 10% of data, the classifier can still avoid reading 75% of future documents (68% of total) while retaining 80% of relevant documents. In sum, the automated procedure of document classification presented here could improve both the precision and efficiency of systematic reviews and facilitate live reviews, where reviews are updated regularly. We expect to obtain a reasonable classifier by taking 20% of retrieved documents as training samples. The proposed classifier could also be used for more meaningfully assembling literature in other research areas and for rapid documents screening with a tight schedule, such as COVID-related work during the crisis.
Collapse
Affiliation(s)
- Xiaoxiao Li
- Department of Statistics, Pennsylvania State University, University Park, PA, United States of America
| | - Amy Zhang
- Department of Statistics, Pennsylvania State University, University Park, PA, United States of America
| | - Rabah Al-Zaidy
- Information and Computer Science Department, King Fahad University of Petroleum and Minerals, Dhahran, Saudi Arabia
| | - Amrita Rao
- Department of Epidemiology, Johns Hopkins University, Baltimore, MD, United States of America
| | - Stefan Baral
- Department of Epidemiology, Johns Hopkins University, Baltimore, MD, United States of America
| | - Le Bao
- Department of Statistics, Pennsylvania State University, University Park, PA, United States of America
| | - C. Lee Giles
- College of Information Sciences and Technology, Pennsylvania State University, University Park, PA, United States of America
| |
Collapse
|
22
|
Rehfuess EA, Burns JB, Pfadenhauer LM, Krishnaratne S, Littlecott H, Meerpohl JJ, Movsisyan A. Lessons learnt: Undertaking rapid reviews on public health and social measures during a global pandemic. Res Synth Methods 2022; 13:558-572. [PMID: 35704478 PMCID: PMC9349463 DOI: 10.1002/jrsm.1580] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/22/2021] [Revised: 06/01/2022] [Accepted: 06/10/2022] [Indexed: 12/04/2022]
Abstract
Public health and social measures (PHSM) have been central to the COVID‐19 response. Consequently, there has been much pressure on decision‐makers to make evidence‐informed decisions and on researchers to synthesize the evidence regarding these measures. This article describes our experiences, responses and lessons learnt regarding key challenges when planning and conducting rapid reviews of PHSM during the COVID‐19 pandemic. Stakeholder consultations and scoping reviews to obtain an overview of the evidence inform the scope of reviews that are policy‐relevant and feasible. Multiple complementary reviews serve to examine the benefits and harms of PHSM across different populations and contexts. Conceiving reviews of effectiveness as adaptable living reviews helps to respond to evolving evidence needs and an expanding evidence base. An appropriately skilled review team and good planning, coordination and communication ensures smooth and rigorous processes and efficient use of resources. Scientific rigor, the practical implications of PHSM‐related complexity and likely time savings should be carefully weighed in deciding on methodological shortcuts. Making the best possible use of modeling studies represents a particular challenge, and methods should be carefully chosen, piloted and implemented. Our experience raises questions regarding the nature of rapid reviews and regarding how different types of evidence should be considered in making decisions about PHSM during a global pandemic. We highlight the need for readily available protocols for conducting studies on the effectiveness, unintended consequences and implementation of PHSM in a timely manner, as well as the need for rapid review standards tailored to “rapid” versus “emergency” mode reviewing.
Collapse
Affiliation(s)
- E A Rehfuess
- Institute for Medical Information Processing, Biometry and Epidemiology, Ludwig-Maximilians-Universität, Munich, Germany.,Pettenkofer School of Public Health, Munich, Germany
| | - J B Burns
- Institute for Medical Information Processing, Biometry and Epidemiology, Ludwig-Maximilians-Universität, Munich, Germany.,Pettenkofer School of Public Health, Munich, Germany
| | - L M Pfadenhauer
- Institute for Medical Information Processing, Biometry and Epidemiology, Ludwig-Maximilians-Universität, Munich, Germany.,Pettenkofer School of Public Health, Munich, Germany
| | - S Krishnaratne
- Institute for Medical Information Processing, Biometry and Epidemiology, Ludwig-Maximilians-Universität, Munich, Germany.,Pettenkofer School of Public Health, Munich, Germany.,Department of Disease Control, Faculty of Infectious and Tropical Diseases, London School of Hygiene and Tropical Medicine, London, United Kingdom
| | - H Littlecott
- Institute for Medical Information Processing, Biometry and Epidemiology, Ludwig-Maximilians-Universität, Munich, Germany.,Pettenkofer School of Public Health, Munich, Germany.,DECIPHer (Centre for Development, Evaluation, Complexity and Implementation in Public Health Improvement), School of Social Sciences, Cardiff University, Cardiff, United Kingdom
| | - J J Meerpohl
- Institute for Evidence in Medicine, Medical Center & Faculty of Medicine, University of Freiburg, Freiburg, Germany.,Cochrane Germany, Cochrane Germany Foundation, Freiburg, Germany
| | - A Movsisyan
- Institute for Medical Information Processing, Biometry and Epidemiology, Ludwig-Maximilians-Universität, Munich, Germany.,Pettenkofer School of Public Health, Munich, Germany
| |
Collapse
|
23
|
Sanchez-Graillet O, Witte C, Grimm F, Grautoff S, Ell B, Cimiano P. Synthesizing evidence from clinical trials with dynamic interactive argument trees. J Biomed Semantics 2022; 13:16. [PMID: 35659056 PMCID: PMC9166347 DOI: 10.1186/s13326-022-00270-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/29/2020] [Accepted: 04/12/2022] [Indexed: 11/24/2022] Open
Abstract
Background Evidence-based medicine propagates that medical/clinical decisions are made by taking into account high-quality evidence, most notably in the form of randomized clinical trials. Evidence-based decision-making requires aggregating the evidence available in multiple trials to reach –by means of systematic reviews– a conclusive recommendation on which treatment is best suited for a given patient population. However, it is challenging to produce systematic reviews to keep up with the ever-growing number of published clinical trials. Therefore, new computational approaches are necessary to support the creation of systematic reviews that include the most up-to-date evidence.We propose a method to synthesize the evidence available in clinical trials in an ad-hoc and on-demand manner by automatically arranging such evidence in the form of a hierarchical argument that recommends a therapy as being superior to some other therapy along a number of key dimensions corresponding to the clinical endpoints of interest. The method has also been implemented as a web tool that allows users to explore the effects of excluding different points of evidence, and indicating relative preferences on the endpoints. Results Through two use cases, our method was shown to be able to generate conclusions similar to the ones of published systematic reviews. To evaluate our method implemented as a web tool, we carried out a survey and usability analysis with medical professionals. The results show that the tool was perceived as being valuable, acknowledging its potential to inform clinical decision-making and to complement the information from existing medical guidelines. Conclusions The method presented is a simple but yet effective argumentation-based method that contributes to support the synthesis of clinical trial evidence. A current limitation of the method is that it relies on a manually populated knowledge base. This problem could be alleviated by deploying natural language processing methods to extract the relevant information from publications.
Collapse
Affiliation(s)
- Olivia Sanchez-Graillet
- Semantic Computing Group, Cluster of Excellence Cognitive Interaction Technology (CITEC), Bielefeld University, Bielefeld, 33619, Germany.
| | - Christian Witte
- Semantic Computing Group, Cluster of Excellence Cognitive Interaction Technology (CITEC), Bielefeld University, Bielefeld, 33619, Germany
| | - Frank Grimm
- Semantic Computing Group, Cluster of Excellence Cognitive Interaction Technology (CITEC), Bielefeld University, Bielefeld, 33619, Germany
| | | | - Basil Ell
- Semantic Computing Group, Cluster of Excellence Cognitive Interaction Technology (CITEC), Bielefeld University, Bielefeld, 33619, Germany.,SIRIUS labs, Oslo University, Oslo, Norway
| | - Philipp Cimiano
- Semantic Computing Group, Cluster of Excellence Cognitive Interaction Technology (CITEC), Bielefeld University, Bielefeld, 33619, Germany
| |
Collapse
|
24
|
Chaboyer W, Coyer F, Harbeck E, Thalib L, Latimer S, Wan CS, Tobiano G, Griffin BR, Campbell JL, Walker R, Carlini JJ, Lockwood I, Clark J, Gillespie BM. Oedema as a predictor of the incidence of new pressure injuries in adults in any care setting: A systematic review and meta-analysis. Int J Nurs Stud 2022; 128:104189. [PMID: 35217433 DOI: 10.1016/j.ijnurstu.2022.104189] [Citation(s) in RCA: 14] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/27/2021] [Revised: 01/27/2022] [Accepted: 01/29/2022] [Indexed: 02/07/2023]
Abstract
BACKGROUND Oedema measurement, also termed sub-epidermal moisture measurement is recommended as an adjunct pressure injury prevention intervention in international guidelines because it indicates early tissue damage. OBJECTIVE To determine the prognostic value of oedema measurement in predicting future pressure injury in adults in any care setting. DESIGN Systematic review and meta-analysis. SETTING Participants were recruited from nursing homes or aged care facilities, hospitals, or post-acute facilities. PARTICIPANTS Adults. METHODS A modified 2-week systematic review was undertaken. Study designs included cohort (prospective and retrospective), case-control, case series if relevant comparisons were reported, randomised controlled trials if the association between oedema measurement and pressure injury was reported, and registry data. Databases searched included: Medical Literature Analysis and Retrieval System Online, The Cumulative Index to Nursing and Allied Health Literature, Excerpta Medica and the Cochrane Library from inception to 13 July 2021 with no language restrictions. Screening, data extraction using Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies - Prognostic Factors (CHARMS-PF) and quality assessment using Quality in Prognostic Factor Studies (QUIPS) were undertaken independently by ≥2 authors and adjudicated by another if required. Meta-analyses and meta-regression were undertaken. The certainty of the evidence was assessed using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach. RESULTS Six studies (n = 483 total) were included. Two studies were set in nursing homes and four in either hospitals or post-acute facilities. Fives studies were prospective cohorts, and one was a randomised control trial. Two studies were assessed as low risk and four studies as moderate risk of bias. The pooled risk ratio in four studies (n = 388) for the relationship between oedema and pressure injury cumulative incidence was 18.87 (95% CI 2.13-38.29) and for time to pressure injury was 4.08 days (95% CI 1.64-6.52). Using GRADE, the certainty of the body of evidence was low for all outcomes. Meta-regression indicated that age, gender, and sample size were poor predictors for the association between oedema and pressure injury. CONCLUSIONS Measuring oedema as a predictor for pressure injury development is showing promise but a stronger body of evidence that takes into consideration other prognostic factors is needed to better understand its benefit. REGISTRATION PROSPERO CRD42021267834. TWEETABLE ABSTRACT Measuring oedema is a promising strategy to prevent pressure injuries but the certainty of evidence for this claim is low.
Collapse
Affiliation(s)
- Wendy Chaboyer
- Menzies Health Institute Queensland, Griffith University Gold Coast Campus, Griffith University, Building G01, Gold Coast, Queensland 4222, Australia.
| | - Fiona Coyer
- School of Nursing, Centre for Healthcare Transformation, Queensland University of Technology, Brisbane, Australia; Intensive Care Services, Royal Brisbane and Women's Hospital, Brisbane, Australia
| | - Emma Harbeck
- Menzies Health Institute Queensland, Griffith University Gold Coast Campus, Griffith University, Building G01, Gold Coast, Queensland 4222, Australia
| | - Lukman Thalib
- Department of Biostatistics Faculty of Medicine, Istanbul Aydin University, Istanbul, Turkey
| | - Sharon Latimer
- Menzies Health Institute Queensland, Griffith University Gold Coast Campus, Griffith University, Building G01, Gold Coast, Queensland 4222, Australia
| | - Ching Shan Wan
- Nursing Research Institute, St Vincent's Health Network Sydney, Australia; St Vincent's Hospital Melbourne, Australia; Australian Catholic University, Melbourne, Australia.
| | - Georgia Tobiano
- Menzies Health Institute Queensland, Griffith University Gold Coast Campus, Griffith University, Building G01, Gold Coast, Queensland 4222, Australia; Gold Coast University Hospital, Gold Coast, Australia.
| | - Bronwyn R Griffin
- Menzies Health Institute Queensland, Griffith University Gold Coast Campus, Griffith University, Building G01, Gold Coast, Queensland 4222, Australia.
| | - Jill L Campbell
- Menzies Health Institute Queensland, Griffith University Gold Coast Campus, Griffith University, Building G01, Gold Coast, Queensland 4222, Australia.
| | - Rachel Walker
- Menzies Health Institute Queensland, Griffith University Gold Coast Campus, Griffith University, Building G01, Gold Coast, Queensland 4222, Australia; The Princess Alexandra Hospital, Brisbane, Australia.
| | - Joan J Carlini
- Menzies Health Institute Queensland, Griffith University Gold Coast Campus, Griffith University, Building G01, Gold Coast, Queensland 4222, Australia; Griffith Business School, Griffith University, Gold Coast, Australia.
| | - Ishtar Lockwood
- Menzies Health Institute Queensland, Griffith University Gold Coast Campus, Griffith University, Building G01, Gold Coast, Queensland 4222, Australia
| | - Justin Clark
- Institute for Evidence-Based Healthcare, Bond University, Gold Coast, Australia
| | - Brigid M Gillespie
- Menzies Health Institute Queensland, Griffith University Gold Coast Campus, Griffith University, Building G01, Gold Coast, Queensland 4222, Australia; Gold Coast University Hospital, Gold Coast, Australia.
| |
Collapse
|
25
|
Pandey NN, Sharma S. Conducting and critically appraising a high-quality systematic review and Meta-analysis pertaining to COVID-19. Curr Med Res Opin 2022; 38:317-325. [PMID: 34870545 DOI: 10.1080/03007995.2021.2015160] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
Abstract
With constantly emerging new information regarding the epidemiology, pathogenesis, diagnosis and management of Coronavirus Disease 2019 (COVID-19), reviewing literature related to it has become increasingly complicated and resource-intensive. In the setting of this global pandemic, clinical decisions are being guided by the results of multiple pertinent studies; however, it has been observed that these studies are often heterogenous in design and population characteristics and results of initial trials may not be replicated in subsequent studies. The resulting clinical conundrum can be resolved by high-quality systematic review and meta-analysis with a robust and reliable methodology, encapsulating and critically appraising all the available literature relevant to the clinical scenario under scrutiny. It can condense the large volume of scientific information available and can also identify the cause of differences in the degree of effect under consideration across different studies. It can identify optimal diagnostic algorithms, assess efficacy of treatment strategies, and analyze inherent factors influencing the efficacy of treatment for COVID-19. The current review aims to provide a basic guide to plan and conduct a high-quality systematic review and meta-analysis pertaining to COVID-19, describing the main steps and addressing the pitfalls commonly encountered at each step. Knowledge of the basic steps would also allow the reader to critically appraise published systematic review and meta-analysis and the quality of evidence provided therein.
Collapse
Affiliation(s)
- Niraj Nirmal Pandey
- Department of Cardiovascular Radiology and Endovascular Interventions, All India Institute of Medical Sciences, New Delhi, India
| | - Sanjiv Sharma
- Department of Cardiovascular Radiology and Endovascular Interventions, All India Institute of Medical Sciences, New Delhi, India
| |
Collapse
|
26
|
Search algorithms and artificial intelligence, an essential aid in the development of systematized reviews. NUTR HOSP 2022; 39:1434-1435. [DOI: 10.20960/nh.04336] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022] Open
|
27
|
Jakhar D, Kaul S, Sinha S. Comparison of artificial intelligence with a conventional search in dermatology: A case study of systematic review of apremilast in hidradenitis suppurativa performed by both methods. Indian Dermatol Online J 2022; 13:277-279. [PMID: 35287422 PMCID: PMC8917485 DOI: 10.4103/idoj.idoj_264_21] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/17/2021] [Revised: 06/13/2021] [Accepted: 06/16/2021] [Indexed: 11/23/2022] Open
|
28
|
Halamoda‐Kenzaoui B, Rolland E, Piovesan J, Puertas Gallardo A, Bremer‐Hoffmann S. Toxic effects of nanomaterials for health applications: How automation can support a systematic review of the literature? J Appl Toxicol 2022; 42:41-51. [PMID: 34050552 PMCID: PMC9292569 DOI: 10.1002/jat.4204] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2021] [Revised: 04/30/2021] [Accepted: 05/14/2021] [Indexed: 11/24/2022]
Abstract
Systematic reviews of the scientific literature can be an important source of information supporting the daily work of the regulators in their decision making, particularly in areas of innovative technologies where the regulatory experience is still limited. Significant research activities in the field of nanotechnology resulted in a huge number of publications in the last decades. However, even if the published data can provide relevant information, scientific articles are often of diverse quality, and it is nearly impossible to manually process and evaluate such amount of data in a systematic manner. In this feasibility study, we investigated to what extent open-access automation tools can support a systematic review of toxic effects of nanomaterials for health applications reported in the scientific literature. In this study, we used a battery of available tools to perform the initial steps of a systematic review such as targeted searches, data curation and abstract screening. This work was complemented with an in-house developed tool that allowed us to extract specific sections of the articles such as the materials and methods part or the results section where we could perform subsequent text analysis. We ranked the articles according to quality criteria based on the reported nanomaterial characterisation and extracted most frequently described toxic effects induced by different types of nanomaterials. Even if further demonstration of the reliability and applicability of automation tools is necessary, this study demonstrated the potential to leverage information from the scientific literature by using automation systems in a tiered strategy.
Collapse
|
29
|
Khalil H, Ameen D, Zarnegar A. Tools to support the automation of systematic reviews: A scoping review. J Clin Epidemiol 2021; 144:22-42. [PMID: 34896236 DOI: 10.1016/j.jclinepi.2021.12.005] [Citation(s) in RCA: 30] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2021] [Revised: 11/09/2021] [Accepted: 12/02/2021] [Indexed: 11/19/2022]
Abstract
OBJECTIVE The objectives of this scoping review are to identify the reliability and validity of the available tools, their limitations and any recommendations to further improve the use of these tools. STUDY DESIGN A scoping review methodology was followed to map the literature published on the challenges and solutions of conducting evidence synthesis using the JBI scoping review methodology. RESULTS A total of 47 publications were included in the review. The current scoping review identified that LitSuggest, Rayyan, Abstractr, BIBOT, R software, RobotAnalyst, DistillerSR, ExaCT and NetMetaXL have potential to be used for the automation of systematic reviews. However, they are not without limitations. The review also identified other studies that employed algorithms that have not yet been developed into user friendly tools. Some of these algorithms showed high validity and reliability but their use is conditional on user knowledge of computer science and algorithms. CONCLUSION Abstract screening has reached maturity; data extraction is still an active area. Developing methods to semi-automate different steps of evidence synthesis via machine learning remains an important research direction. Also, it is important to move from the research prototypes currently available to professionally maintained platforms.
Collapse
Affiliation(s)
- Hanan Khalil
- School of Psychology and Public Health, Department of Public Health, La Trobe University, Melbourne Campus, Victoria, Australia.
| | - Daniel Ameen
- Faculty of Medicine, Nursing and Health Sciences, Monash University, Wellington Road, Clayton Vic 3168, Australia
| | - Armita Zarnegar
- School of Psychology and Public Health, Department of Public Health, La Trobe University, Melbourne Campus, Victoria, Australia.
- School of Science, Computing and engineering technologies, Swinburne University of Technology, Melbourne, Australia
| |
Collapse
|
30
|
Büchter RB, Weise A, Pieper D. Reporting of methods to prepare, pilot and perform data extraction in systematic reviews: analysis of a sample of 152 Cochrane and non-Cochrane reviews. BMC Med Res Methodol 2021; 21:240. [PMID: 34742231 PMCID: PMC8571672 DOI: 10.1186/s12874-021-01438-z] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/13/2021] [Accepted: 10/11/2021] [Indexed: 12/02/2022] Open
Abstract
BACKGROUND Previous research on data extraction methods in systematic reviews has focused on single aspects of the process. We aimed to provide a deeper insight into these methods by analysing a current sample of reviews. METHODS We included systematic reviews of health interventions in humans published in English. We analysed 75 Cochrane reviews from May and June 2020 and a random sample of non-Cochrane reviews published in the same period and retrieved from Medline. We linked reviews with protocols and study registrations. We collected information on preparing, piloting, and performing data extraction and on use of software to assist review conduct (automation tools). Data were extracted by one author, with 20% extracted in duplicate. Data were analysed descriptively. RESULTS Of the 152 included reviews, 77 reported use of a standardized extraction form (51%); 42 provided information on the type of form used (28%); 24 on piloting (16%); 58 on what data was collected (38%); 133 on the extraction method (88%); 107 on resolving disagreements (70%); 103 on methods to obtain additional data or information (68%); 52 on procedures to avoid data errors (34%); and 47 on methods to deal with multiple study reports (31%). Items were more frequently reported in Cochrane than non-Cochrane reviews. The data extraction form used was published in 10 reviews (7%). Use of software was rarely reported except for statistical analysis software and use of RevMan and GRADEpro GDT in Cochrane reviews. Covidence was the most frequent automation tool used: 18 reviews used it for study selection (12%) and 9 for data extraction (6%). CONCLUSIONS Reporting of data extraction methods in systematic reviews is limited, especially in non-Cochrane reviews. This includes core items of data extraction such as methods used to manage disagreements. Few reviews currently use software to assist data extraction and review conduct. Our results can serve as a baseline to assess the uptake of such tools in future analyses.
Collapse
Affiliation(s)
- Roland Brian Büchter
- Institute for Research in Operative Medicine (IFOM), Faculty of Health, School of Medicine, Witten/Herdecke University, Ostmerheimer Str. 200, 51109 Cologne, Germany
| | - Alina Weise
- Institute for Research in Operative Medicine (IFOM), Faculty of Health, School of Medicine, Witten/Herdecke University, Ostmerheimer Str. 200, 51109 Cologne, Germany
| | - Dawid Pieper
- Institute for Research in Operative Medicine (IFOM), Faculty of Health, School of Medicine, Witten/Herdecke University, Ostmerheimer Str. 200, 51109 Cologne, Germany
- Faculty of Health Sciences Brandenburg, Brandenburg Medical School Theodor Fontane, Institute for Health Services and Health System Research, Rüdersdorf, Germany
- Center for Health Services Research, Brandenburg Medical School Theodor Fontane, Rüdersdorf, Germany
| |
Collapse
|
31
|
Pozsgai G, Lövei GL, Vasseur L, Gurr G, Batáry P, Korponai J, Littlewood NA, Liu J, Móra A, Obrycki J, Reynolds O, Stockan JA, VanVolkenburg H, Zhang J, Zhou W, You M. Irreproducibility in searches of scientific literature: A comparative analysis. Ecol Evol 2021; 11:14658-14668. [PMID: 34765132 PMCID: PMC8571571 DOI: 10.1002/ece3.8154] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/11/2021] [Revised: 08/25/2021] [Accepted: 08/27/2021] [Indexed: 11/29/2022] Open
Abstract
Repeatability is the cornerstone of science, and it is particularly important for systematic reviews. However, little is known on how researchers' choice of database, and search platform influence the repeatability of systematic reviews. Here, we aim to unveil how the computer environment and the location where the search was initiated from influence hit results.We present a comparative analysis of time-synchronized searches at different institutional locations in the world and evaluate the consistency of hits obtained within each of the search terms using different search platforms.We revealed a large variation among search platforms and showed that PubMed and Scopus returned consistent results to identical search strings from different locations. Google Scholar and Web of Science's Core Collection varied substantially both in the number of returned hits and in the list of individual articles depending on the search location and computing environment. Inconsistency in Web of Science results has most likely emerged from the different licensing packages at different institutions.To maintain scientific integrity and consistency, especially in systematic reviews, action is needed from both the scientific community and scientific search platforms to increase search consistency. Researchers are encouraged to report the search location and the databases used for systematic reviews, and database providers should make search algorithms transparent and revise access rules to titles behind paywalls. Additional options for increasing the repeatability and transparency of systematic reviews are storing both search metadata and hit results in open repositories and using Application Programming Interfaces (APIs) to retrieve standardized, machine-readable search metadata.
Collapse
Affiliation(s)
- Gábor Pozsgai
- State Key Laboratory of Ecological Pest Control for Fujian and Taiwan CropsInstitute of Applied EcologyFujian Agriculture and Forestry UniversityFuzhouChina
- Joint International Research Laboratory of Ecological Pest ControlMinistry of EducationFuzhouChina
| | - Gábor L. Lövei
- State Key Laboratory of Ecological Pest Control for Fujian and Taiwan CropsInstitute of Applied EcologyFujian Agriculture and Forestry UniversityFuzhouChina
- Joint International Research Laboratory of Ecological Pest ControlMinistry of EducationFuzhouChina
- Department of AgroecologyFlakkebjerg Research CentreAarhus UniversitySlagelseDenmark
| | - Liette Vasseur
- State Key Laboratory of Ecological Pest Control for Fujian and Taiwan CropsInstitute of Applied EcologyFujian Agriculture and Forestry UniversityFuzhouChina
- Joint International Research Laboratory of Ecological Pest ControlMinistry of EducationFuzhouChina
- UNESCO Chair on Community Sustainability: From Local to GlobalDepartment of Biological ScienceBrock UniversitySt. CatharinesONCanada
| | - Geoff Gurr
- State Key Laboratory of Ecological Pest Control for Fujian and Taiwan CropsInstitute of Applied EcologyFujian Agriculture and Forestry UniversityFuzhouChina
- Joint International Research Laboratory of Ecological Pest ControlMinistry of EducationFuzhouChina
- Graham Centre for Agricultural Innovation (Charles Sturt University and NSW Department of Primary Industries)Charles Sturt UniversityOrangeNSWAustralia
| | - Péter Batáry
- AgroecologyUniversity of GoettingenGoettingenGermany
- “Lendület” Landscape and Conservation EcologyInstitute of Ecology and BotanyCentre for Ecological ResearchVácrátótHungary
| | - János Korponai
- Department of BiologySavaria CampusEötvös Loránd UniversitySzombathelyHungary
- Department of Environmental SciencesSapientia Hungarian University of TransylvaniaCluj‐NapocaRomania
- Department of Water Supply and SewerageFaculty of Water ScienceNational University of Public ServiceBajaHungary
- Aquatic Ecological InstituteCentre for Ecological ResearchBudapestHungary
| | - Nick A. Littlewood
- Department of ZoologyUniversity of CambridgeCambridgeUK
- Department of Rural Land UseSRUCAberdeenUK
| | - Jian Liu
- Graham Centre for Agricultural Innovation (Charles Sturt University and NSW Department of Primary Industries)Charles Sturt UniversityOrangeNSWAustralia
| | - Arnold Móra
- Department of HydrobiologyInstitute of BiologyUniversity of PécsPécsHungary
| | | | - Olivia Reynolds
- Joint International Research Laboratory of Ecological Pest ControlMinistry of EducationFuzhouChina
- CesarParkvilleVICAustralia
- Biosecurity and Food SafetyNSW Department of Primary IndustriesNarellanNSWAustralia
| | - Jenni A. Stockan
- Department of Ecological SciencesThe James Hutton InstituteAberdeenUK
| | - Heather VanVolkenburg
- UNESCO Chair on Community Sustainability: From Local to GlobalDepartment of Biological ScienceBrock UniversitySt. CatharinesONCanada
| | - Jie Zhang
- State Key Laboratory of Ecological Pest Control for Fujian and Taiwan CropsInstitute of Applied EcologyFujian Agriculture and Forestry UniversityFuzhouChina
- Joint International Research Laboratory of Ecological Pest ControlMinistry of EducationFuzhouChina
| | - Wenwu Zhou
- State Key Laboratory of Rice BiologyKey Laboratory of Molecular Biology of Crop Pathogens and InsectsMinistry of AgricultureZhejiang UniversityHangzhouChina
| | - Minsheng You
- State Key Laboratory of Ecological Pest Control for Fujian and Taiwan CropsInstitute of Applied EcologyFujian Agriculture and Forestry UniversityFuzhouChina
- Joint International Research Laboratory of Ecological Pest ControlMinistry of EducationFuzhouChina
| |
Collapse
|
32
|
Wagner G, Lukyanenko R, Paré G. Artificial intelligence and the conduct of literature reviews. JOURNAL OF INFORMATION TECHNOLOGY 2021. [DOI: 10.1177/02683962211048201] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Artificial intelligence (AI) is beginning to transform traditional research practices in many areas. In this context, literature reviews stand out because they operate on large and rapidly growing volumes of documents, that is, partially structured (meta)data, and pervade almost every type of paper published in information systems research or related social science disciplines. To familiarize researchers with some of the recent trends in this area, we outline how AI can expedite individual steps of the literature review process. Considering that the use of AI in this context is in an early stage of development, we propose a comprehensive research agenda for AI-based literature reviews (AILRs) in our field. With this agenda, we would like to encourage design science research and a broader constructive discourse on shaping the future of AILRs in research.
Collapse
Affiliation(s)
- Gerit Wagner
- Department of Information Technologies, HEC Montréal, Montréal, Québec, Canada
| | - Roman Lukyanenko
- Department of Information Technologies, HEC Montréal, Montréal, Québec, Canada
| | - Guy Paré
- Department of Information Technologies, HEC Montréal, Montréal, Québec, Canada
| |
Collapse
|
33
|
Scott AM, Forbes C, Clark J, Carter M, Glasziou P, Munn Z. Systematic review automation tools improve efficiency but lack of knowledge impedes their adoption: a survey. J Clin Epidemiol 2021; 138:80-94. [PMID: 34242757 DOI: 10.1016/j.jclinepi.2021.06.030] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/31/2021] [Revised: 06/23/2021] [Accepted: 06/30/2021] [Indexed: 01/12/2023]
Abstract
OBJECTIVE We investigated systematic review automation tool use by systematic reviewers, health technology assessors and clinical guideline developerst. STUDY DESIGN AND SETTING An online, 16-question survey was distributed across several evidence synthesis, health technology assessment and guideline development organizations. We asked the respondents what tools they use and abandon, how often and when do they use the tools, their perceived time savings and accuracy, and desired new tools. Descriptive statistics were used to report the results. RESULTS A total of 253 respondents completed the survey; 89% have used systematic review automation tools - most frequently whilst screening (79%). Respondents' "top 3" tools included: Covidence (45%), RevMan (35%), Rayyan and GRADEPro (both 22%); most commonly abandoned were Rayyan (19%), Covidence (15%), DistillerSR (14%) and RevMan (13%). Tools saved time (80%) and increased accuracy (54%). Respondents taught themselves to how to use the tools (72%); lack of knowledge was the most frequent barrier to tool adoption (51%). New tool development was suggested for the searching and data extraction stages. CONCLUSION Automation tools will likely have an increasingly important role in high-quality and timely reviews. Further work is required in training and dissemination of automation tools and ensuring they meet the desirable features of those conducting systematic reviews.
Collapse
Affiliation(s)
- Anna Mae Scott
- Institute for Evidence-Based Healthcare, Bond University, Gold Coast, Australia.
| | - Connor Forbes
- Institute for Evidence-Based Healthcare, Bond University, Gold Coast, Australia
| | - Justin Clark
- Institute for Evidence-Based Healthcare, Bond University, Gold Coast, Australia
| | - Matt Carter
- Institute for Evidence-Based Healthcare, Bond University, Gold Coast, Australia
| | - Paul Glasziou
- Institute for Evidence-Based Healthcare, Bond University, Gold Coast, Australia
| | - Zachary Munn
- JBI, Faculty of Health and Medical Sciences, The University of Adelaide, Adelaide, Australia
| |
Collapse
|
34
|
Clark J, McFarlane C, Cleo G, Ishikawa Ramos C, Marshall S. The Impact of Systematic Review Automation Tools on Methodological Quality and Time Taken to Complete Systematic Review Tasks: Case Study. JMIR MEDICAL EDUCATION 2021; 7:e24418. [PMID: 34057072 PMCID: PMC8204237 DOI: 10.2196/24418] [Citation(s) in RCA: 15] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/18/2020] [Revised: 03/03/2021] [Accepted: 04/04/2021] [Indexed: 06/12/2023]
Abstract
BACKGROUND Systematic reviews (SRs) are considered the highest level of evidence to answer research questions; however, they are time and resource intensive. OBJECTIVE When comparing SR tasks done manually, using standard methods, versus those same SR tasks done using automated tools, (1) what is the difference in time to complete the SR task and (2) what is the impact on the error rate of the SR task? METHODS A case study compared specific tasks done during the conduct of an SR on prebiotic, probiotic, and synbiotic supplementation in chronic kidney disease. Two participants (manual team) conducted the SR using current methods, comprising a total of 16 tasks. Another two participants (automation team) conducted the tasks where a systematic review automation (SRA) tool was available, comprising of a total of six tasks. The time taken and error rate of the six tasks that were completed by both teams were compared. RESULTS The approximate time for the manual team to produce a draft of the background, methods, and results sections of the SR was 126 hours. For the six tasks in which times were compared, the manual team spent 2493 minutes (42 hours) on the tasks, compared to 708 minutes (12 hours) spent by the automation team. The manual team had a higher error rate in two of the six tasks-regarding Task 5: Run the systematic search, the manual team made eight errors versus three errors made by the automation team; regarding Task 12: Assess the risk of bias, 25 assessments differed from a reference standard for the manual team compared to 20 differences for the automation team. The manual team had a lower error rate in one of the six tasks-regarding Task 6: Deduplicate search results, the manual team removed one unique study and missed zero duplicates versus the automation team who removed two unique studies and missed seven duplicates. Error rates were similar for the two remaining compared tasks-regarding Task 7: Screen the titles and abstracts and Task 9: Screen the full text, zero relevant studies were excluded by both teams. One task could not be compared between groups-Task 8: Find the full text. CONCLUSIONS For the majority of SR tasks where an SRA tool was used, the time required to complete that task was reduced for novice researchers while methodological quality was maintained.
Collapse
Affiliation(s)
- Justin Clark
- Institute for Evidence-Based Healthcare, Bond University, Gold Coast, Australia
| | - Catherine McFarlane
- Bond University Nutrition & Dietetics Research Group, Faculty of Health Sciences and Medicine, Bond University, Gold Coast, Australia
- Renal Department, Sunshine Coast University Hospital, Birtinya, Australia
| | - Gina Cleo
- Institute for Evidence-Based Healthcare, Bond University, Gold Coast, Australia
| | - Christiane Ishikawa Ramos
- Bond University Nutrition & Dietetics Research Group, Faculty of Health Sciences and Medicine, Bond University, Gold Coast, Australia
- Nutrition Programme, Federal University of Sao Paulo, Sao Paulo, Brazil
| | - Skye Marshall
- Bond University Nutrition & Dietetics Research Group, Faculty of Health Sciences and Medicine, Bond University, Gold Coast, Australia
- Department of Science, Nutrition Research Australia, Sydney, Australia
| |
Collapse
|
35
|
Schmidt L, Finnerty Mutlu AN, Elmore R, Olorisade BK, Thomas J, Higgins JPT. Data extraction methods for systematic review (semi)automation: Update of a living systematic review. F1000Res 2021; 10:401. [PMID: 34408850 PMCID: PMC8361807 DOI: 10.12688/f1000research.51117.2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 07/27/2023] [Indexed: 10/12/2023] Open
Abstract
Background: The reliable and usable (semi)automation of data extraction can support the field of systematic review by reducing the workload required to gather information about the conduct and results of the included studies. This living systematic review examines published approaches for data extraction from reports of clinical studies. Methods: We systematically and continually search PubMed, ACL Anthology, arXiv, OpenAlex via EPPI-Reviewer, and the dblp computer science bibliography. Full text screening and data extraction are conducted within an open-source living systematic review application created for the purpose of this review. This living review update includes publications up to December 2022 and OpenAlex content up to March 2023. Results: 76 publications are included in this review. Of these, 64 (84%) of the publications addressed extraction of data from abstracts, while 19 (25%) used full texts. A total of 71 (93%) publications developed classifiers for randomised controlled trials. Over 30 entities were extracted, with PICOs (population, intervention, comparator, outcome) being the most frequently extracted. Data are available from 25 (33%), and code from 30 (39%) publications. Six (8%) implemented publicly available tools Conclusions: This living systematic review presents an overview of (semi)automated data-extraction literature of interest to different types of literature review. We identified a broad evidence base of publications describing data extraction for interventional reviews and a small number of publications extracting epidemiological or diagnostic accuracy data. Between review updates, trends for sharing data and code increased strongly: in the base-review, data and code were available for 13 and 19% respectively, these numbers increased to 78 and 87% within the 23 new publications. Compared with the base-review, we observed another research trend, away from straightforward data extraction and towards additionally extracting relations between entities or automatic text summarisation. With this living review we aim to review the literature continually.
Collapse
Affiliation(s)
- Lena Schmidt
- NIHR Innovation Observatory, Newcastle University, Newcastle upon Tyne, NE4 5TG, UK
- Sciome LLC, Research Triangle Park, North Carolina, 27713, USA
- Bristol Medical School, University of Bristol, Bristol, BS8 2PS, UK
| | | | - Rebecca Elmore
- Sciome LLC, Research Triangle Park, North Carolina, 27713, USA
| | - Babatunde K. Olorisade
- Bristol Medical School, University of Bristol, Bristol, BS8 2PS, UK
- Evaluate Ltd, London, SE1 2RE, UK
- Cardiff School of Technologies, Cardiff Metropolitan University, Cardiff, CF5 2YB, UK
| | - James Thomas
- UCL Social Research Institute, University College London, London, WC1H 0AL, UK
| | | |
Collapse
|
36
|
Schmidt L, Finnerty Mutlu AN, Elmore R, Olorisade BK, Thomas J, Higgins JPT. Data extraction methods for systematic review (semi)automation: A living systematic review. F1000Res 2021; 10:401. [PMID: 34408850 PMCID: PMC8361807 DOI: 10.12688/f1000research.51117.1] [Citation(s) in RCA: 17] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 05/04/2021] [Indexed: 01/07/2023] Open
Abstract
Background: The reliable and usable (semi)automation of data extraction can support the field of systematic review by reducing the workload required to gather information about the conduct and results of the included studies. This living systematic review examines published approaches for data extraction from reports of clinical studies. Methods: We systematically and continually search MEDLINE, Institute of Electrical and Electronics Engineers (IEEE), arXiv, and the dblp computer science bibliography databases. Full text screening and data extraction are conducted within an open-source living systematic review application created for the purpose of this review. This iteration of the living review includes publications up to a cut-off date of 22 April 2020. Results: In total, 53 publications are included in this version of our review. Of these, 41 (77%) of the publications addressed extraction of data from abstracts, while 14 (26%) used full texts. A total of 48 (90%) publications developed and evaluated classifiers that used randomised controlled trials as the main target texts. Over 30 entities were extracted, with PICOs (population, intervention, comparator, outcome) being the most frequently extracted. A description of their datasets was provided by 49 publications (94%), but only seven (13%) made the data publicly available. Code was made available by 10 (19%) publications, and five (9%) implemented publicly available tools. Conclusions: This living systematic review presents an overview of (semi)automated data-extraction literature of interest to different types of systematic review. We identified a broad evidence base of publications describing data extraction for interventional reviews and a small number of publications extracting epidemiological or diagnostic accuracy data. The lack of publicly available gold-standard data for evaluation, and lack of application thereof, makes it difficult to draw conclusions on which is the best-performing system for each data extraction target. With this living review we aim to review the literature continually.
Collapse
Affiliation(s)
- Lena Schmidt
- NIHR Innovation Observatory, Newcastle University, Newcastle upon Tyne, NE4 5TG, UK
- Sciome LLC, Research Triangle Park, North Carolina, 27713, USA
- Bristol Medical School, University of Bristol, Bristol, BS8 2PS, UK
| | | | - Rebecca Elmore
- Sciome LLC, Research Triangle Park, North Carolina, 27713, USA
| | - Babatunde K. Olorisade
- Bristol Medical School, University of Bristol, Bristol, BS8 2PS, UK
- Evaluate Ltd, London, SE1 2RE, UK
- Cardiff School of Technologies, Cardiff Metropolitan University, Cardiff, CF5 2YB, UK
| | - James Thomas
- UCL Social Research Institute, University College London, London, WC1H 0AL, UK
| | | |
Collapse
|
37
|
Michael Clark J, Beller E, Glasziou P, Sanders S. The decisions and processes involved in a systematic search strategy: a hierarchical framework. J Med Libr Assoc 2021; 109:201-211. [PMID: 34285663 PMCID: PMC8270345 DOI: 10.5195/jmla.2021.1086] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/02/2023] Open
Abstract
OBJECTIVE The decisions and processes that may compose a systematic search strategy have not been formally identified and categorized. This study aimed to (1) identify all decisions that could be made and processes that could be used in a systematic search strategy and (2) create a hierarchical framework of those decisions and processes. METHODS The literature was searched for documents or guides on conducting a literature search for a systematic review or other evidence synthesis. The decisions or processes for locating studies were extracted from eligible documents and categorized into a structured hierarchical framework. Feedback from experts was sought to revise the framework. The framework was revised iteratively and tested using recently published literature on systematic searching. RESULTS Guidance documents were identified from expert organizations and a search of the literature and Internet. Data were extracted from 74 eligible documents to form the initial framework. The framework was revised based on feedback from 9 search experts and further review and testing by the authors. The hierarchical framework consists of 119 decisions or processes sorted into 17 categories and arranged under 5 topics. These topics are "Skill of the searcher," "Selecting information to identify," "Searching the literature electronically," "Other ways to identify studies," and "Updating the systematic review." CONCLUSIONS The work identifies and classifies the decisions and processes used in systematic searching. Future work can now focus on assessing and prioritizing research on the best methods for successfully identifying all eligible studies for a systematic review.
Collapse
Affiliation(s)
- Justin Michael Clark
- , Institute for Evidence-Based Healthcare, Bond University, Robina, Queensland, Australia
| | - Elaine Beller
- , Institute for Evidence-Based Healthcare, Bond University, Robina, Queensland, Australia
| | - Paul Glasziou
- , Institute for Evidence-Based Healthcare, Bond University, Robina, Queensland, Australia
| | - Sharon Sanders
- , Institute for Evidence-Based Healthcare, Bond University, Robina, Queensland, Australia
| |
Collapse
|
38
|
Brassey J, Price C, Edwards J, Zlabinger M, Bampoulidis A, Hanbury A. Developing a fully automated evidence synthesis tool for identifying, assessing and collating the evidence. BMJ Evid Based Med 2021; 26:24-27. [PMID: 31467247 DOI: 10.1136/bmjebm-2018-111126] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 08/05/2019] [Indexed: 01/06/2023]
Abstract
Evidence synthesis is a key element of evidence-based medicine. However, it is currently hampered by being labour intensive meaning that many trials are not incorporated into robust evidence syntheses and that many are out of date. To overcome this, a variety of techniques are being explored, including using automation technology. Here, we describe a fully automated evidence synthesis system for intervention studies, one that identifies all the relevant evidence, assesses the evidence for reliability and collates it to estimate the relative effectiveness of an intervention. Techniques used include machine learning, natural language processing and rule-based systems. Results are visualised using modern visualisation techniques. We believe this to be the first, publicly available, automated evidence synthesis system: an evidence mapping tool that synthesises evidence on the fly.
Collapse
Affiliation(s)
| | | | | | - Markus Zlabinger
- Institute of Information Systems Engineering, TU Wien (Vienna University of Technology), Vienna, Austria
| | - Alexandros Bampoulidis
- Institute of Information Systems Engineering, TU Wien (Vienna University of Technology), Vienna, Austria
- Research Studio Data Science, RSA FG, Vienna, Austria
| | - Allan Hanbury
- Institute of Information Systems Engineering, TU Wien (Vienna University of Technology), Vienna, Austria
| |
Collapse
|
39
|
Gates A, Gates M, DaRosa D, Elliott SA, Pillay J, Rahman S, Vandermeer B, Hartling L. Decoding semi-automated title-abstract screening: findings from a convenience sample of reviews. Syst Rev 2020; 9:272. [PMID: 33243276 PMCID: PMC7694314 DOI: 10.1186/s13643-020-01528-x] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/08/2020] [Accepted: 11/11/2020] [Indexed: 01/08/2023] Open
Abstract
BACKGROUND We evaluated the benefits and risks of using the Abstrackr machine learning (ML) tool to semi-automate title-abstract screening and explored whether Abstrackr's predictions varied by review or study-level characteristics. METHODS For a convenience sample of 16 reviews for which adequate data were available to address our objectives (11 systematic reviews and 5 rapid reviews), we screened a 200-record training set in Abstrackr and downloaded the relevance (relevant or irrelevant) of the remaining records, as predicted by the tool. We retrospectively simulated the liberal-accelerated screening approach. We estimated the time savings and proportion missed compared with dual independent screening. For reviews with pairwise meta-analyses, we evaluated changes to the pooled effects after removing the missed studies. We explored whether the tool's predictions varied by review and study-level characteristics. RESULTS Using the ML-assisted liberal-accelerated approach, we wrongly excluded 0 to 3 (0 to 14%) records that were included in the final reports, but saved a median (IQR) 26 (9, 42) h of screening time. One missed study was included in eight pairwise meta-analyses in one systematic review. The pooled effect for just one of those meta-analyses changed considerably (from MD (95% CI) - 1.53 (- 2.92, - 0.15) to - 1.17 (- 2.70, 0.36)). Of 802 records in the final reports, 87% were correctly predicted as relevant. The correctness of the predictions did not differ by review (systematic or rapid, P = 0.37) or intervention type (simple or complex, P = 0.47). The predictions were more often correct in reviews with multiple (89%) vs. single (83%) research questions (P = 0.01), or that included only trials (95%) vs. multiple designs (86%) (P = 0.003). At the study level, trials (91%), mixed methods (100%), and qualitative (93%) studies were more often correctly predicted as relevant compared with observational studies (79%) or reviews (83%) (P = 0.0006). Studies at high or unclear (88%) vs. low risk of bias (80%) (P = 0.039), and those published more recently (mean (SD) 2008 (7) vs. 2006 (10), P = 0.02) were more often correctly predicted as relevant. CONCLUSION Our screening approach saved time and may be suitable in conditions where the limited risk of missing relevant records is acceptable. Several of our findings are paradoxical and require further study to fully understand the tasks to which ML-assisted screening is best suited. The findings should be interpreted in light of the fact that the protocol was prepared for the funder, but not published a priori. Because we used a convenience sample, the findings may be prone to selection bias. The results may not be generalizable to other samples of reviews, ML tools, or screening approaches. The small number of missed studies across reviews with pairwise meta-analyses hindered strong conclusions about the effect of missed studies on the results and conclusions of systematic reviews.
Collapse
Affiliation(s)
- Allison Gates
- Alberta Research Centre for Health Evidence and the University of Alberta Evidence-based Practice Center, Department of Pediatrics, University of Alberta, Edmonton, Alberta Canada
| | - Michelle Gates
- Alberta Research Centre for Health Evidence and the University of Alberta Evidence-based Practice Center, Department of Pediatrics, University of Alberta, Edmonton, Alberta Canada
| | - Daniel DaRosa
- Alberta Research Centre for Health Evidence and the University of Alberta Evidence-based Practice Center, Department of Pediatrics, University of Alberta, Edmonton, Alberta Canada
| | - Sarah A. Elliott
- Alberta Research Centre for Health Evidence and the University of Alberta Evidence-based Practice Center, Department of Pediatrics, University of Alberta, Edmonton, Alberta Canada
| | - Jennifer Pillay
- Alberta Research Centre for Health Evidence and the University of Alberta Evidence-based Practice Center, Department of Pediatrics, University of Alberta, Edmonton, Alberta Canada
| | - Sholeh Rahman
- Alberta Research Centre for Health Evidence and the University of Alberta Evidence-based Practice Center, Department of Pediatrics, University of Alberta, Edmonton, Alberta Canada
| | - Ben Vandermeer
- Alberta Research Centre for Health Evidence and the University of Alberta Evidence-based Practice Center, Department of Pediatrics, University of Alberta, Edmonton, Alberta Canada
| | - Lisa Hartling
- Alberta Research Centre for Health Evidence and the University of Alberta Evidence-based Practice Center, Department of Pediatrics, University of Alberta, Edmonton, Alberta Canada
| |
Collapse
|
40
|
Haddaway NR, Westgate MJ. Creating and curating a community of practice: introducing the evidence synthesis Hackathon and a special series in evidence synthesis technology. ENVIRONMENTAL EVIDENCE 2020; 9:28. [PMID: 33230412 PMCID: PMC7676418 DOI: 10.1186/s13750-020-00212-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 05/07/2020] [Accepted: 11/04/2020] [Indexed: 06/11/2023]
Abstract
Evidence synthesis is a vital part of evidence-informed decision-making, but high growth in the volume of research evidence over recent decades has made efficient evidence synthesis increasingly challenging. As the appreciation and need for timely and rigorous evidence synthesis continue to grow, so too will the need for tools and frameworks to conduct reviews of expanding evidence bases in an efficient and time-sensitive manner. Efforts to future-proof evidence synthesis through the development of new evidence synthesis technology (ESTech) have so far been isolated across interested individuals or groups, with no concerted effort to collaborate or build communities of practice in technology production. We established the evidence synthesis Hackathon to stimulate collaboration and the production of Free and Open Source Software and frameworks to support evidence synthesis. Here, we introduce a special series of papers on ESTech, and invite the readers of environmental evidence to submit manuscripts introducing and validating novel tools and frameworks. We hope this collection will help to consolidate ESTech development efforts and we encourage readers to join the ESTech revolution. In order to future-proof evidence synthesis against the evidence avalanche, we must support community enthusiasm for ESTech, reduce redundancy in tool design, collaborate and share capacity in tool production, and reduce inequalities in software accessibility.
Collapse
Affiliation(s)
- Neal R. Haddaway
- Mercator Research Institute On Global Commons and Climate Change, Torgauer Str. 19, 10829 Berlin, Germany
- Stockholm Environment Institute, Linnégatan 87D, Stockholm, Sweden
- Africa Centre for Evidence, University of Johannesburg, Johannesburg, South Africa
| | - Martin J. Westgate
- Fenner School of Environment and Society, Australian National University, Acton ACT 2601, Australia
| |
Collapse
|
41
|
Büchter RB, Weise A, Pieper D. Development, testing and use of data extraction forms in systematic reviews: a review of methodological guidance. BMC Med Res Methodol 2020; 20:259. [PMID: 33076832 PMCID: PMC7574308 DOI: 10.1186/s12874-020-01143-3] [Citation(s) in RCA: 25] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/11/2020] [Accepted: 10/07/2020] [Indexed: 01/08/2023] Open
Abstract
Background Data extraction forms link systematic reviews with primary research and provide the foundation for appraising, analysing, summarising and interpreting a body of evidence. This makes their development, pilot testing and use a crucial part of the systematic reviews process. Several studies have shown that data extraction errors are frequent in systematic reviews, especially regarding outcome data. Methods We reviewed guidance on the development and pilot testing of data extraction forms and the data extraction process. We reviewed four types of sources: 1) methodological handbooks of systematic review organisations (SRO); 2) textbooks on conducting systematic reviews; 3) method documents from health technology assessment (HTA) agencies and 4) journal articles. HTA documents were retrieved in February 2019 and database searches conducted in December 2019. One author extracted the recommendations and a second author checked them for accuracy. Results are presented descriptively. Results Our analysis includes recommendations from 25 documents: 4 SRO handbooks, 11 textbooks, 5 HTA method documents and 5 journal articles. Across these sources the most common recommendations on form development are to use customized or adapted standardised extraction forms (14/25); provide detailed instructions on their use (10/25); ensure clear and consistent coding and response options (9/25); plan in advance which data are needed (9/25); obtain additional data if required (8/25); and link multiple reports of the same study (8/25). The most frequent recommendations on piloting extractions forms are that forms should be piloted on a sample of studies (18/25); and that data extractors should be trained in the use of the forms (7/25). The most frequent recommendations on data extraction are that extraction should be conducted by at least two people (17/25); that independent parallel extraction should be used (11/25); and that procedures to resolve disagreements between data extractors should be in place (14/25). Conclusions Overall, our results suggest a lack of comprehensiveness of recommendations. This may be particularly problematic for less experienced reviewers. Limitations of our method are the scoping nature of the review and that we did not analyse internal documents of health technology agencies.
Collapse
Affiliation(s)
- Roland Brian Büchter
- Institute for Research in Operative Medicine (IFOM), Faculty of Health - School of Medicine, Witten/Herdecke University, Ostmerheimer Str. 200, 51109, Cologne, Germany.
| | - Alina Weise
- Institute for Research in Operative Medicine (IFOM), Faculty of Health - School of Medicine, Witten/Herdecke University, Ostmerheimer Str. 200, 51109, Cologne, Germany
| | - Dawid Pieper
- Institute for Research in Operative Medicine (IFOM), Faculty of Health - School of Medicine, Witten/Herdecke University, Ostmerheimer Str. 200, 51109, Cologne, Germany
| |
Collapse
|
42
|
Hamel C, Michaud A, Thuku M, Affengruber L, Skidmore B, Nussbaumer-Streit B, Stevens A, Garritty C. Few evaluative studies exist examining rapid review methodology across stages of conduct: a systematic scoping review. J Clin Epidemiol 2020; 126:131-140. [DOI: 10.1016/j.jclinepi.2020.06.027] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/10/2020] [Revised: 06/18/2020] [Accepted: 06/23/2020] [Indexed: 10/24/2022]
|
43
|
Lund H, Juhl CB, Nørgaard B, Draborg E, Henriksen M, Andreasen J, Christensen R, Nasser M, Ciliska D, Clarke M, Tugwell P, Martin J, Blaine C, Brunnhuber K, Robinson KA. Evidence-Based Research Series-Paper 2 : Using an Evidence-Based Research approach before a new study is conducted to ensure value. J Clin Epidemiol 2020; 129:158-166. [PMID: 32987159 DOI: 10.1016/j.jclinepi.2020.07.019] [Citation(s) in RCA: 31] [Impact Index Per Article: 7.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2020] [Accepted: 07/20/2020] [Indexed: 12/13/2022]
Abstract
BACKGROUND AND OBJECTIVES There is considerable actual and potential waste in research. The aim of this article is to describe how using an evidence-based research approach before conducting a study helps to ensure that the new study truly adds value. STUDY DESIGN AND SETTING Evidence-based research is the use of prior research in a systematic and transparent way to inform a new study so that it is answering questions that matter in a valid, efficient, and accessible manner. In this second article of the evidence-based research series, we describe how to apply an evidence-based research approach before starting a new study. RESULTS Before a new study is performed, researchers need to provide a solid justification for it using the available scientific knowledge as well as the perspectives of end users. The key method for both is to conduct a systematic review of earlier relevant studies. CONCLUSION Describing the ideal process illuminates the challenges and opportunities offered through the suggested evidence-based research approach. A systematic and transparent approach is needed to provide justification for and to optimally design a relevant and necessary new study.
Collapse
Affiliation(s)
- Hans Lund
- Section for Evidence-based Practice, Western Norway University of Applied Sciences, Bergen, Norway.
| | - Carsten B Juhl
- Department of Sport Science and Clinical Biomechanics, University of Southern Denmark, Odense, Denmark; Department of Physiotherapy and Occupational Therapy, University Hospital of Copenhagen, Herlev & Gentofte, Denmark
| | - Birgitte Nørgaard
- Department of Public Health, University of Southern Denmark, Odense, Denmark
| | - Eva Draborg
- Department of Public Health, University of Southern Denmark, Odense, Denmark
| | - Marius Henriksen
- The Parker Institute, Bispebjerg and Frederiksberg Hospital, University of Copenhagen, Copenhagen, Denmark
| | - Jane Andreasen
- Department of Health, Science and Technology, Public Health and Epidemiology Group, Aalborg University, Alborg, Denmark; Department of Physical and Occupational Therapy, Aalborg University Hospital, Aalborg, Denmark
| | - Robin Christensen
- Musculoskeletal Statistics Unit, The Parker Institute, Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark; Department of Clinical Research, Research Unit of Rheumatology, University of Southern Denmark, Odense University Hospital, Denmark
| | - Mona Nasser
- Peninsula Dental School, Plymouth University, Plymouth, England, UK
| | - Donna Ciliska
- Section for Evidence-based Practice, Western Norway University of Applied Sciences, Bergen, Norway; School of Nursing, McMaster University, Hamilton, Canada
| | - Mike Clarke
- Northern Ireland Methodology Hub, Queen's University Belfast, Northern Ireland
| | - Peter Tugwell
- Department of Medicine, University of Ottawa, Ottawa, Canada
| | - Janet Martin
- MEDICI Centre, Schulich School of Medicine & Dentistry, Western University, London, Ontario, Canada; Departments of Anesthesia & Biostatistics and Epidemiology & Biostatistics, Western University, London, Canada
| | | | - Klara Brunnhuber
- Digital Content Services, Data Platform Operations, Elsevier, London UK
| | | |
Collapse
|
44
|
Robinson KA, Brunnhuber K, Ciliska D, Juhl CB, Christensen R, Lund H. Evidence-Based Research Series-Paper 1: What Evidence-Based Research is and why is it important? J Clin Epidemiol 2020; 129:151-157. [PMID: 32979491 DOI: 10.1016/j.jclinepi.2020.07.020] [Citation(s) in RCA: 43] [Impact Index Per Article: 10.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2020] [Accepted: 07/20/2020] [Indexed: 01/28/2023]
Abstract
OBJECTIVES There is considerable actual and potential waste in research. Evidence-based research ensures worthwhile and valuable research. The aim of this series, which this article introduces, is to describe the evidence-based research approach. STUDY DESIGN AND SETTING In this first article of a three-article series, we introduce the evidence-based research approach. Evidence-based research is the use of prior research in a systematic and transparent way to inform a new study so that it is answering questions that matter in a valid, efficient, and accessible manner. RESULTS We describe evidence-based research and provide an overview of the approach of systematically and transparently using previous research before starting a new study to justify and design the new study (article #2 in series) and-on study completion-place its results in the context with what is already known (article #3 in series). CONCLUSION This series introduces evidence-based research as an approach to minimize unnecessary and irrelevant clinical health research that is unscientific, wasteful, and unethical.
Collapse
Affiliation(s)
- Karen A Robinson
- Johns Hopkins Evidence-based Practice Center, Division of General Internal Medicine, Department of Medicine, Johns Hopkins University, Baltimore, MD, USA
| | - Klara Brunnhuber
- Digital Content Services, Operations, Elsevier Ltd., 125 London Wall, London, EC2Y 5AS, UK
| | - Donna Ciliska
- School of Nursing, McMaster University, Health Sciences Centre, Room 2J20, 1280 Main Street West, Hamilton, Ontario, Canada, L8S 4K1; Section for Evidence-Based Practice, Western Norway University of Applied Sciences, Inndalsveien 28, Bergen, P.O.Box 7030 N-5020 Bergen, Norway
| | - Carsten Bogh Juhl
- Department of Sport Science and Clinical Biomechanics, University of Southern Denmark, Campusvej 55, 5230, Odense M, Denmark; Department of Physiotherapy and Occupational Therapy, University Hospital of Copenhagen, Herlev & Gentofte, Kildegaardsvej 28, 2900, Hellerup, Denmark
| | - Robin Christensen
- Musculoskeletal Statistics Unit, the Parker Institute, Bispebjerg and Frederiksberg Hospital, Copenhagen, Nordre Fasanvej 57, 2000, Copenhagen F, Denmark; Department of Clinical Research, Research Unit of Rheumatology, University of Southern Denmark, Odense University Hospital, Denmark
| | - Hans Lund
- Section for Evidence-Based Practice, Western Norway University of Applied Sciences, Inndalsveien 28, Bergen, P.O.Box 7030 N-5020 Bergen, Norway.
| | | |
Collapse
|
45
|
Bougioukas KI, Bouras EC, Avgerinos KI, Dardavessis T, Haidich A. How to keep up to date with medical information using web‐based resources: a systematised review and narrative synthesis. Health Info Libr J 2020; 37:254-292. [DOI: 10.1111/hir.12318] [Citation(s) in RCA: 14] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/26/2019] [Accepted: 05/20/2020] [Indexed: 12/30/2022]
Affiliation(s)
- Konstantinos I. Bougioukas
- Department of Hygiene Social‐Preventive Medicine and Medical Statistics Medical School Aristotle University of Thessaloniki Thessaloniki Greece
| | - Emmanouil C. Bouras
- Department of Hygiene Social‐Preventive Medicine and Medical Statistics Medical School Aristotle University of Thessaloniki Thessaloniki Greece
| | | | - Theodore Dardavessis
- Department of Hygiene Social‐Preventive Medicine and Medical Statistics Medical School Aristotle University of Thessaloniki Thessaloniki Greece
| | - Anna‐Bettina Haidich
- Department of Hygiene Social‐Preventive Medicine and Medical Statistics Medical School Aristotle University of Thessaloniki Thessaloniki Greece
| |
Collapse
|
46
|
Artificial intelligence and automation of systematic reviews in women's health. Curr Opin Obstet Gynecol 2020; 32:335-341. [PMID: 32516150 DOI: 10.1097/gco.0000000000000643] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
Abstract
PURPOSE OF REVIEW Evidence-based women's healthcare is underpinned by systematic reviews and guidelines. Generating an evidence synthesis to support guidance for clinical practice is a time-consuming and labour-intensive activity that delays transfer of research into practice. Artificial intelligence has the potential to rapidly collate, combine, and update high-quality medical evidence with accuracy and precision, and without bias. RECENT FINDINGS This article describes the main fields of artificial intelligence with examples of its application to systematic reviews. These include the capabilities of processing natural language texts, retrieving information, reasoning, and learning. The complementarity and interconnection of the various artificial intelligence techniques can be harnessed to solve difficult problems in automation of reviews. Computer science can advance evidence-based medicine through development, testing, and refinement of artificial intelligence tools to deploy automation, creating 'living' evidence syntheses. SUMMARY Groundbreaking, high-quality, and impactful artificial intelligence will accelerate the transfer of individual research studies seamlessly into evidence syntheses for contemporaneously improving the quality of healthcare.
Collapse
|
47
|
Arevalo-Rodriguez I, Steingart KR, Tricco AC, Nussbaumer-Streit B, Kaunelis D, Alonso-Coello P, Baxter S, Bossuyt PM, Emparanza JI, Zamora J. Current methods for development of rapid reviews about diagnostic tests: an international survey. BMC Med Res Methodol 2020; 20:115. [PMID: 32404051 PMCID: PMC7220561 DOI: 10.1186/s12874-020-01004-z] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/20/2020] [Accepted: 04/30/2020] [Indexed: 12/12/2022] Open
Abstract
BACKGROUND Rapid reviews (RRs) have emerged as an efficient alternative to time-consuming systematic reviews-they can help meet the demand for accelerated evidence synthesis to inform decision-making in healthcare. The synthesis of diagnostic evidence has important methodological challenges. Here, we performed an international survey to identify the current practice of producing RRs for diagnostic tests. METHODS We developed and administered an online survey inviting institutions that perform RRs of diagnostic tests from all over the world. RESULTS All participants (N = 25) reported the implementation of one or more methods to define the scope of the RR; however, only one strategy (defining a structured question) was used by ≥90% of participants. All participants used at least one methodological shortcut including the use of a previous review as a starting point (92%) and the use of limits on the search (96%). Parallelization and automation of review tasks were not extensively used (48 and 20%, respectively). CONCLUSION Our survey indicates a greater use of shortcuts and limits for conducting diagnostic test RRs versus the results of a recent scoping review analyzing published RRs. Several shortcuts are used without knowing how their implementation affects the results of the evidence synthesis in the setting of diagnostic test reviews. Thus, a structured evaluation of the challenges and implications of the adoption of these RR methods is warranted.
Collapse
Affiliation(s)
- Ingrid Arevalo-Rodriguez
- Clinical Biostatistics Unit, Hospital Universitario Ramón y Cajal, IRYCIS, CIBER of Epidemiology and Public Health, Madrid, Spain
| | - Karen R. Steingart
- Department of Clinical Sciences, Liverpool School of Tropical Medicine, Liverpool, UK
| | - Andrea C. Tricco
- Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Unity Health Toronto, Toronto, Canada
- Epidemiology Division, Dalla Lana School of Public Health and Institute of Health Policy, Management and Evaluation at the University of Toronto, Toronto, Canada
- Queen’s Collaboration for Health Care Quality, Joanna Briggs Institute Centre of Excellence, Queen’s University, Kingston, Canada
| | | | - David Kaunelis
- Canadian Agency for Drugs and Technologies in Health (CADTH), Ottawa, Canada
| | - Pablo Alonso-Coello
- Iberoamerican Cochrane Center-Servicio de Epidemiología Clínica y Salud Pública, Biomedical Research Institute (IIB-Sant Pau), Barcelona, Spain
- CIBER of Epidemiology and Public Health, Barcelona, Spain
| | - Susan Baxter
- School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK
| | - Patrick M. Bossuyt
- Department of Clinical Epidemiology, Biostatistics and Bioinformatics, Amsterdam University Medical Centres, University of Amsterdam, Amsterdam, The Netherlands
| | - José Ignacio Emparanza
- Clinical Epidemiology Unit, Hospital Universitario Donostia, BioDonostia, CIBER of Epidemiology and Public Health, San Sebastian, Spain
| | - Javier Zamora
- Clinical Biostatistics Unit, Hospital Universitario Ramón y Cajal, IRYCIS, CIBER of Epidemiology and Public Health, Madrid, Spain
- Barts and the London School of Medicine and Dentistry, Queen Mary University, London, UK
| |
Collapse
|
48
|
O'Connor AM, Glasziou P, Taylor M, Thomas J, Spijker R, Wolfe MS. A focus on cross-purpose tools, automated recognition of study design in multiple disciplines, and evaluation of automation tools: a summary of significant discussions at the fourth meeting of the International Collaboration for Automation of Systematic Reviews (ICASR). Syst Rev 2020; 9:100. [PMID: 32366302 PMCID: PMC7199360 DOI: 10.1186/s13643-020-01351-4] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/26/2019] [Accepted: 04/07/2020] [Indexed: 11/10/2022] Open
Abstract
The fourth meeting of the International Collaboration for Automation of Systematic Reviews (ICASR) was held 5-6 November 2019 in The Hague, the Netherlands. ICASR is an interdisciplinary group whose goal is to maximize the use of technology for conducting rapid, accurate, and efficient systematic reviews of scientific evidence. The group seeks to facilitate the development and acceptance of automated techniques for systematic reviews. In 2018, the major themes discussed were the transferability of automation tools (i.e., tools developed for other purposes that might be used by systematic reviewers), the automated recognition of study design in multiple disciplines and applications, and approaches for the evaluation of automation tools.
Collapse
Affiliation(s)
- Annette M O'Connor
- College of Veterinary Medicine, Iowa State University, 1800 Christensen Drive, Ames, IA, 50011-1134, USA. .,Present Address: College of Veterinary Medicine, Michigan State University, East Lansing, MI, 48824, USA.
| | - Paul Glasziou
- Bond University, Robina, Queensland, 4226, Australia
| | - Michele Taylor
- US Environmental Protection Agency, Research Triangle Park, NC, 27709, USA
| | - James Thomas
- EPPI-Centre, University College London, London, WC1E 6BT, UK
| | - René Spijker
- Cochrane Netherlands, Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, University Utrecht, Utrecht, the Netherlands.,Medical Library, Amsterdam Public Health, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
| | - Mary S Wolfe
- US National Institute of Environmental Health Sciences, Research Triangle Park, Raleigh, NC, 27709, USA
| |
Collapse
|
49
|
Schmidt L, Olorisade BK, McGuinness LA, Thomas J, Higgins JPT. Data extraction methods for systematic review (semi)automation: A living review protocol. F1000Res 2020; 9:210. [PMID: 32724560 PMCID: PMC7338918 DOI: 10.12688/f1000research.22781.2] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 06/01/2020] [Indexed: 11/20/2022] Open
Abstract
Background: Researchers in evidence-based medicine cannot keep up with the amounts of both old and newly published primary research articles. Support for the early stages of the systematic review process - searching and screening studies for eligibility - is necessary because it is currently impossible to search for relevant research with precision. Better automated data extraction may not only facilitate the stage of review traditionally labelled 'data extraction', but also change earlier phases of the review process by making it possible to identify relevant research. Exponential improvements in computational processing speed and data storage are fostering the development of data mining models and algorithms. This, in combination with quicker pathways to publication, led to a large landscape of tools and methods for data mining and extraction. Objective: To review published methods and tools for data extraction to (semi)automate the systematic reviewing process. Methods: We propose to conduct a living review. With this methodology we aim to do constant evidence surveillance, bi-monthly search updates, as well as review updates every 6 months if new evidence permits it. In a cross-sectional analysis we will extract methodological characteristics and assess the quality of reporting in our included papers. Conclusions: We aim to increase transparency in the reporting and assessment of automation technologies to the benefit of data scientists, systematic reviewers and funders of health research. This living review will help to reduce duplicate efforts by data scientists who develop data mining methods. It will also serve to inform systematic reviewers about possibilities to support their data extraction.
Collapse
Affiliation(s)
- Lena Schmidt
- Bristol Medical School, University of Bristol, Bristol, BS8 2PS, UK
| | | | | | - James Thomas
- UCL Social Research Institute, University College London, London, WC1H 0AL, UK
| | | |
Collapse
|
50
|
Schmidt L, Olorisade BK, McGuinness LA, Thomas J, Higgins JPT. Data extraction methods for systematic review (semi)automation: A living review protocol. F1000Res 2020; 9:210. [PMID: 32724560 PMCID: PMC7338918 DOI: 10.12688/f1000research.22781.1] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 03/19/2020] [Indexed: 10/12/2023] Open
Abstract
Background: Researchers in evidence-based medicine cannot keep up with the amounts of both old and newly published primary research articles. Conducting and updating of systematic reviews is time-consuming. In practice, data extraction is one of the most complex tasks in this process. Exponential improvements in computational processing speed and data storage are fostering the development of data extraction models and algorithms. This, in combination with quicker pathways to publication, led to a large landscape of tools and methods for data extraction tasks. Objective: To review published methods and tools for data extraction to (semi)automate the systematic reviewing process. Methods: We propose to conduct a living review. With this methodology we aim to do monthly search updates, as well as bi-annual review updates if new evidence permits it. In a cross-sectional analysis we will extract methodological characteristics and assess the quality of reporting in our included papers. Conclusions: We aim to increase transparency in the reporting and assessment of machine learning technologies to the benefit of data scientists, systematic reviewers and funders of health research. This living review will help to reduce duplicate efforts by data scientists who develop data extraction methods. It will also serve to inform systematic reviewers about possibilities to support their data extraction.
Collapse
Affiliation(s)
- Lena Schmidt
- Bristol Medical School, University of Bristol, Bristol, BS8 2PS, UK
| | | | | | - James Thomas
- UCL Social Research Institute, University College London, London, WC1H 0AL, UK
| | | |
Collapse
|