1
|
Mansilla C, Wang Q, Piggott T, Bragge P, Waddell K, Guyatt G, Sweetman A, Lavis JN. A living critical interpretive synthesis to yield a framework on the production and dissemination of living evidence syntheses for decision-making. Implement Sci 2024; 19:67. [PMID: 39334425 PMCID: PMC11429155 DOI: 10.1186/s13012-024-01396-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/12/2024] [Accepted: 09/10/2024] [Indexed: 09/30/2024] Open
Abstract
BACKGROUND The COVID-19 pandemic has had an unprecedented impact in the global research production and has also increased research waste. Living evidence syntheses (LESs) seek to regularly update a body of evidence addressing a specific question. During the COVID-19 pandemic, the production and dissemination of LESs emerged as a cornerstone of the evidence infrastructure. This critical interpretive synthesis answers the questions: What constitutes an LES to support decision-making?; when should one be produced, updated, and discontinued?; and how should one be disseminated? METHODS Searches included the Cochrane Library, EMBASE (Ovid), Health Systems Evidence, MEDLINE (Ovid), PubMed, and Web of Science up to 23 April 2024 and included articles that provide any insights on addressing the compass questions on LESs. Articles were selected and appraised, and their insights extracted. An interpretive and iterative coding process was used to identify relevant thematic categories and create a conceptual framework. RESULTS Among the 16,630 non-duplicate records identified, 208 publications proved eligible. Most were non-empirical articles, followed by actual LESs. Approximately one in three articles were published in response to the COVID-19 pandemic. The conceptual framework addresses six thematic categories: (1) what is an LES; (2) what methodological approaches facilitate LESs production; (3) when to produce an LES; (4) when to update an LES; (5) how to make available the findings of an LES; and (6) when to discontinue LES updates. CONCLUSION LESs can play a critical role in reducing research waste and ensuring alignment with advisory and decision-making processes. This critical interpretive synthesis provides relevant insights on how to better organize the global evidence architecture to support their production. TRIAL REGISTRATION PROSPERO registration: CRD42021241875.
Collapse
Affiliation(s)
- Cristián Mansilla
- McMaster Health Forum, McMaster University, 1280 Main St W MML-417, Hamilton, ON, L8S 4L6, Canada.
- Health Policy PhD Program, McMaster University, 1280 Main St W 2C Area, Hamilton, ON, L8S 4K1, Canada.
| | - Qi Wang
- McMaster Health Forum, McMaster University, 1280 Main St W MML-417, Hamilton, ON, L8S 4L6, Canada
- Health Policy PhD Program, McMaster University, 1280 Main St W 2C Area, Hamilton, ON, L8S 4K1, Canada
| | - Thomas Piggott
- Department of Health Research Methods Evidence and Impact, McMaster University, 1280 Main St W 2C Area, Hamilton, ON, L8S 4K1, Canada
- Peterborough Public Health, 185 King Street, Peterborough, ON, K9J 2R8, Canada
- Department of Family Medicine, Queens University, 220 Bagot St, Kingston, ON, K7L 3G2, Canada
| | - Peter Bragge
- Monash Sustainable Development Institute Evidence Review Service, BehaviourWorks Australia, Monash University, Wellington Rd, Clayton VIC 3800, Melbourne, Australia
| | - Kerry Waddell
- McMaster Health Forum, McMaster University, 1280 Main St W MML-417, Hamilton, ON, L8S 4L6, Canada
- Health Policy PhD Program, McMaster University, 1280 Main St W 2C Area, Hamilton, ON, L8S 4K1, Canada
| | - Gordon Guyatt
- Department of Health Research Methods Evidence and Impact, McMaster University, 1280 Main St W 2C Area, Hamilton, ON, L8S 4K1, Canada
| | - Arthur Sweetman
- Health Policy PhD Program, McMaster University, 1280 Main St W 2C Area, Hamilton, ON, L8S 4K1, Canada
- Department of Economics, McMaster University, 1280 Main St W Kenneth Taylor Hall Rm. 129, Hamilton, ON, L8S 4M4, Canada
| | - John N Lavis
- McMaster Health Forum, McMaster University, 1280 Main St W MML-417, Hamilton, ON, L8S 4L6, Canada
- Department of Health Research Methods Evidence and Impact, McMaster University, 1280 Main St W 2C Area, Hamilton, ON, L8S 4K1, Canada
| |
Collapse
|
2
|
Yao X, Kumar MV, Su E, Flores Miranda A, Saha A, Sussman J. Evaluating the efficacy of artificial intelligence tools for the automation of systematic reviews in cancer research: A systematic review. Cancer Epidemiol 2024; 88:102511. [PMID: 38071872 DOI: 10.1016/j.canep.2023.102511] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/25/2023] [Revised: 11/23/2023] [Accepted: 11/29/2023] [Indexed: 01/27/2024]
Abstract
To evaluate the performance accuracy and workload savings of artificial intelligence (AI)-based automation tools in comparison with human reviewers in medical literature screening for systematic reviews (SR) of primary studies in cancer research in order to gain insights on improving the efficiency of producing SRs. Medline, Embase, the Cochrane Library, and PROSPERO databases were searched from inception to November 30, 2022. Then, forward and backward literature searches were completed, and the experts in this field including the authors of the articles included were contacted for a thorough grey literature search. This SR was registered on PROSPERO (CRD 42023384772). Among the 3947 studies obtained from search, five studies met the preplanned study selection criteria. These five studies evaluated four AI tools: Abstrackr (four studies), RobotAnalyst (one), EPPI-Reviewer (one), and DistillerSR (one). Without missing final included citations, Abstrackr eliminated 20%-88% of titles and abstracts (time saving of 7-86 hours) and 59% of the full-texts (62 h) from human review across four different cancer-related SRs. In comparison, RobotAnalyst (1% of titles and abstracts, 1 h), EPPI Review (38% of titles and abstracts, 58 h; 59% of full-texts, 62 h), DistillerSR (42% of titles and abstracts, 22 h) also provided similar or lower work savings for single cancer-related SRs. AI-based automation tools exhibited promising but varying levels of accuracy and efficiency during the screening process of medical literature for conducting SRs in the cancer field. Until further progress is made and thorough evaluations are conducted, AI tools should be utilized as supplementary aids rather than complete substitutes for human reviewers.
Collapse
Affiliation(s)
- Xiaomei Yao
- Department of Oncology, Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Center for Clinical Practice Guideline Conduction and Evaluation, Children's Hospital of Fudan University, Shanghai, China.
| | - Mithilesh V Kumar
- Faculty of Engineering, McMaster University, Hamilton, ON, Canada; Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
| | - Esther Su
- Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
| | | | - Ashirbani Saha
- Department of Oncology, McMaster University, Hamilton, Ontario, Canada; Escarpment Cancer Research Institute, Hamilton Health Sciences and McMaster University, Hamilton, ON, Canada
| | - Jonathan Sussman
- Department of Oncology, McMaster University, Hamilton, Ontario, Canada
| |
Collapse
|
3
|
Iannizzi C, Akl EA, Anslinger E, Weibel S, Kahale LA, Aminat AM, Piechotta V, Skoetz N. Methods and guidance on conducting, reporting, publishing, and appraising living systematic reviews: a scoping review. Syst Rev 2023; 12:238. [PMID: 38098023 PMCID: PMC10722674 DOI: 10.1186/s13643-023-02396-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/17/2022] [Accepted: 11/22/2023] [Indexed: 12/17/2023] Open
Abstract
BACKGROUND AND OBJECTIVE The living systematic review (LSR) approach is based on ongoing surveillance of the literature and continual updating. Most currently available guidance documents address the conduct, reporting, publishing, and appraisal of systematic reviews (SRs), but are not suitable for LSRs per se and miss additional LSR-specific considerations. In this scoping review, we aim to systematically collate methodological guidance literature on how to conduct, report, publish, and appraise the quality of LSRs and identify current gaps in guidance. METHODS A standard scoping review methodology was used. We searched MEDLINE (Ovid), EMBASE (Ovid), and The Cochrane Library on August 28, 2021. As for searching gray literature, we looked for existing guidelines and handbooks on LSRs from organizations that conduct evidence syntheses. The screening was conducted by two authors independently in Rayyan, and data extraction was done in duplicate using a pilot-tested data extraction form in Excel. Data was extracted according to four pre-defined categories for (i) conducting, (ii) reporting, (iii) publishing, and (iv) appraising LSRs. We mapped the findings by visualizing overview tables created in Microsoft Word. RESULTS Of the 21 included papers, methodological guidance was found in 17 papers for conducting, in six papers for reporting, in 15 papers for publishing, and in two papers for appraising LSRs. Some of the identified key items for (i) conducting LSRs were identifying the rationale, screening tools, or re-revaluating inclusion criteria. Identified items of (ii) the original PRISMA checklist included reporting the registration and protocol, title, or synthesis methods. For (iii) publishing, there was guidance available on publication type and frequency or update trigger, and for (iv) appraising, guidance on the appropriate use of bias assessment or reporting funding of included studies was found. Our search revealed major evidence gaps, particularly for guidance on certain PRISMA items such as reporting results, discussion, support and funding, and availability of data and material of a LSR. CONCLUSION Important evidence gaps were identified for guidance on how to report in LSRs and appraise their quality. Our findings were applied to inform and prepare a PRISMA 2020 extension for LSR.
Collapse
Affiliation(s)
- Claire Iannizzi
- Institute of Population Health, Faculty of Medicine and University Hospital Cologne, University of Cologne, Cologne, Germany.
| | - Elie A Akl
- Department of Medicine, American University of Beirut, Beirut, Lebanon
- Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Canada
| | - Eva Anslinger
- Evidence-Based Medicine, Department I of Internal Medicine, Center for Integrated Oncology Aachen Bonn Cologne Duesseldorf, Faculty of Medicine, University Hospital Cologne, University of Cologne, Kerpener Str. 62, 50937, Cologne, Germany
| | - Stephanie Weibel
- Department of Anaesthesiology, Intensive Care, Emergency and Pain Medicine, University Hospital Würzburg, Würzburg, Germany
| | - Lara A Kahale
- Editorial and Methods Department, Cochrane Central Executive, Cochrane, St Albans House, 57-59 Haymarket, London, SW1Y 4QX, UK
| | - Abina Mosunmola Aminat
- Rafic Hariri School of Nursing, American University of Beirut, Riad El Solh, P.O. Box 11-0236, Beirut, 1107 2020, Lebanon
| | - Vanessa Piechotta
- Evidence-Based Medicine, Department I of Internal Medicine, Center for Integrated Oncology Aachen Bonn Cologne Duesseldorf, Faculty of Medicine, University Hospital Cologne, University of Cologne, Kerpener Str. 62, 50937, Cologne, Germany
| | - Nicole Skoetz
- Institute of Population Health, Faculty of Medicine and University Hospital Cologne, University of Cologne, Cologne, Germany
| |
Collapse
|
4
|
Perlman-Arrow S, Loo N, Bobrovitz N, Yan T, Arora RK. A real-world evaluation of the implementation of NLP technology in abstract screening of a systematic review. Res Synth Methods 2023. [PMID: 37230483 DOI: 10.1002/jrsm.1636] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/14/2022] [Revised: 04/06/2023] [Accepted: 04/27/2023] [Indexed: 05/27/2023]
Abstract
The laborious and time-consuming nature of systematic review production hinders the dissemination of up-to-date evidence synthesis. Well-performing natural language processing (NLP) tools for systematic reviews have been developed, showing promise to improve efficiency. However, the feasibility and value of these technologies have not been comprehensively demonstrated in a real-world review. We developed an NLP-assisted abstract screening tool that provides text inclusion recommendations, keyword highlights, and visual context cues. We evaluated this tool in a living systematic review on SARS-CoV-2 seroprevalence, conducting a quality improvement assessment of screening with and without the tool. We evaluated changes to abstract screening speed, screening accuracy, characteristics of included texts, and user satisfaction. The tool improved efficiency, reducing screening time per abstract by 45.9% and decreasing inter-reviewer conflict rates. The tool conserved precision of article inclusion (positive predictive value; 0.92 with tool vs. 0.88 without) and recall (sensitivity; 0.90 vs. 0.81). The summary statistics of included studies were similar with and without the tool. Users were satisfied with the tool (mean satisfaction score of 4.2/5). We evaluated an abstract screening process where one human reviewer was replaced with the tool's votes, finding that this maintained recall (0.92 one-person, one-tool vs. 0.90 two tool-assisted humans) and precision (0.91 vs. 0.92) while reducing screening time by 70%. Implementing an NLP tool in this living systematic review improved efficiency, maintained accuracy, and was well-received by researchers, demonstrating the real-world effectiveness of NLP in expediting evidence synthesis.
Collapse
Affiliation(s)
- Sara Perlman-Arrow
- School of Population and Global Health, McGill University, Quebec, Canada
| | - Noel Loo
- Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA
| | - Niklas Bobrovitz
- Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada
| | - Tingting Yan
- Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada
| | - Rahul K Arora
- Centre for Health Informatics, University of Calgary, Calgary, Alberta, Canada
- Institute of Biomedical Engineering, University of Oxford, Oxford, UK
| |
Collapse
|
5
|
Oliveira Dos Santos Á, Sergio da Silva E, Machado Couto L, Valadares Labanca Reis G, Silva Belo V. The use of artificial intelligence for automating or semi-automating biomedical literature analyses: a scoping review. J Biomed Inform 2023; 142:104389. [PMID: 37187321 DOI: 10.1016/j.jbi.2023.104389] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2023] [Revised: 04/11/2023] [Accepted: 05/08/2023] [Indexed: 05/17/2023]
Abstract
OBJECTIVE Evidence-based medicine (EBM) is a decision-making process based on the conscious and judicious use of the best available scientific evidence. However, the exponential increase in the amount of information currently available likely exceeds the capacity of human-only analysis. In this context, artificial intelligence (AI) and its branches such as machine learning (ML) can be used to facilitate human efforts in analyzing the literature to foster EBM. The present scoping review aimed to examine the use of AI in the automation of biomedical literature survey and analysis with a view to establishing the state-of-the-art and identifying knowledge gaps. MATERIALS AND METHODS Comprehensive searches of the main databases were performed for articles published up to June 2022 and studies were selected according to inclusion and exclusion criteria. Data were extracted from the included articles and the findings categorized. RESULTS The total number of records retrieved from the databases was 12,145, of which 273 were included in the review. Classification of the studies according to the use of AI in evaluating the biomedical literature revealed three main application groups, namely assembly of scientific evidence (n=127; 47%), mining the biomedical literature (n=112; 41%) and quality analysis (n=34; 12%). Most studies addressed the preparation of systematic reviews, while articles focusing on the development of guidelines and evidence synthesis were the least frequent. The biggest knowledge gap was identified within the quality analysis group, particularly regarding methods and tools that assess the strength of recommendation and consistency of evidence. CONCLUSION Our review shows that, despite significant progress in the automation of biomedical literature surveys and analyses in recent years, intense research is needed to fill knowledge gaps on more difficult aspects of ML, deep learning and natural language processing, and to consolidate the use of automation by end-users (biomedical researchers and healthcare professionals).
Collapse
Affiliation(s)
| | - Eduardo Sergio da Silva
- Federal University of São João del-Rei, Campus Centro-Oeste Dona Lindu, Divinópolis, Minas Gerais, Brazil.
| | - Letícia Machado Couto
- Federal University of São João del-Rei, Campus Centro-Oeste Dona Lindu, Divinópolis, Minas Gerais, Brazil.
| | | | - Vinícius Silva Belo
- Federal University of São João del-Rei, Campus Centro-Oeste Dona Lindu, Divinópolis, Minas Gerais, Brazil.
| |
Collapse
|
6
|
Tian J, Gao Y, Zhang J, Yang Z, Dong S, Zhang T, Sun F, Wu S, Wu J, Wang J, Yao L, Ge L, Li L, Shi C, Wang Q, Li J, Zhao Y, Xiao Y, Yang F, Fan J, Bao S, Song F. Progress and challenges of network meta-analysis. J Evid Based Med 2021; 14:218-231. [PMID: 34463038 DOI: 10.1111/jebm.12443] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/28/2021] [Revised: 08/03/2021] [Accepted: 08/03/2021] [Indexed: 11/28/2022]
Abstract
In the past years, network meta-analysis (NMA) has been widely used among clinicians, guideline makers, and health technology assessment agencies and has played an important role in clinical decision-making and guideline development. To inform further development of NMAs, we conducted a bibliometric analysis to assess the current status of published NMA methodological studies, summarized the methodological progress of seven types of NMAs, and discussed the current challenges of NMAs.
Collapse
Affiliation(s)
- Jinhui Tian
- Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
- Key Laboratory of Evidence-Based Medicine and Knowledge Translation of Gansu Province, Lanzhou, China
| | - Ya Gao
- Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
- Key Laboratory of Evidence-Based Medicine and Knowledge Translation of Gansu Province, Lanzhou, China
| | - Junhua Zhang
- Evidence-Based Medicine Center, Tianjin University of Traditional Chinese Medicine, Tianjin, China
| | - Zhirong Yang
- Primary Care Unit, Department of Public Health and Primary Care, School of Clinical Medicine, University of Cambridge, Cambridge, UK
| | - Shengjie Dong
- Orthopedic Department, Yantaishan Hospital, Yantai, Shandong, China
| | - Tiansong Zhang
- Department of Traditional Chinese Medicine, Jing'an District Central Hospital, Shanghai, China
| | - Feng Sun
- Department of Epidemiology and Biostatistics, School of Public Health, Peking University, Beijing, China
| | - Shanshan Wu
- National Clinical Research Center of Digestive Diseases, Beijing Friendship Hospital, Capital Medical University, Beijing, China
| | - Jiarui Wu
- Department of Clinical Chinese Pharmacy, School of Chinese Materia Medica, Beijing University of Chinese Medicine, Beijing, China
| | - Junfeng Wang
- Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, Utrecht University, Utrecht, The Netherlands
| | - Liang Yao
- Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Canada
| | - Long Ge
- Key Laboratory of Evidence-Based Medicine and Knowledge Translation of Gansu Province, Lanzhou, China
- Evidence-Based Social Science Research Center, School of Public Health, Lanzhou University, Lanzhou, China
| | - Lun Li
- Department of Breast Cancer, The Second Xiangya Hospital, Central South University, Changsha, Hunan, China
| | - Chunhu Shi
- Division of Nursing, Midwifery and Social Work, School of Health Sciences, Faculty of Biology, Medicine and Health, University of Manchester, Manchester, UK
| | - Quan Wang
- Department of Gastrointestinal Surgery, Peking University People's Hospital, Beijing, China
| | - Jiang Li
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
| | - Ye Zhao
- First Clinical Medical College, Lanzhou University, Lanzhou, China
- Departments of Biochemistry and Molecular Biology, Melvin and Bren Simon Comprehensive Cancer Center, Indiana University School of Medicine, Indianapolis, Indiana
| | - Yue Xiao
- China National Health Development Research Center, Beijing, China
| | - Fengwen Yang
- Evidence-Based Medicine Center, Tianjin University of Traditional Chinese Medicine, Tianjin, China
| | - Jinchun Fan
- Epidemiology and Evidence Based-Medicine, School of Public Health, Gansu University of Chinese Medicine, Lanzhou, China
| | - Shisan Bao
- Epidemiology and Evidence Based-Medicine, School of Public Health, Gansu University of Chinese Medicine, Lanzhou, China
- Sydney, NSW, Australia
| | - Fujian Song
- Public Health and Health Services Research, Norwich Medical School, University of East Anglia, Norwich, UK
| |
Collapse
|
7
|
Natural language processing was effective in assisting rapid title and abstract screening when updating systematic reviews. J Clin Epidemiol 2021; 133:121-129. [PMID: 33485929 DOI: 10.1016/j.jclinepi.2021.01.010] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/27/2020] [Revised: 01/06/2021] [Accepted: 01/14/2021] [Indexed: 02/05/2023]
Abstract
BACKGROUND AND OBJECTIVE To examine whether the use of natural language processing (NLP) technology is effective in assisting rapid title and abstract screening when updating a systematic review. STUDY DESIGN Using the searched literature from a published systematic review, we trained and tested an NLP model that enables rapid title and abstract screening when updating a systematic review. The model was a light gradient boosting machine (LightGBM), an ensemble learning classifier which integrates four pretrained Bidirectional Encoder Representations from Transformers (BERT) models. We divided the searched citations into two sets (ie, training and test sets). The model was trained using the training set and assessed for screening performance using the test set. The searched citations, whose eligibility was determined by two independent reviewers, were treated as the reference standard. RESULTS The test set included 947 citations; our model included 340 citations, excluded 607 citations, and achieved 96% sensitivity, and 78% specificity. If the classifier assessment in the case study was accepted, reviewers would lose 8 of 180 eligible citations (4%), none of which were ultimately included in the systematic review after full-text consideration, while decreasing the workload by 64.1%. CONCLUSION NLP technology using the ensemble learning method may effectively assist in rapid literature screening when updating systematic reviews.
Collapse
|
8
|
Bashir R, Dunn AG, Surian D. A rule-based approach for automatically extracting data from systematic reviews and their updates to model the risk of conclusion change. Res Synth Methods 2021; 12:216-225. [PMID: 33350584 DOI: 10.1002/jrsm.1473] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2019] [Revised: 10/31/2020] [Accepted: 12/15/2020] [Indexed: 11/09/2022]
Abstract
Few data-driven approaches are available to estimate the risk of conclusion change in systematic review updates. We developed a rule-based approach to automatically extract information from reviews and updates to be used as features for modelling conclusion change risk. Rules were developed to extract relevant information from published Cochrane reviews and used to construct four features: the number of included trials and participants in the reviews, a measure based on the number of participants, and the time elapsed between the search dates. We compared the performance of random forest, decision tree, and logistic regression to predict the conclusion change risk. The performance was measured by accuracy, precision, recall, F1 -score, and area under ROC (AU-ROC). One rule was developed to extract the conclusion change information (96% accuracy, 100 reviews), one for the search date (100% accuracy, 100 reviews), one for the number of included clinical trials (100% accuracy, 100 reviews), and 22 for the number of participants (97.3% accuracy, 200 reviews). For unseen reviews, the random forest classifier showed the highest accuracy (80.8%) and AU-ROC (0.80). All classifiers showed relatively similar performance with overlapping 95% confidence interval (CI). The coverage score was shown to be the most useful feature for predicting the conclusion change risk. Features mined from Cochrane reviews and updates can estimate conclusion change risk. If data from more published reviews and updates were made accessible, data-driven methods to predict the conclusion change risk may be a feasible way to support decisions about updating reviews.
Collapse
Affiliation(s)
- Rabia Bashir
- Centre for Health Informatics, Australian Institute of Health Innovation, Faculty of Medicine, Health and Human Sciences, Macquarie University, Sydney, New South Wales, Australia
| | - Adam G Dunn
- Centre for Health Informatics, Australian Institute of Health Innovation, Faculty of Medicine, Health and Human Sciences, Macquarie University, Sydney, New South Wales, Australia.,Discipline of Biomedical Informatics and Digital Health, School of Medical Sciences, Faculty of Medicine and Health, The University of Sydney, Sydney, New South Wales, Australia
| | - Didi Surian
- Centre for Health Informatics, Australian Institute of Health Innovation, Faculty of Medicine, Health and Human Sciences, Macquarie University, Sydney, New South Wales, Australia
| |
Collapse
|
9
|
Créquit P, Boutron I, Meerpohl J, Williams HC, Craig J, Ravaud P. Future of evidence ecosystem series: 2. current opportunities and need for better tools and methods. J Clin Epidemiol 2020; 123:143-152. [DOI: 10.1016/j.jclinepi.2020.01.023] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/02/2019] [Revised: 12/26/2019] [Accepted: 01/07/2020] [Indexed: 02/06/2023]
|
10
|
Deng Z, Yin K, Bao Y, Armengol VD, Wang C, Tiwari A, Barzilay R, Parmigiani G, Braun D, Hughes KS. Validation of a Semiautomated Natural Language Processing-Based Procedure for Meta-Analysis of Cancer Susceptibility Gene Penetrance. JCO Clin Cancer Inform 2020; 3:1-9. [PMID: 31419182 DOI: 10.1200/cci.19.00043] [Citation(s) in RCA: 18] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022] Open
Abstract
PURPOSE Quantifying the risk of cancer associated with pathogenic mutations in germline cancer susceptibility genes-that is, penetrance-enables the personalization of preventive management strategies. Conducting a meta-analysis is the best way to obtain robust risk estimates. We have previously developed a natural language processing (NLP) -based abstract classifier which classifies abstracts as relevant to penetrance, prevalence of mutations, both, or neither. In this work, we evaluate the performance of this NLP-based procedure. MATERIALS AND METHODS We compared the semiautomated NLP-based procedure, which involves automated abstract classification and text mining, followed by human review of identified studies, with the traditional procedure that requires human review of all studies. Ten high-quality gene-cancer penetrance meta-analyses spanning 16 gene-cancer associations were used as the gold standard by which to evaluate the performance of our procedure. For each meta-analysis, we evaluated the number of abstracts that required human review (workload) and the ability to identify the studies that were included by the authors in their quantitative analysis (coverage). RESULTS Compared with the traditional procedure, the semiautomated NLP-based procedure led to a lower workload across all 10 meta-analyses, with an overall 84% reduction (2,774 abstracts v 16,941 abstracts) in the amount of human review required. Overall coverage was 93%-we are able to identify 132 of 142 studies-before reviewing references of identified studies. Reasons for the 10 missed studies included blank and poorly written abstracts. After reviewing references, nine of the previously missed studies were identified and coverage improved to 99% (141 of 142 studies). CONCLUSION We demonstrated that an NLP-based procedure can significantly reduce the review workload without compromising the ability to identify relevant studies. NLP algorithms have promising potential for reducing human efforts in the literature review process.
Collapse
Affiliation(s)
| | - Kanhua Yin
- Massachusetts General Hospital, Boston, MA
| | - Yujia Bao
- Massachusetts Institute of Technology, Boston, MA
| | | | - Cathy Wang
- Harvard TH Chan School of Public Health, Boston, MA.,Dana-Farber Cancer Institute, Boston, MA
| | | | | | - Giovanni Parmigiani
- Harvard TH Chan School of Public Health, Boston, MA.,Dana-Farber Cancer Institute, Boston, MA
| | - Danielle Braun
- Harvard TH Chan School of Public Health, Boston, MA.,Dana-Farber Cancer Institute, Boston, MA
| | - Kevin S Hughes
- Massachusetts General Hospital, Boston, MA.,Harvard Medical School, Boston, MA
| |
Collapse
|
11
|
Baclic O, Tunis M, Young K, Doan C, Swerdfeger H, Schonfeld J. Challenges and opportunities for public health made possible by advances in natural language processing. CANADA COMMUNICABLE DISEASE REPORT = RELEVE DES MALADIES TRANSMISSIBLES AU CANADA 2020; 46:161-168. [PMID: 32673380 PMCID: PMC7343054 DOI: 10.14745/ccdr.v46i06a02] [Citation(s) in RCA: 39] [Impact Index Per Article: 9.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
Abstract
Natural language processing (NLP) is a subfield of artificial intelligence devoted to understanding and generation of language. The recent advances in NLP technologies are enabling rapid analysis of vast amounts of text, thereby creating opportunities for health research and evidence-informed decision making. The analysis and data extraction from scientific literature, technical reports, health records, social media, surveys, registries and other documents can support core public health functions including the enhancement of existing surveillance systems (e.g. through faster identification of diseases and risk factors/at-risk populations), disease prevention strategies (e.g. through more efficient evaluation of the safety and effectiveness of interventions) and health promotion efforts (e.g. by providing the ability to obtain expert-level answers to any health related question). NLP is emerging as an important tool that can assist public health authorities in decreasing the burden of health inequality/inequity in the population. The purpose of this paper is to provide some notable examples of both the potential applications and challenges of NLP use in public health.
Collapse
Affiliation(s)
- Oliver Baclic
- Centre for Immunization and Respiratory Infectious Disease, Public Health Agency of Canada, Ottawa, ON
| | - Matthew Tunis
- Centre for Immunization and Respiratory Infectious Disease, Public Health Agency of Canada, Ottawa, ON
| | - Kelsey Young
- Centre for Immunization and Respiratory Infectious Disease, Public Health Agency of Canada, Ottawa, ON
| | - Coraline Doan
- Data, Partnerships and Innovation Hub, Public Health Agency of Canada, Ottawa, ON
| | - Howard Swerdfeger
- Data, Partnerships and Innovation Hub, Public Health Agency of Canada, Ottawa, ON
| | - Justin Schonfeld
- National Microbiology Laboratory, Public Health Agency of Canada, Winnipeg, MB
| |
Collapse
|
12
|
Lee EW, Wallace BC, Galaviz KI, Ho JC. MMiDaS-AE: Multi-modal Missing Data aware Stacked Autoencoder for Biomedical Abstract Screening. PROCEEDINGS OF THE ACM CONFERENCE ON HEALTH, INFERENCE, AND LEARNING 2020; 2020:139-150. [PMID: 34308444 PMCID: PMC8297409 DOI: 10.1145/3368555.3384463] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/24/2022]
Abstract
Systematic review (SR) is an essential process to identify, evaluate, and summarize the findings of all relevant individual studies concerning health-related questions. However, conducting a SR is labor-intensive, as identifying relevant studies is a daunting process that entails multiple researchers screening thousands of articles for relevance. In this paper, we propose MMiDaS-AE, a Multi-modal Missing Data aware Stacked Autoencoder, for semi-automating screening for SRs. We use a multi-modal view that exploits three representations, of: 1) documents, 2) topics, and 3) citation networks. Documents that contain similar words will be nearby in the document embedding space. Models can also exploit the relationship between documents and the associated SR MeSH terms to capture article relevancy. Finally, related works will likely share the same citations, and thus closely related articles would, intuitively, be trained to be close to each other in the embedding space. However, using all three learned representations as features directly result in an unwieldy number of parameters. Thus, motivated by recent work on multi-modal auto-encoders, we adopt a multi-modal stacked autoencoder that can learn a shared representation encoding all three representations in a compressed space. However, in practice one or more of these modalities may be missing for an article (e.g., if we cannot recover citation information). Therefore, we propose to learn to impute the shared representation even when specific inputs are missing. We find this new model significantly improves performance on a dataset consisting of 15 SRs compared to existing approaches.
Collapse
|
13
|
Koroleva A, Olarte Parra C, Paroubek P. On improving the implementation of automatic updating of systematic reviews. JAMIA Open 2019; 2:400-401. [PMID: 32025633 PMCID: PMC6993991 DOI: 10.1093/jamiaopen/ooz044] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/09/2019] [Accepted: 09/10/2019] [Indexed: 11/14/2022] Open
Affiliation(s)
- Anna Koroleva
- LIMSI, CNRS, Université Paris-Saclay, Orsay, France
- Academic Medical Center, University of Amsterdam, Amsterdam, the Netherlands
| | - Camila Olarte Parra
- Academic Medical Center, University of Amsterdam, Amsterdam, the Netherlands
- Department of Applied Mathematics, Computer Science and Statistics, Ghent University, Ghent, Belgium
| | | |
Collapse
|
14
|
Norman CR, Leeflang MMG, Porcher R, Névéol A. Measuring the impact of screening automation on meta-analyses of diagnostic test accuracy. Syst Rev 2019; 8:243. [PMID: 31661028 PMCID: PMC6819363 DOI: 10.1186/s13643-019-1162-x] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/22/2019] [Accepted: 09/13/2019] [Indexed: 11/25/2022] Open
Abstract
BACKGROUND The large and increasing number of new studies published each year is making literature identification in systematic reviews ever more time-consuming and costly. Technological assistance has been suggested as an alternative to the conventional, manual study identification to mitigate the cost, but previous literature has mainly evaluated methods in terms of recall (search sensitivity) and workload reduction. There is a need to also evaluate whether screening prioritization methods leads to the same results and conclusions as exhaustive manual screening. In this study, we examined the impact of one screening prioritization method based on active learning on sensitivity and specificity estimates in systematic reviews of diagnostic test accuracy. METHODS We simulated the screening process in 48 Cochrane reviews of diagnostic test accuracy and re-run 400 meta-analyses based on a least 3 studies. We compared screening prioritization (with technological assistance) and screening in randomized order (standard practice without technology assistance). We examined if the screening could have been stopped before identifying all relevant studies while still producing reliable summary estimates. For all meta-analyses, we also examined the relationship between the number of relevant studies and the reliability of the final estimates. RESULTS The main meta-analysis in each systematic review could have been performed after screening an average of 30% of the candidate articles (range 0.07 to 100%). No systematic review would have required screening more than 2308 studies, whereas manual screening would have required screening up to 43,363 studies. Despite an average 70% recall, the estimation error would have been 1.3% on average, compared to an average 2% estimation error expected when replicating summary estimate calculations. CONCLUSION Screening prioritization coupled with stopping criteria in diagnostic test accuracy reviews can reliably detect when the screening process has identified a sufficient number of studies to perform the main meta-analysis with an accuracy within pre-specified tolerance limits. However, many of the systematic reviews did not identify a sufficient number of studies that the meta-analyses were accurate within a 2% limit even with exhaustive manual screening, i.e., using current practice.
Collapse
Affiliation(s)
- Christopher R. Norman
- LIMSI, CNRS, Université Paris Saclay, Rue du Belvedère, Orsay, 91405 France
- Amsterdam Public Health, Amsterdam UMC, University of Amsterdam, Meibergdreef 9, Amsterdam, 1105 AZ the Netherlands
| | - Mariska M. G. Leeflang
- Amsterdam Public Health, Amsterdam UMC, University of Amsterdam, Meibergdreef 9, Amsterdam, 1105 AZ the Netherlands
| | - Raphaël Porcher
- Center for Clinical Epidemiology, Assistance Publique–Hôpitaux de Paris, Hôtel Dieu Hospital; Team METHODS, CRESS, INSERM U1153; University Paris Descartes, 1 place du Parvis Notre-Dame, Paris, 75004 France
| | - Aurélie Névéol
- LIMSI, CNRS, Université Paris Saclay, Rue du Belvedère, Orsay, 91405 France
| |
Collapse
|
15
|
Norman CR, Gargon E, Leeflang MMG, Névéol A, Williamson PR. Evaluation of an automatic article selection method for timelier updates of the Comet Core Outcome Set database. Database (Oxford) 2019; 2019:baz109. [PMID: 31697361 PMCID: PMC6836711 DOI: 10.1093/database/baz109] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2019] [Revised: 08/07/2019] [Accepted: 08/07/2019] [Indexed: 01/07/2023]
Abstract
Curated databases of scientific literature play an important role in helping researchers find relevant literature, but populating such databases is a labour intensive and time-consuming process. One such database is the freely accessible Comet Core Outcome Set database, which was originally populated using manual screening in an annually updated systematic review. In order to reduce the workload and facilitate more timely updates we are evaluating machine learning methods to reduce the number of references needed to screen. In this study we have evaluated a machine learning approach based on logistic regression to automatically rank the candidate articles. Data from the original systematic review and its four first review updates were used to train the model and evaluate performance. We estimated that using automatic screening would yield a workload reduction of at least 75% while keeping the number of missed references around 2%. We judged this to be an acceptable trade-off for this systematic review, and the method is now being used for the next round of the Comet database update.
Collapse
Affiliation(s)
- Christopher R Norman
- LIMSI, CNRS, Université Paris-Saclay, Bât 507, rue du Belvédère, Campus Universitaire, F-91405 Orsay
| | - Elizabeth Gargon
- MRC NWHMTR, Department of Biostatistics, University of Liverpool, Liverpool, UK
| | - Mariska M G Leeflang
- Amsterdam Public Health, Amsterdam Umc, University of Amsterdam, Meibergdreef 9, 1105 az, Amsterdam, the Netherlands
| | - Aurélie Névéol
- LIMSI, CNRS, Université Paris-Saclay, Bât 507, rue du Belvédère, Campus Universitaire, F-91405 Orsay
| | - Paula R Williamson
- MRC NWHMTR, Department of Biostatistics, University of Liverpool, Liverpool, UK
| |
Collapse
|