1
|
Harmsen W, de Groot J, Harkema A, van Dusseldorp I, de Bruin J, van den Brand S, van de Schoot R. Machine learning to optimize literature screening in medical guideline development. Syst Rev 2024; 13:177. [PMID: 38992684 PMCID: PMC11238391 DOI: 10.1186/s13643-024-02590-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/21/2022] [Accepted: 06/20/2024] [Indexed: 07/13/2024] Open
Abstract
OBJECTIVES In a time of exponential growth of new evidence supporting clinical decision-making, combined with a labor-intensive process of selecting this evidence, methods are needed to speed up current processes to keep medical guidelines up-to-date. This study evaluated the performance and feasibility of active learning to support the selection of relevant publications within medical guideline development and to study the role of noisy labels. DESIGN We used a mixed-methods design. Two independent clinicians' manual process of literature selection was evaluated for 14 searches. This was followed by a series of simulations investigating the performance of random reading versus using screening prioritization based on active learning. We identified hard-to-find papers and checked the labels in a reflective dialogue. MAIN OUTCOME MEASURES Inter-rater reliability was assessed using Cohen's Kappa (ĸ). To evaluate the performance of active learning, we used the Work Saved over Sampling at 95% recall (WSS@95) and percentage Relevant Records Found at reading only 10% of the total number of records (RRF@10). We used the average time to discovery (ATD) to detect records with potentially noisy labels. Finally, the accuracy of labeling was discussed in a reflective dialogue with guideline developers. RESULTS Mean ĸ for manual title-abstract selection by clinicians was 0.50 and varied between - 0.01 and 0.87 based on 5.021 abstracts. WSS@95 ranged from 50.15% (SD = 17.7) based on selection by clinicians to 69.24% (SD = 11.5) based on the selection by research methodologist up to 75.76% (SD = 12.2) based on the final full-text inclusion. A similar pattern was seen for RRF@10, ranging from 48.31% (SD = 23.3) to 62.8% (SD = 21.20) and 65.58% (SD = 23.25). The performance of active learning deteriorates with higher noise. Compared with the final full-text selection, the selection made by clinicians or research methodologists deteriorated WSS@95 by 25.61% and 6.25%, respectively. CONCLUSION While active machine learning tools can accelerate the process of literature screening within guideline development, they can only work as well as the input given by human raters. Noisy labels make noisy machine learning.
Collapse
Affiliation(s)
- Wouter Harmsen
- Knowlegde Institute for the Federation of Medical Specialists, Utrecht, The Netherlands
| | - Janke de Groot
- Knowlegde Institute for the Federation of Medical Specialists, Utrecht, The Netherlands
| | - Albert Harkema
- Department of Methodology and Statistics, Faculty of Social and Behavioral Sciences, Utrecht University, Utrecht, The Netherlands
| | | | - Jonathan de Bruin
- Department of Research and Data Management Services, Information Technology Services, Utrecht University, Utrecht, the Netherlands
| | - Sofie van den Brand
- Department of Methodology and Statistics, Faculty of Social and Behavioral Sciences, Utrecht University, Utrecht, The Netherlands
| | - Rens van de Schoot
- Department of Methodology and Statistics, Faculty of Social and Behavioral Sciences, Utrecht University, Utrecht, The Netherlands.
| |
Collapse
|
2
|
Luo X, Chen F, Zhu D, Wang L, Wang Z, Liu H, Lyu M, Wang Y, Wang Q, Chen Y. Potential Roles of Large Language Models in the Production of Systematic Reviews and Meta-Analyses. J Med Internet Res 2024; 26:e56780. [PMID: 38819655 PMCID: PMC11234072 DOI: 10.2196/56780] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/20/2024] [Revised: 05/21/2024] [Accepted: 05/29/2024] [Indexed: 06/01/2024] Open
Abstract
Large language models (LLMs) such as ChatGPT have become widely applied in the field of medical research. In the process of conducting systematic reviews, similar tools can be used to expedite various steps, including defining clinical questions, performing the literature search, document screening, information extraction, and language refinement, thereby conserving resources and enhancing efficiency. However, when using LLMs, attention should be paid to transparent reporting, distinguishing between genuine and false content, and avoiding academic misconduct. In this viewpoint, we highlight the potential roles of LLMs in the creation of systematic reviews and meta-analyses, elucidating their advantages, limitations, and future research directions, aiming to provide insights and guidance for authors planning systematic reviews and meta-analyses.
Collapse
Affiliation(s)
- Xufei Luo
- Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
- World Health Organization Collaboration Center for Guideline Implementation and Knowledge Translation, Lanzhou, China
- Institute of Health Data Science, Lanzhou University, Lanzhou, China
- Key Laboratory of Evidence Based Medicine and Knowledge Translation of Gansu Province, Lanzhou University, Lanzhou, China
- Research Unit of Evidence-Based Evaluation and Guidelines, Chinese Academy of Medical Sciences (2021RU017), School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
| | - Fengxian Chen
- School of Information Science & Engineering, Lanzhou University, Lanzhou, China
| | - Di Zhu
- School of Public Health, Lanzhou University, Lanzhou, China
| | - Ling Wang
- School of Public Health, Lanzhou University, Lanzhou, China
| | - Zijun Wang
- Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
- World Health Organization Collaboration Center for Guideline Implementation and Knowledge Translation, Lanzhou, China
- Institute of Health Data Science, Lanzhou University, Lanzhou, China
- Key Laboratory of Evidence Based Medicine and Knowledge Translation of Gansu Province, Lanzhou University, Lanzhou, China
- Research Unit of Evidence-Based Evaluation and Guidelines, Chinese Academy of Medical Sciences (2021RU017), School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
| | - Hui Liu
- Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
- World Health Organization Collaboration Center for Guideline Implementation and Knowledge Translation, Lanzhou, China
- Institute of Health Data Science, Lanzhou University, Lanzhou, China
- Key Laboratory of Evidence Based Medicine and Knowledge Translation of Gansu Province, Lanzhou University, Lanzhou, China
- Research Unit of Evidence-Based Evaluation and Guidelines, Chinese Academy of Medical Sciences (2021RU017), School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
| | - Meng Lyu
- School of Public Health, Lanzhou University, Lanzhou, China
| | - Ye Wang
- School of Public Health, Lanzhou University, Lanzhou, China
| | - Qi Wang
- Department of Health Research Methods, Evidence and Impact, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
- McMaster Health Forum, McMaster University, Hamilton, ON, Canada
| | - Yaolong Chen
- Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
- World Health Organization Collaboration Center for Guideline Implementation and Knowledge Translation, Lanzhou, China
- Institute of Health Data Science, Lanzhou University, Lanzhou, China
- Key Laboratory of Evidence Based Medicine and Knowledge Translation of Gansu Province, Lanzhou University, Lanzhou, China
- Research Unit of Evidence-Based Evaluation and Guidelines, Chinese Academy of Medical Sciences (2021RU017), School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
| |
Collapse
|
3
|
Khalil H, Pollock D, McInerney P, Evans C, Moraes EB, Godfrey CM, Alexander L, Tricco A, Peters MDJ, Pieper D, Saran A, Ameen D, Taneri PE, Munn Z. Automation tools to support undertaking scoping reviews. Res Synth Methods 2024. [PMID: 38885942 DOI: 10.1002/jrsm.1731] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2023] [Revised: 05/15/2024] [Accepted: 06/02/2024] [Indexed: 06/20/2024]
Abstract
OBJECTIVE This paper describes several automation tools and software that can be considered during evidence synthesis projects and provides guidance for their integration in the conduct of scoping reviews. STUDY DESIGN AND SETTING The guidance presented in this work is adapted from the results of a scoping review and consultations with the JBI Scoping Review Methodology group. RESULTS This paper describes several reliable, validated automation tools and software that can be used to enhance the conduct of scoping reviews. Developments in the automation of systematic reviews, and more recently scoping reviews, are continuously evolving. We detail several helpful tools in order of the key steps recommended by the JBI's methodological guidance for undertaking scoping reviews including team establishment, protocol development, searching, de-duplication, screening titles and abstracts, data extraction, data charting, and report writing. While we include several reliable tools and software that can be used for the automation of scoping reviews, there are some limitations to the tools mentioned. For example, some are available in English only and their lack of integration with other tools results in limited interoperability. CONCLUSION This paper highlighted several useful automation tools and software programs to use in undertaking each step of a scoping review. This guidance has the potential to inform collaborative efforts aiming at the development of evidence informed, integrated automation tools and software packages for enhancing the conduct of high-quality scoping reviews.
Collapse
Affiliation(s)
- Hanan Khalil
- School of Psychology and Public Health, Department of Public Health, La Trobe University, Melbourne, Australia
- The Queensland Centre of Evidence Based Nursing and Midwifery: A JBI Centre of Excellence, Brisbane, Queensland, Australia
| | - Danielle Pollock
- JBI, University of Adelaide, Adelaide, Australia
- Health Evidence Synthesis, Recommendations and Impact (HESRI), School of Public Health, University of Adelaide, Adelaide, Australia
| | - Patricia McInerney
- The Wits JBI Centre for Evidence-Based Practice: A JBI Centre of Excellence, Faculty of Health Sciences, University of the Witwatersrand, South Africa
| | - Catrin Evans
- The Nottingham Centre for Evidence Based Healthcare: A JBI Centre of Excellence, University of Nottingham, UK
| | - Erica B Moraes
- Nursing School, Department of Nursing Fundamentals and Administration, Federal Fluminense University, Rio de Janeiro, Brazil
- The Brazilian Centre of Evidence-based Healthcare: A JBI Centre of Excellence - JBI, Brazil
| | - Christina M Godfrey
- Queen's Collaboration for Health Care Quality: A JBI Centre of Excellence, Queen's University School of Nursing, Kingston, Ontario, Canada
| | - Lyndsay Alexander
- The Scottish Centre for Evidence-based, Multi-Professional Practice: A JBI Centre of Excellence, Aberdeen, UK
- School of Health Sciences, Robert Gordon University, Aberdeen, UK
| | - Andrea Tricco
- Queen's Collaboration for Health Care Quality: A JBI Centre of Excellence, Queen's University School of Nursing, Kingston, Ontario, Canada
- Epidemiology Division and Institute for Health, Management, and Evaluation, Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, Toronto, Ontario, Canada
| | - Micah D J Peters
- Health Evidence Synthesis, Recommendations and Impact (HESRI), School of Public Health, University of Adelaide, Adelaide, Australia
- University of South Australia, Clinical and Health Sciences, Rosemary Bryant AO Research Centre, Adelaide, South Australia, Australia
- University of Adelaide, Faculty of Health and Medical Sciences, Adelaide Nursing School, Adelaide, Australia
| | - Dawid Pieper
- Faculty of Health Sciences Brandenburg, Brandenburg Medical School (Theodor Fontane), Institute for Health Services and Health System Research, Rüdersdorf, Germany
- Center for Health Services Research, Brandenburg Medical School (Theodor Fontane), Rüdersdorf, Germany
| | | | - Daniel Ameen
- Faculty of Medicine, Nursing and Health Sciences, School of Medicine, Monash University, Australia
| | - Petek Eylul Taneri
- HRB-Trials Methodology Research Network, College of Medicine, Nursing and Health Sciences, University of Galway, Galway, Ireland
| | - Zachary Munn
- JBI, University of Adelaide, Adelaide, Australia
- Health Evidence Synthesis, Recommendations and Impact (HESRI), School of Public Health, University of Adelaide, Adelaide, Australia
| |
Collapse
|
4
|
Dennstädt F, Zink J, Putora PM, Hastings J, Cihoric N. Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain. Syst Rev 2024; 13:158. [PMID: 38879534 PMCID: PMC11180407 DOI: 10.1186/s13643-024-02575-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/17/2023] [Accepted: 05/30/2024] [Indexed: 06/19/2024] Open
Abstract
BACKGROUND Systematically screening published literature to determine the relevant publications to synthesize in a review is a time-consuming and difficult task. Large language models (LLMs) are an emerging technology with promising capabilities for the automation of language-related tasks that may be useful for such a purpose. METHODS LLMs were used as part of an automated system to evaluate the relevance of publications to a certain topic based on defined criteria and based on the title and abstract of each publication. A Python script was created to generate structured prompts consisting of text strings for instruction, title, abstract, and relevant criteria to be provided to an LLM. The relevance of a publication was evaluated by the LLM on a Likert scale (low relevance to high relevance). By specifying a threshold, different classifiers for inclusion/exclusion of publications could then be defined. The approach was used with four different openly available LLMs on ten published data sets of biomedical literature reviews and on a newly human-created data set for a hypothetical new systematic literature review. RESULTS The performance of the classifiers varied depending on the LLM being used and on the data set analyzed. Regarding sensitivity/specificity, the classifiers yielded 94.48%/31.78% for the FlanT5 model, 97.58%/19.12% for the OpenHermes-NeuralChat model, 81.93%/75.19% for the Mixtral model and 97.58%/38.34% for the Platypus 2 model on the ten published data sets. The same classifiers yielded 100% sensitivity at a specificity of 12.58%, 4.54%, 62.47%, and 24.74% on the newly created data set. Changing the standard settings of the approach (minor adaption of instruction prompt and/or changing the range of the Likert scale from 1-5 to 1-10) had a considerable impact on the performance. CONCLUSIONS LLMs can be used to evaluate the relevance of scientific publications to a certain review topic and classifiers based on such an approach show some promising results. To date, little is known about how well such systems would perform if used prospectively when conducting systematic literature reviews and what further implications this might have. However, it is likely that in the future researchers will increasingly use LLMs for evaluating and classifying scientific publications.
Collapse
Affiliation(s)
- Fabio Dennstädt
- Department of Radiation Oncology, Cantonal Hospital of St. Gallen, St. Gallen, Switzerland.
- Department of Radiation Oncology, Inselspital, Bern University Hospital and University of Bern, Bern, Switzerland.
| | - Johannes Zink
- Institute for Computer Science, University of Würzburg, Würzburg, Germany
| | - Paul Martin Putora
- Department of Radiation Oncology, Cantonal Hospital of St. Gallen, St. Gallen, Switzerland
- Department of Radiation Oncology, Inselspital, Bern University Hospital and University of Bern, Bern, Switzerland
| | - Janna Hastings
- Institute for Implementation Science in Health Care, University of Zurich, Zurich, Switzerland
- School of Medicine, University of St. Gallen, St. Gallen, Switzerland
- Swiss Institute of Bioinformatics, Lausanne, Switzerland
| | - Nikola Cihoric
- Department of Radiation Oncology, Inselspital, Bern University Hospital and University of Bern, Bern, Switzerland
| |
Collapse
|
5
|
Guo Q, Jiang G, Zhao Q, Long Y, Feng K, Gu X, Xu Y, Li Z, Huang J, Du L. Rapid review: A review of methods and recommendations based on current evidence. J Evid Based Med 2024; 17:434-453. [PMID: 38512942 DOI: 10.1111/jebm.12594] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/26/2023] [Accepted: 02/28/2024] [Indexed: 03/23/2024]
Abstract
Rapid review (RR) could accelerate the traditional systematic review (SR) process by simplifying or omitting steps using various shortcuts. With the increasing popularity of RR, numerous shortcuts had emerged, but there was no consensus on how to choose the most appropriate ones. This study conducted a literature search in PubMed from inception to December 21, 2023, using terms such as "rapid review" "rapid assessment" "rapid systematic review" and "rapid evaluation". We also scanned the reference lists and performed citation tracking of included impact studies to obtain more included studies. We conducted a narrative synthesis of all RR approaches, shortcuts and studies assessing their effectiveness at each stage of RRs. Based on the current evidence, we provided recommendations on utilizing certain shortcuts in RRs. Ultimately, we identified 185 studies focusing on summarizing RR approaches and shortcuts, or evaluating their impact. There was relatively sufficient evidence to support the use of the following shortcuts in RRs: limiting studies to those published in English-language; conducting abbreviated database searches (e.g., only searching PubMed/MEDLINE, Embase, and CENTRAL); omitting retrieval of grey literature; restricting the search timeframe to the recent 20 years for medical intervention and the recent 15 years for reviewing diagnostic test accuracy; conducting a single screening by an experienced screener. To some extent, the above shortcuts were also applicable to SRs. This study provided a reference for future RR researchers in selecting shortcuts, and it also presented a potential research topic for methodologists.
Collapse
Affiliation(s)
- Qiong Guo
- Innovation Institute for Integration of Medicine and Engineering, West China Hospital, Sichuan University, Chengdu, P. R. China
- West China Medical Publishers, West China Hospital, Sichuan University, Chengdu, P. R. China
| | - Guiyu Jiang
- West China School of Public Health, Sichuan University, Chengdu, P. R. China
| | - Qingwen Zhao
- West China School of Public Health, Sichuan University, Chengdu, P. R. China
| | - Youlin Long
- Innovation Institute for Integration of Medicine and Engineering, West China Hospital, Sichuan University, Chengdu, P. R. China
- Chinese Evidence-Based Medicine Center, West China Hospital, Sichuan University, Chengdu, P. R. China
| | - Kun Feng
- Innovation Institute for Integration of Medicine and Engineering, West China Hospital, Sichuan University, Chengdu, P. R. China
- Chinese Evidence-Based Medicine Center, West China Hospital, Sichuan University, Chengdu, P. R. China
| | - Xianlin Gu
- Innovation Institute for Integration of Medicine and Engineering, West China Hospital, Sichuan University, Chengdu, P. R. China
- Chinese Evidence-Based Medicine Center, West China Hospital, Sichuan University, Chengdu, P. R. China
| | - Yihan Xu
- Innovation Institute for Integration of Medicine and Engineering, West China Hospital, Sichuan University, Chengdu, P. R. China
- Chinese Evidence-Based Medicine Center, West China Hospital, Sichuan University, Chengdu, P. R. China
- Center for education of medical humanities, West China Hospital, Sichuan University, Chengdu, P. R. China
| | - Zhengchi Li
- Center for education of medical humanities, West China Hospital, Sichuan University, Chengdu, P. R. China
| | - Jin Huang
- Innovation Institute for Integration of Medicine and Engineering, West China Hospital, Sichuan University, Chengdu, P. R. China
| | - Liang Du
- Innovation Institute for Integration of Medicine and Engineering, West China Hospital, Sichuan University, Chengdu, P. R. China
- West China Medical Publishers, West China Hospital, Sichuan University, Chengdu, P. R. China
- Chinese Evidence-Based Medicine Center, West China Hospital, Sichuan University, Chengdu, P. R. China
| |
Collapse
|
6
|
França TFA. Exploring undiscovered public knowledge in neuroscience. Eur J Neurosci 2024. [PMID: 38782707 DOI: 10.1111/ejn.16396] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2024] [Revised: 04/21/2024] [Accepted: 05/03/2024] [Indexed: 05/25/2024]
Abstract
In this essay, I argue that the combination of research synthesis and philosophical methods can fill an important methodological gap in neuroscience. While experimental research and formal modelling have seen their methods progressively increase in rigour and sophistication over the years, the task of analysing and synthesizing the vast literature reporting new results and models has lagged behind. The problem is aggravated because neuroscience has grown and expanded into a vast mosaic of related but partially independent subfields, each with their own literatures. This fragmentation not only makes it difficult to see the full picture emerging from neuroscience research but also limits progress in individual subfields. The current neuroscience literature has the perfect conditions to create what the information scientist Don Swanson called "undiscovered public knowledge"-knowledge that exists in the mutual implications of different published pieces of information but that is nonetheless undiscovered because those pieces have not been explicitly connected. Current methods for rigorous research synthesis, such as systematic reviews and meta-analyses, mostly focus on combining similar studies and are not suited for exploring undiscovered public knowledge. To that aim, they need to be adapted and supplemented. I argue that successful exploration of the hidden implications in the neuroscience literature will require the combination of these adapted research synthesis methods with philosophical methods for rigorous (and creative) analysis and synthesis.
Collapse
Affiliation(s)
- Thiago F A França
- Departamento de Psicobiologia, Escola Paulista de Medicina, Universidade Federal de São Paulo, São Paulo, Brazil
| |
Collapse
|
7
|
Agboola F, Wright AC. A framework for evaluating the diversity of clinical trials. J Clin Epidemiol 2024; 169:111299. [PMID: 38395092 DOI: 10.1016/j.jclinepi.2024.111299] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/22/2023] [Revised: 02/13/2024] [Accepted: 02/15/2024] [Indexed: 02/25/2024]
Abstract
OBJECTIVES The topic of diversity in clinical trials is rising to the forefront of many conversations in evidence-based medicine, and efforts are being made to improve the diversity of clinical trials. However, there is little uniformity in the methods used to evaluate these efforts. In this article, we describe our Clinical trial Diversity Rating (CDR) framework and the development process, including the broader considerations for evaluating the demographic diversity of clinical trials and their implications, and demonstrate its use through an illustrative example. STUDY DESIGN AND SETTING The development of the framework was a four-step process, including a scoping review, a cross-sectional study, creation of the tool, and integration of feedback from an advisory group. RESULTS Our scoping review identified 110 publications that examined clinical trial diversity. Race/ethnicity, sex, and age were the most common characteristics evaluated. About 85% clearly defined the benchmark used for evaluation, but less than half (48%) used disease prevalence as the benchmark. Only 64% of studies defined what would be considered adequate representation. The cross-sectional study, which applied some of the approaches identified in the literature, helped to identify the complexities of evaluating multinational trials and certain demographic characteristics. Key decisions for the CDR framework, such as the demographic characteristics to be evaluated, the benchmark and thresholds for evaluation, and how these factors contribute to the overall rating of clinical trial diversity, were informed by the two earlier phases and feedback from an advisory group. CONCLUSION The CDR framework provides an objective and transparent approach to evaluating clinical trial diversity. Groups such as Health Technology Assessment bodies, clinical trial regulators, policymakers, journal editors, and individual researchers can use this tool to examine, monitor, and improve diversity in clinical trials.
Collapse
Affiliation(s)
- Foluso Agboola
- Institute for Clinical and Economic Review (ICER), Boston, MA, 02108, USA.
| | - Abigail C Wright
- Institute for Clinical and Economic Review (ICER), Boston, MA, 02108, USA
| |
Collapse
|
8
|
Holt J, Bhar S, Schofield P, Koder D, Owen P, Seitz D, Bhowmik J. Protocol for a systematic review and meta-analysis of the prevalence of mental illness among nursing home residents. Syst Rev 2024; 13:109. [PMID: 38627826 PMCID: PMC11020180 DOI: 10.1186/s13643-024-02516-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/04/2023] [Accepted: 03/19/2024] [Indexed: 04/19/2024] Open
Abstract
BACKGROUND There is a high prevalence of mental illness in nursing home residents compared to older adults living in the community. This was highlighted in the most recent comprehensive systematic review on the topic, published in 2010. In the context of a rapidly aging population and increased numbers of older adults requiring residential care, this study aims to provide a contemporary account of the prevalence of mental illness among nursing home residents. METHODS This protocol was prepared in line with the PRISMA-P 2015 Statement. Systematic searches will be undertaken across six electronic databases: PubMed, Embase, Web of Science, PsycNET, CINAHL, and Abstracts in Social Gerontology. Peer-reviewed studies published from 2009 onwards which report the prevalence of mental illness within nursing home populations will be included. Database searches will be supplemented by forward and backward citation searching. Titles and abstracts of records will be screened using a semi-automated process. The full text of selected records will be assessed to confirm inclusion criteria are met. Study selection will be recorded in a PRISMA flowchart. A pilot-tested form will be used to extract data from included studies, alongside the JBI Critical Appraisal Checklist for Studies Reporting Prevalence Data. A study characteristics and results table will be prepared to present key details from each included study, supported by a narrative synthesis. Random-effects restricted maximum likelihood meta-analyses will be performed to compute pooled prevalence estimates for mental illnesses represented in the identified studies. Heterogeneity will be assessed using Cochran's Q and Higgins' I2 statistics. A Funnel plot and Egger's test will be used to assess publication bias. The GRADE approach will be used to assess the quality of the body of evidence identified. DISCUSSION The study will provide a comprehensive and contemporary account of the prevalence of mental illness among nursing home residents. Meta-analyses will provide robust prevalence estimates across a range of presentations. Key insights will be highlighted, including potential sources of heterogeneity. Implications for residents, researchers, care providers, and policymakers will be noted. SYSTEMATIC REVIEW REGISTRATION PROSPERO: CRD42023456226.
Collapse
Affiliation(s)
- Jared Holt
- Department of Psychological Sciences, Swinburne University of Technology, John Street, Hawthorn, VIC, 3122, Australia.
| | - Sunil Bhar
- Department of Psychological Sciences, Swinburne University of Technology, John Street, Hawthorn, VIC, 3122, Australia
| | - Penelope Schofield
- Department of Psychological Sciences, Swinburne University of Technology, John Street, Hawthorn, VIC, 3122, Australia
- Iverson Health Innovation Research Institute, Swinburne University of Technology, John Street, Hawthorn, VIC, 3122, Australia
- Health Services Research and Implementation Sciences, Peter MacCallum Cancer Centre, 305 Grattan Street, Melbourne, VIC, 3052, Australia
- Department of Oncology, Sir Peter MacCallum, The University of Melbourne, Grattan Street, Parkville, VIC, 3010, Australia
| | - Deborah Koder
- Department of Psychological Sciences, Swinburne University of Technology, John Street, Hawthorn, VIC, 3122, Australia
| | - Patrick Owen
- Eastern Health Emergency Medicine Program, Melbourne, VIC, Australia
- Eastern Health Clinical School, Monash University, Melbourne, VIC, Australia
| | - Dallas Seitz
- Department of Psychiatry, Cumming School of Medicine, University of Calgary, 3330 Hospital Drive NW, Calgary, AB, T2N 4N1, Canada
| | - Jahar Bhowmik
- Department of Health Science and Biostatistics, Swinburne University of Technology, John Street, Hawthorn, VIC, 3122, Australia
| |
Collapse
|
9
|
Nguyen Y, Beydon M, Foulquier N, Gordon R, Bouillot C, Hammitt KM, Bowman SJ, Mariette X, McCoy SS, Cornec D, Seror R. Identification of outcome domains in primary Sjögren's disease: A scoping review by the OMERACT Sjögren disease working group. Semin Arthritis Rheum 2024; 65:152385. [PMID: 38340608 DOI: 10.1016/j.semarthrit.2024.152385] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/09/2023] [Revised: 12/29/2023] [Accepted: 01/03/2024] [Indexed: 02/12/2024]
Abstract
OBJECTIVES Sjögren's disease (SjD) is a heterogenous disease with a wide range of manifestations, ranging from symptoms of dryness, fatigue, and pain, to systemic involvement. Considerable advances have been made to evaluate systemic activity or patient-reported outcomes, but most of the instruments were not able to assess all domains of this multifaceted disease. The aim of this scoping review was to generate domains that have been assessed in randomized controlled trials, as the first phase of the Outcome Measures in Rheumatology (OMERACT) process of core domain set development. METHODS We systematically searched Medline (Pubmed) and EMBASE between 2002 and March 2023 to identify all randomized controlled trials assessing relevant domains, using both a manual approach and an artificial intelligence software (BIBOT) that applies natural language processing to automatically identify relevant abstracts. Domains were mapped to core areas, as suggested by the OMERACT 2.1 Filter. RESULTS Among the 5,420 references, we included 60 randomized controlled trials, focusing either on overall disease manifestations (53%) or on a single organ/symptom: dry eyes (17%), xerostomia (15%), fatigue (12%), or pulmonary function (3%). The most frequently assessed domains were perceived dryness (52% for overall dryness), fatigue (57%), pain (52%), systemic disease activity (45%), lacrimal gland function (47%) and salivary function (55%), B-cell activation (60%), and health-related quality of life (40%). CONCLUSION Our scoping review highlighted the heterogeneity of SjD, in the study designs and domains. This will inform the OMERACT SjD working group to select the most appropriate core domains to be used in SjD clinical trials and to guide the future agenda for outcome measure research in SjD.
Collapse
Affiliation(s)
- Yann Nguyen
- Service de Rhumatologie, Assistance Publique - Hôpitaux de Paris, Hôpital Bicêtre, Le Kremlin-Bicêtre, France; Center for Immunology of Viral Infections and Auto-immune Diseases (IMVA), Institut pour la Santé et la Recherche Médicale (INSERM), UMR1184, Université Paris-Saclay, Le Kremlin Bicêtre, Paris, France
| | - Maxime Beydon
- Service de Rhumatologie, Assistance Publique - Hôpitaux de Paris, Hôpital Bicêtre, Le Kremlin-Bicêtre, France
| | | | - Rachael Gordon
- Department of Medicine, Division of Rheumatology, University of Pittsburgh School of Medicine, Pittsburgh, Pennsylvania, USA
| | | | | | - Simon J Bowman
- Rheumatology Department, University Hospitals Birmingham NHS Foundation Trust, Birmingham B15 2TH, UK
| | - Xavier Mariette
- Service de Rhumatologie, Assistance Publique - Hôpitaux de Paris, Hôpital Bicêtre, Le Kremlin-Bicêtre, France; Center for Immunology of Viral Infections and Auto-immune Diseases (IMVA), Institut pour la Santé et la Recherche Médicale (INSERM), UMR1184, Université Paris-Saclay, Le Kremlin Bicêtre, Paris, France
| | - Sara S McCoy
- Division of Rheumatology, Department of Medicine, School of Medicine and Public Health, University of Wisconsin, Madison, USA
| | - Divi Cornec
- LBAI, UMR1227, Univ Brest, Inserm, Brest, France; INSERM, UMR1227, Lymphocytes B, Autoimmunité et Immunothérapies, Université de Bretagne Occidentale, Service de Rhumatologie, CHU de Brest, Brest, France
| | - Raphaèle Seror
- Service de Rhumatologie, Assistance Publique - Hôpitaux de Paris, Hôpital Bicêtre, Le Kremlin-Bicêtre, France; Center for Immunology of Viral Infections and Auto-immune Diseases (IMVA), Institut pour la Santé et la Recherche Médicale (INSERM), UMR1184, Université Paris-Saclay, Le Kremlin Bicêtre, Paris, France.
| |
Collapse
|
10
|
Boetje J, van de Schoot R. The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses. Syst Rev 2024; 13:81. [PMID: 38429798 PMCID: PMC10908130 DOI: 10.1186/s13643-024-02502-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/24/2023] [Accepted: 02/19/2024] [Indexed: 03/03/2024] Open
Abstract
Active learning has become an increasingly popular method for screening large amounts of data in systematic reviews and meta-analyses. The active learning process continually improves its predictions on the remaining unlabeled records, with the goal of identifying all relevant records as early as possible. However, determining the optimal point at which to stop the active learning process is a challenge. The cost of additional labeling of records by the reviewer must be balanced against the cost of erroneous exclusions. This paper introduces the SAFE procedure, a practical and conservative set of stopping heuristics that offers a clear guideline for determining when to end the active learning process in screening software like ASReview. The eclectic mix of stopping heuristics helps to minimize the risk of missing relevant papers in the screening process. The proposed stopping heuristic balances the costs of continued screening with the risk of missing relevant records, providing a practical solution for reviewers to make informed decisions on when to stop screening. Although active learning can significantly enhance the quality and efficiency of screening, this method may be more applicable to certain types of datasets and problems. Ultimately, the decision to stop the active learning process depends on careful consideration of the trade-off between the costs of additional record labeling against the potential errors of the current model for the specific dataset and context.
Collapse
Affiliation(s)
- Josien Boetje
- Research Group Digital Ethics, Knowledge Center Learning and Innovation (LENI), Archimedes Institute, HU University of Applied Sciences Utrecht, Utrecht, the Netherlands.
| | - Rens van de Schoot
- Department of Methodology and Statistics, Faculty of Social and Behavioral Sciences, Utrecht University, Utrecht, The Netherlands
| |
Collapse
|
11
|
Yao X, Kumar MV, Su E, Flores Miranda A, Saha A, Sussman J. Evaluating the efficacy of artificial intelligence tools for the automation of systematic reviews in cancer research: A systematic review. Cancer Epidemiol 2024; 88:102511. [PMID: 38071872 DOI: 10.1016/j.canep.2023.102511] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/25/2023] [Revised: 11/23/2023] [Accepted: 11/29/2023] [Indexed: 01/27/2024]
Abstract
To evaluate the performance accuracy and workload savings of artificial intelligence (AI)-based automation tools in comparison with human reviewers in medical literature screening for systematic reviews (SR) of primary studies in cancer research in order to gain insights on improving the efficiency of producing SRs. Medline, Embase, the Cochrane Library, and PROSPERO databases were searched from inception to November 30, 2022. Then, forward and backward literature searches were completed, and the experts in this field including the authors of the articles included were contacted for a thorough grey literature search. This SR was registered on PROSPERO (CRD 42023384772). Among the 3947 studies obtained from search, five studies met the preplanned study selection criteria. These five studies evaluated four AI tools: Abstrackr (four studies), RobotAnalyst (one), EPPI-Reviewer (one), and DistillerSR (one). Without missing final included citations, Abstrackr eliminated 20%-88% of titles and abstracts (time saving of 7-86 hours) and 59% of the full-texts (62 h) from human review across four different cancer-related SRs. In comparison, RobotAnalyst (1% of titles and abstracts, 1 h), EPPI Review (38% of titles and abstracts, 58 h; 59% of full-texts, 62 h), DistillerSR (42% of titles and abstracts, 22 h) also provided similar or lower work savings for single cancer-related SRs. AI-based automation tools exhibited promising but varying levels of accuracy and efficiency during the screening process of medical literature for conducting SRs in the cancer field. Until further progress is made and thorough evaluations are conducted, AI tools should be utilized as supplementary aids rather than complete substitutes for human reviewers.
Collapse
Affiliation(s)
- Xiaomei Yao
- Department of Oncology, Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Center for Clinical Practice Guideline Conduction and Evaluation, Children's Hospital of Fudan University, Shanghai, China.
| | - Mithilesh V Kumar
- Faculty of Engineering, McMaster University, Hamilton, ON, Canada; Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
| | - Esther Su
- Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
| | | | - Ashirbani Saha
- Department of Oncology, McMaster University, Hamilton, Ontario, Canada; Escarpment Cancer Research Institute, Hamilton Health Sciences and McMaster University, Hamilton, ON, Canada
| | - Jonathan Sussman
- Department of Oncology, McMaster University, Hamilton, Ontario, Canada
| |
Collapse
|
12
|
Haby MM, Barreto JOM, Kim JYH, Peiris S, Mansilla C, Torres M, Guerrero-Magaña DE, Reveiz L. What are the best methods for rapid reviews of the research evidence? A systematic review of reviews and primary studies. Res Synth Methods 2024; 15:2-20. [PMID: 37696668 DOI: 10.1002/jrsm.1664] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/08/2023] [Revised: 07/09/2023] [Accepted: 08/07/2023] [Indexed: 09/13/2023]
Abstract
Rapid review methodology aims to facilitate faster conduct of systematic reviews to meet the needs of the decision-maker, while also maintaining quality and credibility. This systematic review aimed to determine the impact of different methodological shortcuts for undertaking rapid reviews on the risk of bias (RoB) of the results of the review. Review stages for which reviews and primary studies were sought included the preparation of a protocol, question formulation, inclusion criteria, searching, selection, data extraction, RoB assessment, synthesis, and reporting. We searched 11 electronic databases in April 2022, and conducted some supplementary searching. Reviewers worked in pairs to screen, select, extract data, and assess the RoB of included reviews and studies. We included 15 systematic reviews, 7 scoping reviews, and 65 primary studies. We found that several commonly used shortcuts in rapid reviews are likely to increase the RoB in the results. These include restrictions based on publication date, use of a single electronic database as a source of studies, and use of a single reviewer for screening titles and abstracts, selecting studies based on the full-text, and for extracting data. Authors of rapid reviews should be transparent in reporting their use of these shortcuts and acknowledge the possibility of them causing bias in the results. This review also highlights shortcuts that can save time without increasing the risk of bias. Further research is needed for both systematic and rapid reviews on faster methods for accurate data extraction and RoB assessment, and on development of more precise search strategies.
Collapse
Affiliation(s)
- Michelle M Haby
- Science and Knowledge Unit, Evidence and Intelligence for Action in Health Department, Pan American Health Organization, Washington, DC, USA
- Department of Chemical and Biological Sciences, University of Sonora, Hermosillo, Mexico
- Melbourne School of Population and Global Health, The University of Melbourne, Parkville, Victoria, Australia
| | | | - Jenny Yeon Hee Kim
- Science and Knowledge Unit, Evidence and Intelligence for Action in Health Department, Pan American Health Organization, Washington, DC, USA
| | - Sasha Peiris
- Science and Knowledge Unit, Evidence and Intelligence for Action in Health Department, Pan American Health Organization, Washington, DC, USA
| | - Cristián Mansilla
- McMaster Health Forum, McMaster University, Hamilton, Ontario, Canada
- Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada
| | - Marcela Torres
- Science and Knowledge Unit, Evidence and Intelligence for Action in Health Department, Pan American Health Organization, Washington, DC, USA
| | - Diego Emmanuel Guerrero-Magaña
- Doctoral Program in Chemical and Biological Sciences and Health, Department of Chemical and Biological Sciences, University of Sonora, Hermosillo, Mexico
| | - Ludovic Reveiz
- Science and Knowledge Unit, Evidence and Intelligence for Action in Health Department, Pan American Health Organization, Washington, DC, USA
| |
Collapse
|
13
|
Kolaski K, Logan LR, Ioannidis JPA. Guidance to best tools and practices for systematic reviews. Br J Pharmacol 2024; 181:180-210. [PMID: 37282770 DOI: 10.1111/bph.16100] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/26/2023] [Accepted: 04/26/2023] [Indexed: 06/08/2023] Open
Abstract
Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy. A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work. Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.
Collapse
Affiliation(s)
- Kat Kolaski
- Departments of Orthopaedic Surgery, Pediatrics, and Neurology, Wake Forest School of Medicine, Winston-Salem, North Carolina, USA
| | - Lynne Romeiser Logan
- Department of Physical Medicine and Rehabilitation, SUNY Upstate Medical University, Syracuse, New York, USA
| | - John P A Ioannidis
- Departments of Medicine, of Epidemiology and Population Health, of Biomedical Data Science, and of Statistics, and Meta-Research Innovation Center at Stanford (METRICS), Stanford University School of Medicine, Stanford, California, USA
| |
Collapse
|
14
|
Kamso MM, Pardo JP, Whittle SL, Buchbinder R, Wells G, Glennon V, Tugwell P, Deardon R, Sajobi T, Tomlinson G, Elliott J, Kelly SE, Hazlewood GS. Crowd-sourcing and automation facilitated the identification and classification of randomized controlled trials in a living review. J Clin Epidemiol 2023; 164:1-8. [PMID: 37865299 DOI: 10.1016/j.jclinepi.2023.10.007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/21/2023] [Revised: 10/05/2023] [Accepted: 10/14/2023] [Indexed: 10/23/2023]
Abstract
OBJECTIVES To evaluate an approach using automation and crowdsourcing to identify and classify randomized controlled trials (RCTs) for rheumatoid arthritis (RA) in a living systematic review (LSR). METHODS Records from a database search for RCTs in RA were screened first by machine learning and Cochrane Crowd to exclude non-RCTs, then by trainee reviewers using a Population, Intervention, Comparison, and Outcome (PICO) annotator platform to assess eligibility and classify the trial to the appropriate review. Disagreements were resolved by experts using a custom online tool. We evaluated the efficiency gains, sensitivity, accuracy, and interrater agreement (kappa scores) between reviewers. RESULTS From 42,452 records, machine learning and Cochrane Crowd excluded 28,777 (68%), trainee reviewers excluded 4,529 (11%), and experts excluded 7,200 (17%). The 1,946 records eligible for our LSR represented 220 RCTs and included 148/149 (99.3%) of known eligible trials from prior reviews. Although excluded from our LSRs, 6,420 records were classified as other RCTs in RA to inform future reviews. False negative rates among trainees were highest for the RCT domain (12%), although only 1.1% of these were for the primary record. Kappa scores for two reviewers ranged from moderate to substantial agreement (0.40-0.69). CONCLUSION A screening approach combining machine learning, crowdsourcing, and trainee participation substantially reduced the screening burden for expert reviewers and was highly sensitive.
Collapse
Affiliation(s)
- Mohammed Mujaab Kamso
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Canada.
| | - Jordi Pardo Pardo
- Centre for Practice-Changing Research, Ottawa Hospital Research Institute, The Ottawa Hospital - General Campus, Ottawa, Canada
| | - Samuel L Whittle
- Department Rheumatology, The Queen Elizabeth Hospital, Adelaide, Australia; School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
| | - Rachelle Buchbinder
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
| | - George Wells
- School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada
| | - Vanessa Glennon
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
| | - Peter Tugwell
- Department of Medicine, University of Ottawa, Ottawa, Canada; Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada; Department of Medicine, University of Ottawa, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Bruyere Research institute, Ottawa, Canada
| | - Rob Deardon
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Canada; Department of Mathematics & Statistics, Faculty of Science, University of Calgary, Calgary, Canada
| | - Tolulope Sajobi
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Canada
| | - George Tomlinson
- Department of Medicine, Dalla Lana School of Public Health, University of Toronto, Toronto, Canada
| | - Jesse Elliott
- Cardiovascular Research Methods Centre, University of Ottawa Heart Institute, Ottawa, Canada
| | - Shannon E Kelly
- School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada; Cardiovascular Research Methods Centre, University of Ottawa Heart Institute, Ottawa, Canada
| | - Glen S Hazlewood
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Canada
| |
Collapse
|
15
|
Cooper N, Germeni E, Freeman SC, Jaiswal N, Nevill CR, Sutton AJ, Taylor-Rowan M, Quinn TJ. New horizons in evidence synthesis for older adults. Age Ageing 2023; 52:afad211. [PMID: 37955937 DOI: 10.1093/ageing/afad211] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/04/2023] [Indexed: 11/14/2023] Open
Abstract
Evidence synthesis, embedded within a systematic review of the literature, is a well-established approach for collating and combining all the relevant information on a particular research question. A robust synthesis can establish the evidence base, which underpins best practice guidance. Such endeavours are frequently used by policymakers and practitioners to inform their decision making. Traditionally, an evidence synthesis of interventions consisted of a meta-analysis of quantitative data comparing two treatment alternatives addressing a specific and focussed clinical question. However, as the methods in the field have evolved, especially in response to the increasingly complex healthcare questions, more advanced evidence synthesis techniques have been developed. These can deal with extended data structures considering more than two treatment alternatives (network meta-analysis) and complex multicomponent interventions. The array of questions capable of being answered has also increased with specific approaches being developed for different evidence types including diagnostic, prognostic and qualitative data. Furthermore, driven by a desire for increasingly up-to-date evidence summaries, living systematic reviews have emerged. All of these methods can potentially have a role in informing older adult healthcare decisions. The aim of this review is to increase awareness and uptake of the increasingly comprehensive array of newer synthesis methods available and highlight their utility for answering clinically relevant questions in the context of older adult research, giving examples of where such techniques have already been effectively applied within the field. Their strengths and limitations are discussed, and we suggest user-friendly software options to implement the methods described.
Collapse
Affiliation(s)
- Nicola Cooper
- NIHR Evidence Synthesis Group @Complex Review Support Unit
- Department of Population Health Sciences, University of Leicester, Leicester, UK
| | - Evi Germeni
- NIHR Evidence Synthesis Group @Complex Review Support Unit
- Health Economics and Health Technology Assessment (HEHTA), School of Health and Wellbeing, University of Glasgow, Glasgow, UK
| | - Suzanne C Freeman
- NIHR Evidence Synthesis Group @Complex Review Support Unit
- Department of Population Health Sciences, University of Leicester, Leicester, UK
| | - Nishant Jaiswal
- NIHR Evidence Synthesis Group @Complex Review Support Unit
- Health Economics and Health Technology Assessment (HEHTA), School of Health and Wellbeing, University of Glasgow, Glasgow, UK
| | - Clareece R Nevill
- NIHR Evidence Synthesis Group @Complex Review Support Unit
- Department of Population Health Sciences, University of Leicester, Leicester, UK
| | - Alex J Sutton
- NIHR Evidence Synthesis Group @Complex Review Support Unit
- Department of Population Health Sciences, University of Leicester, Leicester, UK
| | - Martin Taylor-Rowan
- NIHR Evidence Synthesis Group @Complex Review Support Unit
- Health Economics and Health Technology Assessment (HEHTA), School of Health and Wellbeing, University of Glasgow, Glasgow, UK
| | - Terence J Quinn
- NIHR Evidence Synthesis Group @Complex Review Support Unit
- School of Cardiovascular and Medical Sciences, University of Glasgow, Glasgow, UK
| |
Collapse
|
16
|
Kolaski K, Logan LR, Ioannidis JPA. Guidance to best tools and practices for systematic reviews. Acta Anaesthesiol Scand 2023; 67:1148-1177. [PMID: 37288997 DOI: 10.1111/aas.14295] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/26/2023] [Accepted: 04/26/2023] [Indexed: 06/09/2023]
Abstract
Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy. A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work. Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.
Collapse
Affiliation(s)
- Kat Kolaski
- Departments of Orthopaedic Surgery, Pediatrics, and Neurology, Wake Forest School of Medicine, Winston-Salem, North Carolina, USA
| | - Lynne Romeiser Logan
- Department of Physical Medicine and Rehabilitation, SUNY Upstate Medical University, Syracuse, New York, USA
| | - John P A Ioannidis
- Departments of Medicine, of Epidemiology and Population Health, of Biomedical Data Science, and of Statistics, and Meta-Research Innovation Center at Stanford (METRICS), Stanford University School of Medicine, Stanford, California, USA
| |
Collapse
|
17
|
Li J, Kabouji J, Bouhadoun S, Tanveer S, Filion KB, Gore G, Josephson CB, Kwon CS, Jette N, Bauer PR, Day GS, Subota A, Roberts JI, Lukmanji S, Sauro K, Ismaili AA, Rahmani F, Chelabi K, Kerdougli Y, Seulami NM, Soumana A, Khalil S, Maynard N, Keezer MR. Sensitivity and specificity of alternative screening methods for systematic reviews using text mining tools. J Clin Epidemiol 2023; 162:72-80. [PMID: 37506951 DOI: 10.1016/j.jclinepi.2023.07.010] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2023] [Revised: 07/03/2023] [Accepted: 07/19/2023] [Indexed: 07/30/2023]
Abstract
OBJECTIVES To evaluate the impact of text mining (TM) on the sensitivity and specificity of title and abstract screening strategies for systematic reviews (SRs). STUDY DESIGN AND SETTING Twenty reviewers each evaluated a 500-citation set. We compared five screening methods: conventional double screen (CDS), single screen, double screen with TM, combined double screen and single screen with TM, and single screen with TM. Rayyan, Abstrackr, and SWIFT-Review were used for each TM method. The results of a published SR were used as the reference standard. RESULTS The mean sensitivity and specificity achieved by CDS were 97.0% (95% confidence interval [CI]: 94.7, 99.3) and 95.0% (95% CI: 93.0, 97.1). When compared with single screen, CDS provided a greater sensitivity without a decrease in specificity. Rayyan, Abstrackr, and SWIFT-Review identified all relevant studies. Specificity was often higher for TM-assisted methods than that for CDS, although with mean differences of only one-to-two percentage points. For every 500 citations not requiring manual screening, 216 minutes (95% CI: 169, 264) could be saved. CONCLUSION TM-assisted screening methods resulted in similar sensitivity and modestly improved specificity as compared to CDS. The time saved with TM makes this a promising new tool for SR.
Collapse
Affiliation(s)
- Jimmy Li
- Neurology Division, Centre Hospitalier de l'Université de Sherbrooke (CHUS), Sherbrooke, Canada; Centre de Recherche du Centre Hospitalier de l'Université de Montréal (CRCHUM), Montreal, Canada
| | - Joudy Kabouji
- Department of Pharmacy, University of Laval, Quebec City, Canada
| | - Sarah Bouhadoun
- Department of Neurology, McGill University, Montreal, Canada
| | - Sarah Tanveer
- Department of Pharmaceutical Health Services Research, University of Maryland, Baltimore, MD, USA
| | - Kristian B Filion
- Departments of Medicine and of Epidemiology, Biostatistics, and Occupational Health, McGill University, Montreal, Canada; Centre for Clinical Epidemiology, Jewish General Hospital - Lady Davis Institute, Montreal, Canada
| | - Genevieve Gore
- Schulich Library of Physical Sciences, Life Sciences, and Engineering, McGill University, Montreal, Canada
| | - Colin Bruce Josephson
- Department of Clinical Neurosciences, University of Calgary, Calgary, Canada; Department of Community Health Sciences, University of Calgary, Calgary, Canada; O'Brien Institute for Public Health, University of Calgary, Calgary, Canada; Hotchkiss Brain Institute, University of Calgary, Calgary, Canada; Center for Health Informatics, University of Calgary, Calgary, Canada
| | - Churl-Su Kwon
- Department of Neurology, Epidemiology, Neurosurgery and the Gertrude H. Sergievsky Center, Columbia University, New York, NY, USA
| | - Nathalie Jette
- Department of Clinical Neurosciences, University of Calgary, Calgary, Canada; Department of Community Health Sciences, University of Calgary, Calgary, Canada
| | - Prisca Rachel Bauer
- Department of Psychosomatic Medicine and Psychotherapy, Faculty of Medicine, Medical Center - University of Freiburg, Freiburg, Germany
| | - Gregory S Day
- Department of Neurology, Mayo Clinic Florida, Jacksonville, FL, USA
| | - Ann Subota
- Department of Community Health Sciences, University of Calgary, Calgary, Canada; Department of Medicine, University of Calgary, Calgary, Canada
| | - Jodie I Roberts
- Department of Community Health Sciences, University of Calgary, Calgary, Canada
| | - Sara Lukmanji
- Department of Community Health Sciences, University of Calgary, Calgary, Canada
| | - Khara Sauro
- Department of Community Health Sciences, University of Calgary, Calgary, Canada; Department of Surgery, University of Calgary, Calgary, Canada; Department of Oncology & Arnie Charbonneau Cancer Institute, University of Calgary, Calgary, Canada
| | | | - Feriel Rahmani
- Faculty of Medicine, McGill University, Montreal, Canada
| | | | | | | | - Aminata Soumana
- Department of Family Medicine, McGill University, Montreal, Canada
| | - Sarah Khalil
- Department of Family Medicine, McGill University, Montreal, Canada
| | - Noémie Maynard
- Department of Internal Medicine, McGill University, Montreal, Canada
| | - Mark Robert Keezer
- Centre de Recherche du Centre Hospitalier de l'Université de Montréal (CRCHUM), Montreal, Canada; Departments of Medicine and of Epidemiology, Biostatistics, and Occupational Health, McGill University, Montreal, Canada; Department of Neurosciences, Université de Montréal, Montreal, Canada; School of Public Health, Université de Montréal, Montreal, Canada.
| |
Collapse
|
18
|
Kurian N, Cherian JM, Cherian KK, Varghese KG. AI-assisted Boolean search. Br Dent J 2023; 235:363. [PMID: 37737385 DOI: 10.1038/s41415-023-6345-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/30/2023] [Accepted: 08/07/2023] [Indexed: 09/23/2023]
Affiliation(s)
- N Kurian
- Christian Dental College, Ludhiana, Punjab, India.
| | - J M Cherian
- Christian Dental College, Ludhiana, Punjab, India.
| | - K K Cherian
- Christian Dental College, Ludhiana, Punjab, India.
| | - K G Varghese
- Christian Dental College, Ludhiana, Punjab, India.
| |
Collapse
|
19
|
Abstract
Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy. A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work. Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.
Collapse
Affiliation(s)
- Kat Kolaski
- Departments of Orthopaedic Surgery, Pediatrics, and Neurology, Wake Forest School of Medicine, Winston-Salem, NC, USA
| | - Lynne Romeiser Logan
- Department of Physical Medicine and Rehabilitation, SUNY Upstate Medical University, Syracuse, NY, USA
| | - John P.A. Ioannidis
- Departments of Medicine, of Epidemiology and Population Health, of Biomedical Data Science, and of Statistics, and Meta-Research Innovation Center at Stanford (METRICS), Stanford University School of Medicine, Stanford, CA, USA
| |
Collapse
|
20
|
Kolaski K, Logan LR, Ioannidis JPA. Improving systematic reviews: guidance on guidance and other options and challenges. J Clin Epidemiol 2023; 159:266-273. [PMID: 37196861 DOI: 10.1016/j.jclinepi.2023.05.008] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/04/2023] [Revised: 05/07/2023] [Accepted: 05/09/2023] [Indexed: 05/19/2023]
Affiliation(s)
- Kat Kolaski
- Department of Orthopaedic Surgery, Wake Forest School of Medicine, Winston-Salem, NC, USA; Department of Pediatrics, Wake Forest School of Medicine, Winston-Salem, NC, USA; Department of Neurology, Wake Forest School of Medicine, Winston-Salem, NC, USA
| | - Lynne Romeiser Logan
- Department of Physical Medicine and Rehabilitation, SUNY Upstate Medical University, Syracuse, NY, USA
| | - John P A Ioannidis
- Department of Medicine, Stanford University School of Medicine, Stanford, CA, USA; Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, CA, USA; Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, CA, USA; Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, CA, USA; Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, USA.
| |
Collapse
|
21
|
Kolaski K, Logan LR, Ioannidis JPA. Guidance to best tools and practices for systematic reviews. BMC Infect Dis 2023; 23:383. [PMID: 37286949 DOI: 10.1186/s12879-023-08304-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/21/2023] [Accepted: 05/03/2023] [Indexed: 06/09/2023] Open
Abstract
Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.
Collapse
Affiliation(s)
- Kat Kolaski
- Departments of Orthopaedic Surgery, Pediatrics, and Neurology, Wake Forest School of Medicine, Winston-Salem, NC, USA.
| | - Lynne Romeiser Logan
- Department of Physical Medicine and Rehabilitation, SUNY Upstate Medical University, Syracuse, NY, USA
| | - John P A Ioannidis
- Departments of Medicine, of Epidemiology and Population Health, of Biomedical Data Science, and of Statistics, and Meta-Research Innovation Center at Stanford (METRICS), Stanford University School of Medicine, Stanford, CA, USA
| |
Collapse
|
22
|
Kolaski K, Logan LR, Ioannidis JPA. Guidance to best tools and practices for systematic reviews. Syst Rev 2023; 12:96. [PMID: 37291658 DOI: 10.1186/s13643-023-02255-9] [Citation(s) in RCA: 16] [Impact Index Per Article: 16.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/03/2022] [Accepted: 02/19/2023] [Indexed: 06/10/2023] Open
Abstract
Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.
Collapse
Affiliation(s)
- Kat Kolaski
- Departments of Orthopaedic Surgery, Pediatrics, and Neurology, Wake Forest School of Medicine, Winston-Salem, NC, USA.
| | - Lynne Romeiser Logan
- Department of Physical Medicine and Rehabilitation, SUNY Upstate Medical University, Syracuse, NY, USA
| | - John P A Ioannidis
- Departments of Medicine, of Epidemiology and Population Health, of Biomedical Data Science, and of Statistics, and Meta-Research Innovation Center at Stanford (METRICS), Stanford University School of Medicine, Stanford, CA, USA
| |
Collapse
|
23
|
Kolaski K, Logan LR, Ioannidis JPA. Guidance to Best Tools and Practices for Systematic Reviews. JBJS Rev 2023; 11:01874474-202306000-00009. [PMID: 37285444 DOI: 10.2106/jbjs.rvw.23.00077] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Abstract
» Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.» A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.» Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.
Collapse
Affiliation(s)
- Kat Kolaski
- Departments of Orthopaedic Surgery, Pediatrics, and Neurology, Wake Forest School of Medicine, Winston-Salem, North Carolina
| | - Lynne Romeiser Logan
- Department of Physical Medicine and Rehabilitation, SUNY Upstate Medical University, Syracuse, New York
| | - John P A Ioannidis
- Departments of Medicine, of Epidemiology and Population Health, of Biomedical Data Science, and of Statistics, and Meta-Research Innovation Center at Stanford (METRICS), Stanford University School of Medicine, Stanford, California
| |
Collapse
|
24
|
Oliveira Dos Santos Á, Sergio da Silva E, Machado Couto L, Valadares Labanca Reis G, Silva Belo V. The use of artificial intelligence for automating or semi-automating biomedical literature analyses: a scoping review. J Biomed Inform 2023; 142:104389. [PMID: 37187321 DOI: 10.1016/j.jbi.2023.104389] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2023] [Revised: 04/11/2023] [Accepted: 05/08/2023] [Indexed: 05/17/2023]
Abstract
OBJECTIVE Evidence-based medicine (EBM) is a decision-making process based on the conscious and judicious use of the best available scientific evidence. However, the exponential increase in the amount of information currently available likely exceeds the capacity of human-only analysis. In this context, artificial intelligence (AI) and its branches such as machine learning (ML) can be used to facilitate human efforts in analyzing the literature to foster EBM. The present scoping review aimed to examine the use of AI in the automation of biomedical literature survey and analysis with a view to establishing the state-of-the-art and identifying knowledge gaps. MATERIALS AND METHODS Comprehensive searches of the main databases were performed for articles published up to June 2022 and studies were selected according to inclusion and exclusion criteria. Data were extracted from the included articles and the findings categorized. RESULTS The total number of records retrieved from the databases was 12,145, of which 273 were included in the review. Classification of the studies according to the use of AI in evaluating the biomedical literature revealed three main application groups, namely assembly of scientific evidence (n=127; 47%), mining the biomedical literature (n=112; 41%) and quality analysis (n=34; 12%). Most studies addressed the preparation of systematic reviews, while articles focusing on the development of guidelines and evidence synthesis were the least frequent. The biggest knowledge gap was identified within the quality analysis group, particularly regarding methods and tools that assess the strength of recommendation and consistency of evidence. CONCLUSION Our review shows that, despite significant progress in the automation of biomedical literature surveys and analyses in recent years, intense research is needed to fill knowledge gaps on more difficult aspects of ML, deep learning and natural language processing, and to consolidate the use of automation by end-users (biomedical researchers and healthcare professionals).
Collapse
Affiliation(s)
| | - Eduardo Sergio da Silva
- Federal University of São João del-Rei, Campus Centro-Oeste Dona Lindu, Divinópolis, Minas Gerais, Brazil.
| | - Letícia Machado Couto
- Federal University of São João del-Rei, Campus Centro-Oeste Dona Lindu, Divinópolis, Minas Gerais, Brazil.
| | | | - Vinícius Silva Belo
- Federal University of São João del-Rei, Campus Centro-Oeste Dona Lindu, Divinópolis, Minas Gerais, Brazil.
| |
Collapse
|
25
|
Allen JP, Kim EK, Jimerson SR. Meta-Analyses and Systematic Reviews Advancing the Practice of School Psychology: The Imperative of Bringing Science to Practice. SCHOOL PSYCHOLOGY REVIEW 2023. [DOI: 10.1080/2372966x.2023.2178769] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/29/2023]
|
26
|
Kolaski K, Romeiser Logan L, Ioannidis JPA. Guidance to best tools and practices for systematic reviews1. J Pediatr Rehabil Med 2023; 16:241-273. [PMID: 37302044 DOI: 10.3233/prm-230019] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 06/12/2023] Open
Abstract
Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.
Collapse
Affiliation(s)
- Kat Kolaski
- Departments of Orthopaedic Surgery, Pediatrics, and Neurology, Wake Forest School of Medicine, Winston-Salem, NC, USA
| | - Lynne Romeiser Logan
- Department of Physical Medicine and Rehabilitation, SUNY Upstate Medical University, Syracuse, NY, USA
| | - John P A Ioannidis
- Departments of Medicine, of Epidemiology and Population Health, of Biomedical Data Science, and of Statistics, and Meta-Research Innovation Center at Stanford (METRICS), Stanford University School of Medicine, Stanford, CA, USA
| |
Collapse
|
27
|
Lee C, Thomas M, Ejaredar M, Kassam A, Whittle SL, Buchbinder R, Tugwell P, Wells G, Pardo JP, Hazlewood GS. Crowdsourcing trainees in a living systematic review provided valuable experiential learning opportunities: a mixed-methods study. J Clin Epidemiol 2022; 147:142-150. [PMID: 35364231 DOI: 10.1016/j.jclinepi.2022.03.019] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/17/2021] [Revised: 03/18/2022] [Accepted: 03/24/2022] [Indexed: 10/18/2022]
Abstract
OBJECTIVES To understand trainee experiences of participating in a living systematic review (LSR) for rheumatoid arthritis and the potential benefits in terms of experiential evidence-based medicine (EBM) education. STUDY DESIGN AND SETTING We conducted a mixed-methods study with trainees who participated in the LSR and who were recruited broadly from training programs in two countries. Trainees received task-specific training and completed one or more tasks in the review: assessing article eligibility, data extraction, and quality assessment. Trainees completed a survey followed by a one-on-one interview. Data were triangulated to produce broad themes. RESULTS Twenty one trainees, most of whom had a little prior experience with systematic reviews, reported a positive overall experience. Key benefits included learning opportunities, task segmentation (ability to focus on a single task, as opposed to an entire review), working in a supportive environment, international collaboration, and incentives such as authorship or acknowledgment. Trainees reported improvement in their competency as a Scholar, Collaborator, Leader, and Medical Expert. Challenges included communication and technical difficulties and appropriate matching of tasks to trainee skillsets. CONCLUSION Participating in an LSR provided benefits to a wide range of trainees and may provide an opportunity for experiential EBM training, while helping LSR sustainability.
Collapse
Affiliation(s)
- Chloe Lee
- Faculty of Medicine and Dentistry MD Program, University of Alberta, Edmonton, Canada
| | - Megan Thomas
- Department of Community Health Sciences, University of Calgary, Calgary, Canada
| | - Maede Ejaredar
- Cumming School of Medicine, Department of Medicine, University of Calgary, Calgary, Canada
| | - Aliya Kassam
- Department of Community Health Sciences, University of Calgary, Calgary, Canada
| | - Samuel L Whittle
- Department of Epidemiology and Preventive Medicine, School of Public Health and Preventive Medicine, Monash University, Melbourne, Monash-Cabrini Department of Musculoskeletal Health and Clinical Epidemiology, Cabrini Health, Melbourne and The Queen Elizabeth Hospital, Adelaide South Australia, Australia
| | - Rachelle Buchbinder
- Department of Epidemiology and Preventive Medicine, School of Public Health and Preventive Medicine, Monash University and Monash-Cabrini Department of Musculoskeletal Health and Clinical Epidemiology, Cabrini Health, Melbourne, Australia
| | - Peter Tugwell
- Department of Medicine, University of Ottawa, Ottawa, Ontario, Canada
| | - George Wells
- Cardiovascular Research Methods Centre, University of Ottawa Heart Institute, Ottawa, Ontario, Canada
| | - Jordi Pardo Pardo
- Centre for Global Health, University of Ottawa, Ottawa, Ontario, Canada
| | - Glen S Hazlewood
- Departments of Medicine and Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Canada.
| |
Collapse
|
28
|
Hartling L, Gates A. Friend or Foe? The Role of Robots in Systematic Reviews. Ann Intern Med 2022; 175:1045-1046. [PMID: 35635849 DOI: 10.7326/m22-1439] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/22/2022] Open
Affiliation(s)
- Lisa Hartling
- Alberta Research Centre for Health Evidence, Department of Pediatrics, Faculty of Medicine & Dentistry, University of Alberta, Edmonton, Alberta, Canada
| | - Allison Gates
- Alberta Research Centre for Health Evidence, Department of Pediatrics, Faculty of Medicine & Dentistry, University of Alberta, Edmonton, Alberta, Canada
| |
Collapse
|
29
|
Frandsen TF, Nielsen MFB, Eriksen MB. Avoiding searching for outcomes called for additional search strategies: A study of Cochrane review searches. J Clin Epidemiol 2022; 149:83-88. [PMID: 35661816 DOI: 10.1016/j.jclinepi.2022.05.015] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/22/2022] [Revised: 04/26/2022] [Accepted: 05/24/2022] [Indexed: 10/18/2022]
Abstract
OBJECTIVE A search strategy for a systematic review that use the PICO-model as framework, should include the population, the intervention(s), and the type(s) of study design. According to existing guidelines outcome should generally be excluded from the search strategy unless the search is multistranded. However, a recent study found that approximately 10% (51) of recent Cochrane reviews on interventions included outcomes in their literature search strategies. This study aims to analyze the alternatives to including outcome in a search strategy, by analyzing these recent Cochrane reviews. STUDY DESIGN This study analyses the 51 Cochrane reviews that included outcomes in their literature search strategies and analyzes the results of alternative search strategies that follow current recommendations. RESULTS Despite a small study sample of 51 reviews, the results show that many of the reviews excluded some of the recommended elements due to very broadly defined elements (e.g., all interventions or all people). Furthermore, excluding outcome from the search strategy is followed by an enormous increase in the number of retrieved records making it unmanageable to screen, if using a single stranded search strategy. CONCLUSION Recommendations for search strategies in difficult cases are called for.
Collapse
Affiliation(s)
- Tove Faber Frandsen
- University of Southern Denmark, Department of Design and Communication, Universitetsparken 1, 5000 Kolding, Denmark.
| | | | - Mette Brandt Eriksen
- University Library of Southern Denmark, Odense, Denmark, Campusvej 55, 5230 Odense M, Denmark
| |
Collapse
|
30
|
Semi-Automatic Systematic Literature Reviews and Information Extraction of COVID-19 Scientific Evidence: Description and Preliminary Results of the COKE Project. INFORMATION 2022. [DOI: 10.3390/info13030117] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/04/2023] Open
Abstract
The COVID-19 pandemic highlighted the importance of validated and updated scientific information to help policy makers, healthcare professionals, and the public. The speed in disseminating reliable information and the subsequent guidelines and policy implementation are also essential to save as many lives as possible. Trustworthy guidelines should be based on a systematic evidence review which uses reproducible analytical methods to collect secondary data and analyse them. However, the guidelines’ drafting process is time consuming and requires a great deal of resources. This paper aims to highlight the importance of accelerating and streamlining the extraction and synthesis of scientific evidence, specifically within the systematic review process. To do so, this paper describes the COKE (COVID-19 Knowledge Extraction framework for next generation discovery science) Project, which involves the use of machine reading and deep learning to design and implement a semi-automated system that supports and enhances the systematic literature review and guideline drafting processes. Specifically, we propose a framework for aiding in the literature selection and navigation process that employs natural language processing and clustering techniques for selecting and organizing the literature for human consultation, according to PICO (Population/Problem, Intervention, Comparison, and Outcome) elements. We show some preliminary results of the automatic classification of sentences on a dataset of abstracts related to COVID-19.
Collapse
|
31
|
Schmidt L, Finnerty Mutlu AN, Elmore R, Olorisade BK, Thomas J, Higgins JPT. Data extraction methods for systematic review (semi)automation: Update of a living systematic review. F1000Res 2021; 10:401. [PMID: 34408850 PMCID: PMC8361807 DOI: 10.12688/f1000research.51117.2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 07/27/2023] [Indexed: 10/12/2023] Open
Abstract
Background: The reliable and usable (semi)automation of data extraction can support the field of systematic review by reducing the workload required to gather information about the conduct and results of the included studies. This living systematic review examines published approaches for data extraction from reports of clinical studies. Methods: We systematically and continually search PubMed, ACL Anthology, arXiv, OpenAlex via EPPI-Reviewer, and the dblp computer science bibliography. Full text screening and data extraction are conducted within an open-source living systematic review application created for the purpose of this review. This living review update includes publications up to December 2022 and OpenAlex content up to March 2023. Results: 76 publications are included in this review. Of these, 64 (84%) of the publications addressed extraction of data from abstracts, while 19 (25%) used full texts. A total of 71 (93%) publications developed classifiers for randomised controlled trials. Over 30 entities were extracted, with PICOs (population, intervention, comparator, outcome) being the most frequently extracted. Data are available from 25 (33%), and code from 30 (39%) publications. Six (8%) implemented publicly available tools Conclusions: This living systematic review presents an overview of (semi)automated data-extraction literature of interest to different types of literature review. We identified a broad evidence base of publications describing data extraction for interventional reviews and a small number of publications extracting epidemiological or diagnostic accuracy data. Between review updates, trends for sharing data and code increased strongly: in the base-review, data and code were available for 13 and 19% respectively, these numbers increased to 78 and 87% within the 23 new publications. Compared with the base-review, we observed another research trend, away from straightforward data extraction and towards additionally extracting relations between entities or automatic text summarisation. With this living review we aim to review the literature continually.
Collapse
Affiliation(s)
- Lena Schmidt
- NIHR Innovation Observatory, Newcastle University, Newcastle upon Tyne, NE4 5TG, UK
- Sciome LLC, Research Triangle Park, North Carolina, 27713, USA
- Bristol Medical School, University of Bristol, Bristol, BS8 2PS, UK
| | | | - Rebecca Elmore
- Sciome LLC, Research Triangle Park, North Carolina, 27713, USA
| | - Babatunde K. Olorisade
- Bristol Medical School, University of Bristol, Bristol, BS8 2PS, UK
- Evaluate Ltd, London, SE1 2RE, UK
- Cardiff School of Technologies, Cardiff Metropolitan University, Cardiff, CF5 2YB, UK
| | - James Thomas
- UCL Social Research Institute, University College London, London, WC1H 0AL, UK
| | | |
Collapse
|