1
|
Lunny C, Whitelaw S, Reid EK, Chi Y, Ferri N, Zhang JHJ, Pieper D, Kanji S, Veroniki AA, Shea B, Dourka J, Ardern C, Pham B, Bagheri E, Tricco AC. Exploring decision-makers' challenges and strategies when selecting multiple systematic reviews: insights for AI decision support tools in healthcare. BMJ Open 2024; 14:e084124. [PMID: 38969371 DOI: 10.1136/bmjopen-2024-084124] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 07/07/2024] Open
Abstract
BACKGROUND Systematic reviews (SRs) are being published at an accelerated rate. Decision-makers may struggle with comparing and choosing between multiple SRs on the same topic. We aimed to understand how healthcare decision-makers (eg, practitioners, policymakers, researchers) use SRs to inform decision-making and to explore the potential role of a proposed artificial intelligence (AI) tool to assist in critical appraisal and choosing among SRs. METHODS We developed a survey with 21 open and closed questions. We followed a knowledge translation plan to disseminate the survey through social media and professional networks. RESULTS Our survey response rate was lower than expected (7.9% of distributed emails). Of the 684 respondents, 58.2% identified as researchers, 37.1% as practitioners, 19.2% as students and 13.5% as policymakers. Respondents frequently sought out SRs (97.1%) as a source of evidence to inform decision-making. They frequently (97.9%) found more than one SR on a given topic of interest to them. Just over half (50.8%) struggled to choose the most trustworthy SR among multiple. These difficulties related to lack of time (55.2%), or difficulties comparing due to varying methodological quality of SRs (54.2%), differences in results and conclusions (49.7%) or variation in the included studies (44.6%). Respondents compared SRs based on the relevance to their question of interest, methodological quality, and recency of the SR search. Most respondents (87.0%) were interested in an AI tool to help appraise and compare SRs. CONCLUSIONS Given the identified barriers of using SR evidence, an AI tool to facilitate comparison of the relevance of SRs, the search and methodological quality, could help users efficiently choose among SRs and make healthcare decisions.
Collapse
Affiliation(s)
- Carole Lunny
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, UBC, Toronto, Ontario, Canada
- Evidence Synthesis, Precisionheor LLC, Vancouver, British Columbia, Canada
| | - Sera Whitelaw
- Faculty of Medicine and Health Sciences, McGill University, Montreal, Québec, Canada
| | - Emma K Reid
- Department of Pharmacy, Nova Scotia Health Authority, Halifax, Nova Scotia, Canada
| | - Yuan Chi
- Yealth Network, Beijing Health Technology Co., Ltd, Beijing, China
| | - Nicola Ferri
- Department of Biomedical and Neuromotor Sciences, University of Bologna, Bologna, Italy
| | - Jia He Janet Zhang
- Anesthesiology, Pharmacology & Therapeutics, The University of British Columbia, Vancouver, British Columbia, Canada
| | - Dawid Pieper
- Institute for Health Services and Health System Research, Faculty of Health Sciences Brandenburg, Brandenburg Medical School Theodor Fontane, Neuruppin, Brandenburg, Germany
| | - Salmaan Kanji
- Department of Pharmacy, Ottawa Hospital, Ottawa, Ontario, Canada
- Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada
| | - Areti-Angeliki Veroniki
- Li Ka Shing Knowledge Institute of St Michael's Hospital, Knowledge Translation Program, St Michael's Hospital, Toronto, Ontario, Canada
- Institute of Health Policy Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
| | | | - Jasmeen Dourka
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, Ontario, Canada
| | - Clare Ardern
- Department of Family Practice, The University of British Columbia-Vancouver Campus, Vancouver, British Columbia, Canada
| | - Ba Pham
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, University of Toronto, Toronto, Ontario, Canada
| | - Ebrahim Bagheri
- Department of Electrical and Computer Engineering, Toronto Metropolitan University, Toronto, Ontario, Canada
| | - Andrea C Tricco
- Institute of Health Policy Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
- Knowledge Translation Program, St Michael's Hospital, Toronto, Ontario, Canada
| |
Collapse
|
2
|
Pilic A, Reda S, Jo CL, Burchett H, Bastías M, Campbell P, Gamage D, Henaff L, Kagina B, Külper-Schiek W, Lunny C, Marti M, Muloiwa R, Pieper D, Thomas J, Tunis MC, Younger Z, Wichmann O, Harder T. Use of existing systematic reviews for the development of evidence-based vaccination recommendations: Guidance from the SYSVAC expert panel. Vaccine 2023; 41:1968-1978. [PMID: 36804216 PMCID: PMC10015272 DOI: 10.1016/j.vaccine.2023.02.027] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/05/2022] [Revised: 02/08/2023] [Accepted: 02/09/2023] [Indexed: 02/18/2023]
Abstract
National immunization technical advisory groups (NITAGs) develop immunization-related recommendations and assist policy-makers in making evidence informed decisions. Systematic reviews (SRs) that summarize the available evidence on a specific topic are a valuable source of evidence in the development of such recommendations. However, conducting SRs requires significant human, time, and financial resources, which many NITAGs lack. Given that SRs already exist for many immunization-related topics, and to prevent duplication and overlap of reviews, a more practical approach may be for NITAGs to use existing SRs. Nevertheless, it can be challenging to identify relevant SRs, to select one SR from among multiple SRs, or to critically assess and effectively use them. To support NITAGs, the London School of Hygiene and Tropical Medicine, Robert Koch Institute and collaborators developed the SYSVAC project, which consists of an online registry of systematic reviews on immunization-related topics and an e-learning course, that supports the use of them (both freely accessible at https://www.nitag-resource.org/sysvac-systematic-reviews). Drawing from the e-learning course and recommendations from an expert panel, this paper outlines methods for using existing systematic reviews when making immunization-related recommendations. With specific examples and reference to the SYSVAC registry and other resources, it offers guidance on locating existing systematic reviews; assessing their relevance to a research question, up-to-dateness, and methodological quality and/or risk of bias; and considering the transferability and applicability of their findings to other populations or settings.
Collapse
Affiliation(s)
- Antonia Pilic
- Robert Koch Institute, Seestrasse 10, 13353 Berlin, Germany.
| | - Sarah Reda
- Robert Koch Institute, Seestrasse 10, 13353 Berlin, Germany
| | - Catherine L Jo
- Robert Koch Institute, Seestrasse 10, 13353 Berlin, Germany
| | - Helen Burchett
- Faculty of Public Health & Policy, London School of Hygiene and Tropical Medicine (LSHTM), 15-17 Tavistock Place, London WC1H 9SH, United Kingdom
| | | | - Pauline Campbell
- Nursing, Midwifery and Allied Health Professions Research Unit, Glasgow Caledonian University, Govan Mbeki Building, Glasgow G4 0BA, United Kingdom
| | - Deepa Gamage
- Epidemiology Unit and Advisory Committee on Communicable Diseases, Ministry of Health, #231, De Saram Place, Colombo 10, Sri Lanka
| | - Louise Henaff
- World Health Organization, Avenue Appia 20, 1211 Geneva, Switzerland
| | - Benjamin Kagina
- University of Cape Town, Faculty of Health Sciences, Observatory, 7925 Cape Town, South Africa
| | | | - Carole Lunny
- Knowledge Translation Program, St Michael's Hospital, Unity Health Toronto, and Cochrane Hypertension Review Group, University of British Columbia, 2176 Health Sciences Mall, Vancouver, BC V6T1Z2, Canada
| | - Melanie Marti
- World Health Organization, Avenue Appia 20, 1211 Geneva, Switzerland
| | - Rudzani Muloiwa
- University of Cape Town, Faculty of Health Sciences, Observatory, 7925 Cape Town, South Africa
| | - Dawid Pieper
- Brandenburg Medical School Theodor Fontane, Faculty of Health Sciences Brandenburg, Institute for Health Services and Health System Research, 15562 Rüdersdorf bei Berlin, Germany; Brandenburg Medical School Theodor Fontane, Center for Health Services Research, 15562 Rüdersdorf bei Berlin, Germany
| | - James Thomas
- Evidence for Policy and Practice Information and Co-ordinating (EPPI-) Centre, UCL Social Research Institute, University College London, 10 Woburn Square, London WC1H 0NR, United Kingdom
| | - Matthew C Tunis
- Public Health Agency of Canada, Centre for Immunization Readiness, 130 Colonnade Road, A.L. 6501H, Ottawa, Ontario K1A 0K9, Canada
| | - Zane Younger
- Robert Koch Institute, Seestrasse 10, 13353 Berlin, Germany
| | - Ole Wichmann
- Robert Koch Institute, Seestrasse 10, 13353 Berlin, Germany
| | - Thomas Harder
- Robert Koch Institute, Seestrasse 10, 13353 Berlin, Germany
| |
Collapse
|
3
|
Nørgaard B, Briel M, Chrysostomou S, Ristic Medic D, Buttigieg SC, Kiisk E, Puljak L, Bala M, Pericic TP, Lesniak W, Zając J, Lund H, Pieper D. A systematic review of meta-research studies finds substantial methodological heterogeneity in citation analyses to monitor evidence-based research. J Clin Epidemiol 2022; 150:126-141. [DOI: 10.1016/j.jclinepi.2022.06.021] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2022] [Revised: 06/21/2022] [Accepted: 06/29/2022] [Indexed: 10/17/2022]
|