1
|
Lunny C, Whitelaw S, Reid EK, Chi Y, Ferri N, Zhang JHJ, Pieper D, Kanji S, Veroniki AA, Shea B, Dourka J, Ardern C, Pham B, Bagheri E, Tricco AC. Exploring decision-makers' challenges and strategies when selecting multiple systematic reviews: insights for AI decision support tools in healthcare. BMJ Open 2024; 14:e084124. [PMID: 38969371 PMCID: PMC11227798 DOI: 10.1136/bmjopen-2024-084124] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/10/2024] [Accepted: 06/24/2024] [Indexed: 07/07/2024] Open
Abstract
BACKGROUND Systematic reviews (SRs) are being published at an accelerated rate. Decision-makers may struggle with comparing and choosing between multiple SRs on the same topic. We aimed to understand how healthcare decision-makers (eg, practitioners, policymakers, researchers) use SRs to inform decision-making and to explore the potential role of a proposed artificial intelligence (AI) tool to assist in critical appraisal and choosing among SRs. METHODS We developed a survey with 21 open and closed questions. We followed a knowledge translation plan to disseminate the survey through social media and professional networks. RESULTS Our survey response rate was lower than expected (7.9% of distributed emails). Of the 684 respondents, 58.2% identified as researchers, 37.1% as practitioners, 19.2% as students and 13.5% as policymakers. Respondents frequently sought out SRs (97.1%) as a source of evidence to inform decision-making. They frequently (97.9%) found more than one SR on a given topic of interest to them. Just over half (50.8%) struggled to choose the most trustworthy SR among multiple. These difficulties related to lack of time (55.2%), or difficulties comparing due to varying methodological quality of SRs (54.2%), differences in results and conclusions (49.7%) or variation in the included studies (44.6%). Respondents compared SRs based on the relevance to their question of interest, methodological quality, and recency of the SR search. Most respondents (87.0%) were interested in an AI tool to help appraise and compare SRs. CONCLUSIONS Given the identified barriers of using SR evidence, an AI tool to facilitate comparison of the relevance of SRs, the search and methodological quality, could help users efficiently choose among SRs and make healthcare decisions.
Collapse
Affiliation(s)
- Carole Lunny
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, UBC, Toronto, Ontario, Canada
- Evidence Synthesis, Precisionheor LLC, Vancouver, British Columbia, Canada
| | - Sera Whitelaw
- Faculty of Medicine and Health Sciences, McGill University, Montreal, Québec, Canada
| | - Emma K Reid
- Department of Pharmacy, Nova Scotia Health Authority, Halifax, Nova Scotia, Canada
| | - Yuan Chi
- Yealth Network, Beijing Health Technology Co., Ltd, Beijing, China
| | - Nicola Ferri
- Department of Biomedical and Neuromotor Sciences, University of Bologna, Bologna, Italy
| | - Jia He Janet Zhang
- Anesthesiology, Pharmacology & Therapeutics, The University of British Columbia, Vancouver, British Columbia, Canada
| | - Dawid Pieper
- Institute for Health Services and Health System Research, Faculty of Health Sciences Brandenburg, Brandenburg Medical School Theodor Fontane, Neuruppin, Brandenburg, Germany
| | - Salmaan Kanji
- Department of Pharmacy, Ottawa Hospital, Ottawa, Ontario, Canada
- Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada
| | - Areti-Angeliki Veroniki
- Li Ka Shing Knowledge Institute of St Michael's Hospital, Knowledge Translation Program, St Michael's Hospital, Toronto, Ontario, Canada
- Institute of Health Policy Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
| | | | - Jasmeen Dourka
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, Ontario, Canada
| | - Clare Ardern
- Department of Family Practice, The University of British Columbia-Vancouver Campus, Vancouver, British Columbia, Canada
| | - Ba Pham
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, University of Toronto, Toronto, Ontario, Canada
| | - Ebrahim Bagheri
- Department of Electrical and Computer Engineering, Toronto Metropolitan University, Toronto, Ontario, Canada
| | - Andrea C Tricco
- Institute of Health Policy Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
- Knowledge Translation Program, St Michael's Hospital, Toronto, Ontario, Canada
| |
Collapse
|
2
|
Lunny C, Kanji S, Thabet P, Haidich AB, Bougioukas KI, Pieper D. Assessing the methodological quality and risk of bias of systematic reviews: primer for authors of overviews of systematic reviews. BMJ MEDICINE 2024; 3:e000604. [PMID: 38826514 PMCID: PMC11141200 DOI: 10.1136/bmjmed-2023-000604] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/24/2023] [Accepted: 05/03/2024] [Indexed: 06/04/2024]
Affiliation(s)
- Carole Lunny
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, ON, Canada
- Cochrane Hyptertension Review Group, Cochrane Canada, Vancouver, BC, Canada
| | - Salmaan Kanji
- Ottawa Hospital, Ottawa, ON, Canada
- Ottawa Hospital Research Institute, Ottawa, ON, Canada
| | - Pierre Thabet
- School of Pharmaceutical Sciences University of Ottawa, Hôpital Montfort, Ottawa, ON, Canada
| | - Anna-Bettina Haidich
- Department of Hygiene, Social-Preventive Medicine and Medical Statistics, Aristotle University of Thessaloniki Faculty of Health Sciences, Thessaloniki, Central Macedonia, Greece
| | - Konstantinos I Bougioukas
- Department of Hygiene, Social-Preventive Medicine and Medical Statistics, Aristotle University of Thessaloniki Faculty of Health Sciences, Thessaloniki, Central Macedonia, Greece
| | - Dawid Pieper
- Faculty of Health Sciences Brandenburg, Brandenburg Medical School Theodor Fontane Ruppin Clinics, Neuruppin, Brandenburg, Germany
| |
Collapse
|
3
|
Lunny C, Veroniki AA, Higgins JPT, Dias S, Hutton B, Wright JM, White IR, Whiting P, Tricco AC. Methodological review of NMA bias concepts provides groundwork for the development of a list of concepts for potential inclusion in a new risk of bias tool for network meta-analysis (RoB NMA Tool). Syst Rev 2024; 13:25. [PMID: 38217041 PMCID: PMC10785511 DOI: 10.1186/s13643-023-02388-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/18/2022] [Accepted: 11/10/2023] [Indexed: 01/14/2024] Open
Abstract
INTRODUCTION Network meta-analyses (NMAs) have gained popularity and grown in number due to their ability to provide estimates of the comparative effectiveness of multiple treatments for the same condition. The aim of this study is to conduct a methodological review to compile a preliminary list of concepts related to bias in NMAs. METHODS AND ANALYSIS We included papers that present items related to bias, reporting or methodological quality, papers assessing the quality of NMAs, or method papers. We searched MEDLINE, the Cochrane Library and unpublished literature (up to July 2020). We extracted items related to bias in NMAs. An item was excluded if it related to general systematic review quality or bias and was included in currently available tools such as ROBIS or AMSTAR 2. We reworded items, typically structured as questions, into concepts (i.e. general notions). RESULTS One hundred eighty-one articles were assessed in full text and 58 were included. Of these articles, 12 were tools, checklists or journal standards; 13 were guidance documents for NMAs; 27 were studies related to bias or NMA methods; and 6 were papers assessing the quality of NMAs. These studies yielded 99 items of which the majority related to general systematic review quality and biases and were therefore excluded. The 22 items we included were reworded into concepts specific to bias in NMAs. CONCLUSIONS A list of 22 concepts was included. This list is not intended to be used to assess biases in NMAs, but to inform the development of items to be included in our tool.
Collapse
Affiliation(s)
- Carole Lunny
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, 209 Victoria Street, East Building, Toronto, ON, M5B 1T8, Canada.
- Cochrane Hypertension Review Group, the Therapeutics Initiative, University of British Columbia, Vancouver, Canada.
| | - Areti-Angeliki Veroniki
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, 209 Victoria Street, East Building, Toronto, ON, M5B 1T8, Canada
| | - Julian P T Higgins
- Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK
- NIHR Bristol Biomedical Research Centre at University Hospitals Bristol and Weston NHS Foundation Trust and the University of Bristol, Bristol, UK
- NIHR Applied Research Collaboration West (ARC West) at University Hospitals Bristol and Weston NHS Foundation Trust, Bristol, UK
| | - Sofia Dias
- Centre for Reviews and Dissemination, University of York, York, UK
| | - Brian Hutton
- Ottawa Hospital Research Institute, Ottawa, Canada
- Ottawa University, School of Epidemiology and Public Health, Ottawa, Canada
| | - James M Wright
- Cochrane Hypertension Review Group, the Therapeutics Initiative, University of British Columbia, Vancouver, Canada
| | | | - Penny Whiting
- Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK
| | - Andrea C Tricco
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, 209 Victoria Street, East Building, Toronto, ON, M5B 1T8, Canada
- Dalla Lana School of Public Health & Institute of Health Policy, Management, and Evaluation, University of Toronto, Toronto, Canada
- Queen's Collaboration for Health Care Quality Joanna Briggs Institute Centre of Excellence, Queen's University, Kingston, Canada
| |
Collapse
|
4
|
Veroniki AA, Franco JVA. Exploring advanced methods for network meta-analysis. BMJ Evid Based Med 2023; 28:285-286. [PMID: 37495271 DOI: 10.1136/bmjebm-2023-112482] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 07/06/2023] [Indexed: 07/28/2023]
Affiliation(s)
- Areti Angeliki Veroniki
- Institute for Health Policy, Management, and Evaluation, University of Toronto, Toronto, Ontario, Canada
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, Toronto, Ontario, Canada
| | | |
Collapse
|