1
|
Lunny C, Veroniki AA, Hutton B, White I, Higgins J, Wright JM, Kim JY, Thirugnanasampanthar SS, Siddiqui S, Watt J, Moja L, Taske N, Lorenz RC, Gerrish S, Straus S, Minogue V, Hu F, Lin K, Kapani A, Nagi S, Chen L, Akbar-Nejad M, Tricco AC. Knowledge user survey and Delphi process to inform development of a new risk of bias tool to assess systematic reviews with network meta-analysis (RoB NMA tool). BMJ Evid Based Med 2023; 28:58-67. [PMID: 35948412 DOI: 10.1136/bmjebm-2022-111944] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 07/16/2022] [Indexed: 01/25/2023]
Abstract
BACKGROUND Network meta-analysis (NMA) is increasingly used in guideline development and other aspects of evidence-based decision-making. We aimed to develop a risk of bias (RoB) tool to assess NMAs (RoB NMA tool). An international steering committee recommended that the RoB NMA tool to be used in combination with the Risk of Bias in Systematic reviews (ROBIS) tool (i.e. because it was designed to assess biases only) or other similar quality appraisal tools (eg, A MeaSurement Tool to Assess systematic Reviews 2 [AMSTAR 2]) to assess quality of systematic reviews. The RoB NMA tool will assess NMA biases and limitations regarding how the analysis was planned, data were analysed and results were presented, including the way in which the evidence was assembled and interpreted. OBJECTIVES Conduct (a) a Delphi process to determine expert opinion on an item's inclusion and (b) a knowledge user survey to widen its impact. DESIGN Cross-sectional survey and Delphi process. METHODS Delphi panellists were asked to rate whether items should be included. All agreed-upon item were included in a second round of the survey (defined as 70% agreement). We surveyed knowledge users' views and preferences about the importance, utility and willingness to use the RoB NMA tool to evaluate evidence in practice and in policymaking. We included 12 closed and 10 open-ended questions, and we followed a knowledge translation plan to disseminate the survey through social media and professional networks. RESULTS 22 items were entered into a Delphi survey of which 28 respondents completed round 1, and 22 completed round 2. Seven items did not reach consensus in round 2. A total of 298 knowledge users participated in the survey (14% respondent rate). 75% indicated that their organisation produced NMAs, and 78% showed high interest in the tool, especially if they had received adequate training (84%). Most knowledge users and Delphi panellists preferred a tool to assess both bias in individual NMA results and authors' conclusions. Response bias in our sample is a major limitation as knowledge users working in high-income countries were more represented. One of the limitations of the Delphi process is that it depends on the purposive selection of experts and their availability, thus limiting the variability in perspectives and scientific disciplines. CONCLUSIONS This Delphi process and knowledge user survey informs the development of the RoB NMA tool.
Collapse
Affiliation(s)
- Carole Lunny
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, Ontario, Canada
- Cochrane Hypertension Review Group, The University of British Columbia, Vancouver, British Columbia, Canada
| | - Areti Angeliki Veroniki
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, Ontario, Canada
- Epidemiology Division, Dalla Lana School of Public Health and Institute for Health, Management, and Evaluation, University of Toronto, Toronto, Ontario, Canada
| | - Brian Hutton
- Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
- School of Epidemiology and Public Health, University of Ottawa, Ottawa, Ontario, Canada
| | - Ian White
- MRC Clinical Trials Unit, University College London, London, UK
| | - Jpt Higgins
- Population Health Sciences, NIHR Applied Research Collaboration West (ARC West), University Hospitals Bristol and Weston NHS Foundation Trust, Bristol, UK
| | - James M Wright
- Cochrane Hypertension Review Group, The University of British Columbia, Vancouver, British Columbia, Canada
| | | | | | - Shazia Siddiqui
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, Ontario, Canada
| | - Jennifer Watt
- Department of Medicine, University of Tornto, Toronto, ON, Canada
| | - Lorenzo Moja
- Department of Health Product Policy and Standards, World Health Organization, Geneva, Switzerland
| | - Nichole Taske
- Centre for Guidelines, National Institute for Health and Care Excellence (NICE), London, UK
| | - Robert C Lorenz
- Medical Consultancy Department, Federal Joint Committee - Gemeinsamer Bundesausschuss (G-BA), Berlin, Germany
| | | | - Sharon Straus
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, Ontario, Canada
- Department of Medicine, University of Tornto, Toronto, ON, Canada
| | | | - Franklin Hu
- Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, BC, Canada
| | - Kevin Lin
- Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, BC, Canada
| | - Ayah Kapani
- Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, BC, Canada
| | - Samin Nagi
- Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, BC, Canada
| | - Lillian Chen
- Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, BC, Canada
| | - Mona Akbar-Nejad
- Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, BC, Canada
| | - Andrea C Tricco
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, Ontario, Canada
- Epidemiology Division, Dalla Lana School of Public Health and Institute for Health, Management, and Evaluation, University of Toronto, Toronto, Ontario, Canada
| |
Collapse
|
2
|
Lunny C, Ramasubbu C, Puil L, Liu T, Gerrish S, Salzwedel DM, Mintzes B, Wright JM. Over half of clinical practice guidelines use non-systematic methods to inform recommendations: A methods study. PLoS One 2021; 16:e0250356. [PMID: 33886670 PMCID: PMC8062080 DOI: 10.1371/journal.pone.0250356] [Citation(s) in RCA: 34] [Impact Index Per Article: 11.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/22/2020] [Accepted: 04/06/2021] [Indexed: 12/15/2022] Open
Abstract
INTRODUCTION Assessing the process used to synthesize the evidence in clinical practice guidelines enables users to determine the trustworthiness of the recommendations. Clinicians are increasingly dependent on guidelines to keep up with vast quantities of medical literature, and guidelines are followed to avoid malpractice suits. We aimed to assess whether systematic methods were used when synthesizing the evidence for guidelines; and to determine the type of review cited in support of recommendations. METHODS Guidelines published in 2017 and 2018 were retrieved from the TRIP and Epistemonikos databases. We randomly sorted and sequentially screened clinical guidelines on all topics to select the first 50 that met our inclusion criteria. Our primary outcomes were the number of guidelines using either a systematic or non-systematic process to gather, assess, and synthesise evidence; and the numbers of recommendations within guidelines based on different types of evidence synthesis (systematic or non-systematic reviews). If a review was cited, we looked for evidence that it was critically appraised, and recorded which quality assessment tool was used. Finally, we examined the relation between the use of the GRADE approach, systematic review process, and type of funder. RESULTS Of the 50 guidelines, 17 (34%) systematically synthesised the evidence to inform recommendations. These 17 guidelines clearly reported their objectives and eligibility criteria, conducted comprehensive search strategies, and assessed the quality of the studies. Of the 29/50 guidelines that included reviews, 6 (21%) assessed the risk of bias of the review. The quality of primary studies was reported in 30/50 (60%) guidelines. CONCLUSIONS High quality, systematic review products provide the best available evidence to inform guideline recommendations. Using non-systematic methods compromises the validity and reliability of the evidence used to inform guideline recommendations, leading to potentially misleading and untrustworthy results.
Collapse
Affiliation(s)
- Carole Lunny
- Department of Anesthesiology, Pharmacology & Therapeutics, Faculty of Medicine, Cochrane Hypertension Review Group, Therapeutics Initiative, University of British Columbia, Vancouver, BC, Canada
- * E-mail:
| | - Cynthia Ramasubbu
- Department of Anesthesiology, Pharmacology & Therapeutics, Faculty of Medicine, Cochrane Hypertension Review Group, Therapeutics Initiative, University of British Columbia, Vancouver, BC, Canada
| | - Lorri Puil
- Department of Anesthesiology, Pharmacology & Therapeutics, Faculty of Medicine, Cochrane Hypertension Review Group, Therapeutics Initiative, University of British Columbia, Vancouver, BC, Canada
| | - Tracy Liu
- Department of Anesthesiology, Pharmacology & Therapeutics, Faculty of Medicine, Cochrane Hypertension Review Group, Therapeutics Initiative, University of British Columbia, Vancouver, BC, Canada
| | - Savannah Gerrish
- Department of Anesthesiology, Pharmacology & Therapeutics, Faculty of Medicine, Cochrane Hypertension Review Group, Therapeutics Initiative, University of British Columbia, Vancouver, BC, Canada
| | - Douglas M. Salzwedel
- Department of Anesthesiology, Pharmacology & Therapeutics, Faculty of Medicine, Cochrane Hypertension Review Group, Therapeutics Initiative, University of British Columbia, Vancouver, BC, Canada
| | - Barbara Mintzes
- Charles Perkins Centre, and School of Pharmacy, The University of Sydney, Camperdown, NSW, Australia
| | - James M. Wright
- Department of Anesthesiology, Pharmacology & Therapeutics, Faculty of Medicine, Cochrane Hypertension Review Group, Therapeutics Initiative, University of British Columbia, Vancouver, BC, Canada
| |
Collapse
|
3
|
Lunny C, Salzwedel DM, Mintzes B, Puil L, Liu T, Ramasubbu C, Gerrish S, Wright JM. Reply to Kelly Farrah and David Kaunelis regarding our study "Validation of five search filters for retrieval of clinical practice guidelines produced low precision". J Clin Epidemiol 2020; 133:156-157. [PMID: 33031933 DOI: 10.1016/j.jclinepi.2020.07.023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/22/2020] [Accepted: 07/22/2020] [Indexed: 11/30/2022]
Affiliation(s)
- Carole Lunny
- Department of Anesthesiology, Pharmacology and Therapeutics, Cochrane Hypertension Review Group, Therapeutics Initiative, Faculty of Medicine, University of British Columbia, 2176 Health Science Mall, Vancouver, BC, Canada, V6T 1Z3.
| | - Douglas M Salzwedel
- Department of Anesthesiology, Pharmacology and Therapeutics, Cochrane Hypertension Review Group, Therapeutics Initiative, Faculty of Medicine, University of British Columbia, 2176 Health Science Mall, Vancouver, BC, Canada, V6T 1Z3
| | - Barbara Mintzes
- The University of Sydney, Charles Perkins Centre, and School of Pharmacy, 6W75, The Hub, Sydney, New South Wales, 2006
| | - Lorri Puil
- Department of Anesthesiology, Pharmacology and Therapeutics, Cochrane Hypertension Review Group, Therapeutics Initiative, Faculty of Medicine, University of British Columbia, 2176 Health Science Mall, Vancouver, BC, Canada, V6T 1Z3
| | - Tracy Liu
- Faculty of Pharmaceutical Sciences, University of British Columbia, 2176 Health Science Mall, Vancouver, BC, Canada, V6T 1Z3
| | - Cynthia Ramasubbu
- Faculty of Pharmaceutical Sciences, University of British Columbia, 2176 Health Science Mall, Vancouver, BC, Canada, V6T 1Z3
| | - Savannah Gerrish
- Department of Anesthesiology, Pharmacology and Therapeutics, Cochrane Hypertension Review Group, Therapeutics Initiative, Faculty of Medicine, University of British Columbia, 2176 Health Science Mall, Vancouver, BC, Canada, V6T 1Z3
| | - James M Wright
- Department of Anesthesiology, Pharmacology and Therapeutics, Cochrane Hypertension Review Group, Therapeutics Initiative, Faculty of Medicine, University of British Columbia, 2176 Health Science Mall, Vancouver, BC, Canada, V6T 1Z3
| |
Collapse
|
4
|
Lunny C, Ramasubbu C, Gerrish S, Liu T, Salzwedel DM, Puil L, Mintzes B, Wright JJ. Impact and use of reviews and 'overviews of reviews' to inform clinical practice guideline recommendations: protocol for a methods study. BMJ Open 2020; 10:e031442. [PMID: 31964662 PMCID: PMC7044835 DOI: 10.1136/bmjopen-2019-031442] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/07/2019] [Revised: 12/03/2019] [Accepted: 12/03/2019] [Indexed: 12/11/2022] Open
Abstract
INTRODUCTION Guidelines are systematically developed recommendations to assist practitioner and patient decisions about treatments for clinical conditions. High quality and comprehensive systematic reviews and 'overviews of systematic reviews' (overviews) represent the best available evidence. Many guideline developers, such as the WHO and the Australian National Health and Medical Research Council, recommend the use of these research syntheses to underpin guideline recommendations. We aim to evaluate the impact and use of systematic reviews with and without pairwise meta-analysis or network meta-analyses (NMAs) and overviews in clinical practice guideline (CPG) recommendations. METHODS AND ANALYSIS CPGs will be retrieved from Turning Research Into Practice and Epistemonikos (2017-2018). The retrieved citations will be sorted randomly and then screened sequentially by two independent reviewers until 50 CPGs have been identified. We will include CPGs that provide at least two explicit recommendations for the management of any clinical condition. We will assess whether reviews or overviews were cited in a recommendation as part of the development process for guidelines. Data extraction will be done independently by two authors and compared. We will assess the risk of bias by examining how each guideline developed clinical recommendations. We will calculate the number and frequency of citations of reviews with or without pairwise meta-analysis, reviews with NMAs and overviews, and whether they were systematically or non-systematically developed. Results will be described, tabulated and categorised based on review type (reviews or overviews). CPGs reporting the use of the Grading of Recommendations, Assessment, Development and Evaluation approach will be compared with those using a different system, and pharmacological versus non-pharmacological CPGs will be compared. ETHICS AND DISSEMINATION No ethics approval is required. We will present at the Cochrane Colloquium and the Guidelines International Network conference.
Collapse
Affiliation(s)
- Carole Lunny
- Anesthesiology, Pharmacology & Therapeutics, Faculty of Medicine, University of British Columbia, Vancouver, British Columbia, Canada
| | - Cynthia Ramasubbu
- Faculty of Pharmacy, University of British Columbia, Vancouver, British Columbia, Canada
| | - Savannah Gerrish
- Anesthesiology, Pharmacology & Therapeutics, Faculty of Medicine, University of British Columbia, Vancouver, British Columbia, Canada
| | - Tracy Liu
- Faculty of Pharmacy, University of British Columbia, Vancouver, British Columbia, Canada
| | - Douglas M Salzwedel
- Anesthesiology, Pharmacology & Therapeutics, Faculty of Medicine, University of British Columbia, Vancouver, British Columbia, Canada
| | - Lorri Puil
- Anesthesiology, Pharmacology & Therapeutics, Faculty of Medicine, University of British Columbia, Vancouver, British Columbia, Canada
| | - Barbara Mintzes
- Faculty of Pharmacy and Charles Perkins Centre, University of Sydney Faculty of Health Sciences, The University of Sydney, New South Wales, Australia
| | - James Jim Wright
- Anesthesiology, Pharmacology & Therapeutics, Faculty of Medicine, University of British Columbia, Vancouver, British Columbia, Canada
| |
Collapse
|
5
|
Lunny C, Salzwedel DM, Liu T, Ramasubbu C, Gerrish S, Puil L, Mintzes B, Wright JM. Validation of five search filters for retrieval of clinical practice guidelines produced low precision. J Clin Epidemiol 2020; 117:109-116. [DOI: 10.1016/j.jclinepi.2019.09.022] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/19/2019] [Revised: 08/31/2019] [Accepted: 09/24/2019] [Indexed: 10/25/2022]
|