1
|
Lunny C, Whitelaw S, Reid EK, Chi Y, Ferri N, Zhang JHJ, Pieper D, Kanji S, Veroniki AA, Shea B, Dourka J, Ardern C, Pham B, Bagheri E, Tricco AC. Exploring decision-makers' challenges and strategies when selecting multiple systematic reviews: insights for AI decision support tools in healthcare. BMJ Open 2024; 14:e084124. [PMID: 38969371 PMCID: PMC11227798 DOI: 10.1136/bmjopen-2024-084124] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/10/2024] [Accepted: 06/24/2024] [Indexed: 07/07/2024] Open
Abstract
BACKGROUND Systematic reviews (SRs) are being published at an accelerated rate. Decision-makers may struggle with comparing and choosing between multiple SRs on the same topic. We aimed to understand how healthcare decision-makers (eg, practitioners, policymakers, researchers) use SRs to inform decision-making and to explore the potential role of a proposed artificial intelligence (AI) tool to assist in critical appraisal and choosing among SRs. METHODS We developed a survey with 21 open and closed questions. We followed a knowledge translation plan to disseminate the survey through social media and professional networks. RESULTS Our survey response rate was lower than expected (7.9% of distributed emails). Of the 684 respondents, 58.2% identified as researchers, 37.1% as practitioners, 19.2% as students and 13.5% as policymakers. Respondents frequently sought out SRs (97.1%) as a source of evidence to inform decision-making. They frequently (97.9%) found more than one SR on a given topic of interest to them. Just over half (50.8%) struggled to choose the most trustworthy SR among multiple. These difficulties related to lack of time (55.2%), or difficulties comparing due to varying methodological quality of SRs (54.2%), differences in results and conclusions (49.7%) or variation in the included studies (44.6%). Respondents compared SRs based on the relevance to their question of interest, methodological quality, and recency of the SR search. Most respondents (87.0%) were interested in an AI tool to help appraise and compare SRs. CONCLUSIONS Given the identified barriers of using SR evidence, an AI tool to facilitate comparison of the relevance of SRs, the search and methodological quality, could help users efficiently choose among SRs and make healthcare decisions.
Collapse
Affiliation(s)
- Carole Lunny
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, UBC, Toronto, Ontario, Canada
- Evidence Synthesis, Precisionheor LLC, Vancouver, British Columbia, Canada
| | - Sera Whitelaw
- Faculty of Medicine and Health Sciences, McGill University, Montreal, Québec, Canada
| | - Emma K Reid
- Department of Pharmacy, Nova Scotia Health Authority, Halifax, Nova Scotia, Canada
| | - Yuan Chi
- Yealth Network, Beijing Health Technology Co., Ltd, Beijing, China
| | - Nicola Ferri
- Department of Biomedical and Neuromotor Sciences, University of Bologna, Bologna, Italy
| | - Jia He Janet Zhang
- Anesthesiology, Pharmacology & Therapeutics, The University of British Columbia, Vancouver, British Columbia, Canada
| | - Dawid Pieper
- Institute for Health Services and Health System Research, Faculty of Health Sciences Brandenburg, Brandenburg Medical School Theodor Fontane, Neuruppin, Brandenburg, Germany
| | - Salmaan Kanji
- Department of Pharmacy, Ottawa Hospital, Ottawa, Ontario, Canada
- Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada
| | - Areti-Angeliki Veroniki
- Li Ka Shing Knowledge Institute of St Michael's Hospital, Knowledge Translation Program, St Michael's Hospital, Toronto, Ontario, Canada
- Institute of Health Policy Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
| | | | - Jasmeen Dourka
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, Ontario, Canada
| | - Clare Ardern
- Department of Family Practice, The University of British Columbia-Vancouver Campus, Vancouver, British Columbia, Canada
| | - Ba Pham
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, University of Toronto, Toronto, Ontario, Canada
| | - Ebrahim Bagheri
- Department of Electrical and Computer Engineering, Toronto Metropolitan University, Toronto, Ontario, Canada
| | - Andrea C Tricco
- Institute of Health Policy Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
- Knowledge Translation Program, St Michael's Hospital, Toronto, Ontario, Canada
| |
Collapse
|
2
|
Lunny C, Kanji S, Thabet P, Haidich AB, Bougioukas KI, Pieper D. Assessing the methodological quality and risk of bias of systematic reviews: primer for authors of overviews of systematic reviews. BMJ MEDICINE 2024; 3:e000604. [PMID: 38826514 PMCID: PMC11141200 DOI: 10.1136/bmjmed-2023-000604] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/24/2023] [Accepted: 05/03/2024] [Indexed: 06/04/2024]
Affiliation(s)
- Carole Lunny
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, ON, Canada
- Cochrane Hyptertension Review Group, Cochrane Canada, Vancouver, BC, Canada
| | - Salmaan Kanji
- Ottawa Hospital, Ottawa, ON, Canada
- Ottawa Hospital Research Institute, Ottawa, ON, Canada
| | - Pierre Thabet
- School of Pharmaceutical Sciences University of Ottawa, Hôpital Montfort, Ottawa, ON, Canada
| | - Anna-Bettina Haidich
- Department of Hygiene, Social-Preventive Medicine and Medical Statistics, Aristotle University of Thessaloniki Faculty of Health Sciences, Thessaloniki, Central Macedonia, Greece
| | - Konstantinos I Bougioukas
- Department of Hygiene, Social-Preventive Medicine and Medical Statistics, Aristotle University of Thessaloniki Faculty of Health Sciences, Thessaloniki, Central Macedonia, Greece
| | - Dawid Pieper
- Faculty of Health Sciences Brandenburg, Brandenburg Medical School Theodor Fontane Ruppin Clinics, Neuruppin, Brandenburg, Germany
| |
Collapse
|
3
|
Millett DT, Benson PE, Cunningham SJ, McIntyre GT, Tsichlaki A, Naini FB, Laide C, Fleming PS. "Over-reviewing" of research? An analysis of orthodontic reviews. Am J Orthod Dentofacial Orthop 2024; 165:385-398.e5. [PMID: 38149957 DOI: 10.1016/j.ajodo.2023.10.013] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2023] [Revised: 10/01/2023] [Accepted: 10/01/2023] [Indexed: 12/28/2023]
Abstract
INTRODUCTION Research overviews may be undertaken to identify gaps in the literature, evaluate existing systematic reviews (SRs), and summarize evidence. This paper aims to profile overviews that have been conducted in orthodontics and related interventions since 2012 and to evaluate the degree of overlap among these overviews. METHODS Overviews published between January 1, 2012 and June 20, 2023 were identified using an electronic search involving Google Scholar and PubMed. A descriptive summary was produced, and citation matrices were used to evaluate the percentage of overlap between overviews using corrected covered area and covered area. This was classified as slight, moderate, high, or very high. RESULTS A total of 35 overviews were identified across a wide range of topics. Eight overviews included <10 SRs; 21 had 10-20 SRs; and 6 included >20 SRs (median no. of SRs per overview, 15; range, 3-62). Meta-analysis was conducted in only 5 overviews. Overlap between overviews on the same topic ranged from slight (2.7%) to very high (53.8%). CONCLUSIONS Almost all overview topics address treatments and their effects, with a wide variation in the number and quality of SRs included. There is considerable overlap in some orthodontic overviews, suggesting unnecessary duplication and research waste. Researchers should be encouraged to focus on primary data collection to add more high-quality data to SRs, which will ultimately enhance the yield from secondary and tertiary orthodontic research.
Collapse
Affiliation(s)
- Declan T Millett
- Cork University Dental School and Hospital, University College Cork, Cork, Ireland.
| | - Philip E Benson
- Academic Unit of Oral Health, Dentistry and Society, School of Clinical Dentistry, University of Sheffield, Sheffield, United Kingdom
| | - Susan J Cunningham
- Department of Orthodontics, University College London Eastman Dental Institute, London, United Kingdom
| | - Grant T McIntyre
- Dundee Dental Hospital, School of Denistry, University of Dundee, Dundee, United Kingdom
| | - Aliki Tsichlaki
- Department of Orthodontics, Barts and the London School of Medicine and Dentistry, Barts Health NHS Trust, London, United Kingdom
| | - Farhad B Naini
- St. George's University Hospitals NHS Foundation Trust, Kingston Hospital NHS Foundation Trust, London, United Kingdom
| | - Claire Laide
- Cork University Dental School and Hospital, University College Cork, Cork, Ireland
| | - Padhraig S Fleming
- Division of Public and Child Dental Health, Dublin Dental University Hospital, Trinity College Dublin, Dublin, Ireland
| |
Collapse
|
4
|
Belbasis L, Brooker RD, Zavalis E, Pezzullo AM, Axfors C, Ioannidis JP. Mapping and systematic appraisal of umbrella reviews in epidemiological research: a protocol for a meta-epidemiological study. Syst Rev 2023; 12:123. [PMID: 37452309 PMCID: PMC10347720 DOI: 10.1186/s13643-023-02265-7] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/24/2022] [Accepted: 06/01/2023] [Indexed: 07/18/2023] Open
Abstract
INTRODUCTION Umbrella review is one of the terms used to describe an overview of systematic reviews. During the last years, a rapid increase in the number of umbrella reviews on epidemiological studies has been observed, but there is no systematic assessment of their methodological and reporting characteristics. Our study aims to fill this gap by performing a systematic mapping of umbrella reviews in epidemiological research. METHODS We will perform a meta-epidemiological study including a systematic review in MEDLINE and EMBASE to identify all the umbrella reviews that focused on systematic reviews of epidemiological studies and were published from inception until December 31, 2022. We will consider eligible any research article which was designed as an umbrella review and summarized systematic reviews and meta-analyses of epidemiological studies. From each eligible article, we will extract information about the research topic, the methodological characteristics, and the reporting characteristics. We will examine whether the umbrella reviews assessed the strength of the available evidence and the rigor of the included systematic reviews. We will also examine whether these characteristics change across time. DISCUSSION Our study will systematically appraise the methodological and reporting characteristics of published umbrella reviews in epidemiological literature. The findings of our study can be used to improve the design and conduct of future umbrella reviews, to derive a standardized set of reporting and methodological guidelines for umbrella reviews, and to allow further meta-epidemiological work. SYSTEMATIC REVIEW REGISTRATION osf.io/sxzc6.
Collapse
Affiliation(s)
- Lazaros Belbasis
- Meta-Research Innovation Center Berlin, QUEST Center, Berlin Institute of Health, Charité-Universitätsmedizin Berlin, Berlin, Germany.
- Clinical Trials and Epidemiological Studies Unit, Nuffield Department of Population Health, University of Oxford, Oxford, UK.
| | - Robin D Brooker
- Department of Sociology, University of Essex, Colchester, UK
| | - Emmanuel Zavalis
- Meta-Research Innovation Center at Stanford, Stanford University, Stanford, USA
| | - Angelo Maria Pezzullo
- Section of Hygiene, Department of Life Sciences and Public Health, Università Cattolica del Sacro Cuore, Rome, Italy
| | - Cathrine Axfors
- Meta-Research Innovation Center at Stanford, Stanford University, Stanford, USA
- Department of Women's and Children's Health, Uppsala University, Uppsala, Sweden
| | - John Pa Ioannidis
- Meta-Research Innovation Center Berlin, QUEST Center, Berlin Institute of Health, Charité-Universitätsmedizin Berlin, Berlin, Germany
- Meta-Research Innovation Center at Stanford, Stanford University, Stanford, USA
- Department of Medicine, Stanford University School of Medicine, Stanford, CA, USA
- Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, CA, USA
- Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, CA, USA
- Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, CA, USA
| |
Collapse
|
5
|
Gosling CJ, Solanes A, Fusar-Poli P, Radua J. metaumbrella: the first comprehensive suite to perform data analysis in umbrella reviews with stratification of the evidence. BMJ MENTAL HEALTH 2023; 26:e300534. [PMID: 36792173 PMCID: PMC10035783 DOI: 10.1136/bmjment-2022-300534] [Citation(s) in RCA: 13] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/27/2022] [Accepted: 11/05/2022] [Indexed: 02/17/2023]
Abstract
OBJECTIVE Umbrella reviews are a new form of literature review that summarises the strength and/or quality of the evidence from all systematic reviews and meta-analyses conducted on a broad topic. This type of review thus provides an exhaustive examination of a vast body of information, providing the highest synthesis of knowledge. A critical strength of umbrella reviews is recalculating the meta-analytic estimates within a uniform framework to allow a consistent evidence stratification. To our best knowledge, there is no comprehensive package or software to conduct umbrella reviews. METHODS The R package metaumbrella accomplishes this aim by building on three core functions that (1) automatically perform all required calculations in an umbrella review (including but not limited to pairwise meta-analyses), (2) stratify evidence according to various classification criteria and (3) generate a visual representation of the results. In addition, this package allows flexible inputs for each review or meta-analysis analysed (eg, means plus SD, or effect size estimate and CI) and customisation (eg, stratification criteria following Ioannidis, algorithmic GRADE or personalised classification). RESULTS The R package metaumbrella thus provides the first comprehensive range of facilities to perform umbrella reviews with stratification of the evidence. CONCLUSION To facilitate the use of this package, even for researchers unfamiliar with R, we also provide a JAMOVI module and an open-access, browser-based graphical interface that allow use of the core functions of the package with a few mouse clicks.
Collapse
Affiliation(s)
- Corentin J Gosling
- DysCo Lab, Department of Psychology, Université Paris Nanterre, F-92000 Nanterre, France
- Laboratoire de Psychopathologie et Processus de Santé, Université de Paris, F-92100 Boulogne-Billancourt, France
- Centre for Innovation in Mental Health (CIMH), School of Psychology, Faculty of Environmental and Life Sciences, University of Southampton, Southampton, UK
| | - Aleix Solanes
- Institut d'Investigacions Biomediques August Pi i Sunyer, University of Barcelona, Barcelona, Spain
| | - Paolo Fusar-Poli
- Department of Psychosis Studies, King's College London, London, UK
- Department of Brain and Behavioral Sciences, University of Pavia, Pavia, Italy
| | - Joaquim Radua
- Institut d'Investigacions Biomediques August Pi i Sunyer, University of Barcelona, Barcelona, Spain
- Department of Psychosis Studies, King's College London, London, UK
- Department of Clinical Neuroscience, Centre for Psychiatric Research and Education, Karolinska Institutet, Solna, Sweden
| |
Collapse
|
6
|
Bougioukas KI, Pamporis K, Vounzoulaki E, Karagiannis T, Haidich AB. Types and associated methodologies of overviews of reviews in health care: a methodological study with published examples. J Clin Epidemiol 2023; 153:13-25. [PMID: 36351511 DOI: 10.1016/j.jclinepi.2022.11.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/25/2022] [Revised: 10/16/2022] [Accepted: 11/02/2022] [Indexed: 11/07/2022]
Abstract
OBJECTIVES To provide a descriptive insight into the different types of research questions/objectives and associated methodologies of overviews of reviews, supplemented by representative examples from the health care literature. STUDY DESIGN AND SETTING We searched in methodological articles for information on types and methodologies used in overviews and we explored the typology of reviews to identify similar types in literature of overviews. We categorized the types of overviews based on the research question/objective and the methodological approach used. Indicative examples for each category were selected from a sample of 2,121 overviews that were retrieved between 2000 and 2022 from MEDLINE, Scopus, and Cochrane Database of Systematic Reviews. RESULTS Based on type of research question, overviews were classified as overviews of reviews of interventions, associations, prediction, diagnostic accuracy, prevalence/incidence, experiences/views, economic evaluation, and measurement properties. Based on the methodological approach, we identified a variety of methods (systematic, living, rapid, scoping, evidence mapping, framework, and methodological) used in overviews. CONCLUSION The proposed classification and examples provide an essential starting point for future theory-building research on typologies and study designs of overviews of reviews. It is important for methodologists to make vigorous effort to create consensus-based methodological and reporting guidelines to cover these diverse types and key methodological challenges.
Collapse
Affiliation(s)
- Konstantinos I Bougioukas
- Department of Hygiene, Social-Preventive Medicine & Medical Statistics, Medical School, Aristotle University of Thessaloniki, University Campus, 54124 Thessaloniki, Greece
| | - Konstantinos Pamporis
- Department of Hygiene, Social-Preventive Medicine & Medical Statistics, Medical School, Aristotle University of Thessaloniki, University Campus, 54124 Thessaloniki, Greece
| | - Elpida Vounzoulaki
- Diabetes Research Centre, Leicester General Hospital, University of Leicester, Leicester LE5 4PW, UK
| | - Thomas Karagiannis
- Clinical Research and Evidence-Based Medicine Unit, Second Medical Department, Aristotle University of Thessaloniki, Thessaloniki, Greece; Diabetes Centre, Second Medical Department, Aristotle University of Thessaloniki, Thessaloniki, Greece
| | - Anna-Bettina Haidich
- Department of Hygiene, Social-Preventive Medicine & Medical Statistics, Medical School, Aristotle University of Thessaloniki, University Campus, 54124 Thessaloniki, Greece.
| |
Collapse
|
7
|
Lunny C, Thirugnanasampanthar SS, Kanji S, Ferri N, Pieper D, Whitelaw S, Tasnim S, Nelson H, Reid EK, Zhang JH(J, Kalkat B, Chi Y, Abdoulrezzak R, Zheng DW, Pangka LR, Wang D(XR, Safavi P, Sooch A, Kang KT, Tricco AC. How can clinicians choose between conflicting and discordant systematic reviews? A replication study of the Jadad algorithm. BMC Med Res Methodol 2022; 22:276. [PMID: 36289496 PMCID: PMC9597955 DOI: 10.1186/s12874-022-01750-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2022] [Accepted: 10/04/2022] [Indexed: 11/10/2022] Open
Abstract
Introduction The exponential growth of published systematic reviews (SRs) presents challenges for decision makers seeking to answer clinical, public health or policy questions. In 1997, an algorithm was created by Jadad et al. to choose the best SR across multiple. Our study aims to replicate author assessments using the Jadad algorithm to determine: (i) if we chose the same SR as the authors; and (ii) if we reach the same results. Methods We searched MEDLINE, Epistemonikos, and Cochrane Database of SRs. We included any study using the Jadad algorithm. We used consensus building strategies to operationalise the algorithm and to ensure a consistent approach to interpretation. Results We identified 21 studies that used the Jadad algorithm to choose one or more SRs. In 62% (13/21) of cases, we were unable to replicate the Jadad assessment and ultimately chose a different SR than the authors. Overall, 18 out of the 21 (86%) independent Jadad assessments agreed in direction of the findings despite 13 having chosen a different SR. Conclusions Our results suggest that the Jadad algorithm is not reproducible between users as there are no prescriptive instructions about how to operationalise the algorithm. In the absence of a validated algorithm, we recommend that healthcare providers, policy makers, patients and researchers address conflicts between review findings by choosing the SR(s) with meta-analysis of RCTs that most closely resemble their clinical, public health, or policy question, are the most recent, comprehensive (i.e. number of included RCTs), and at the lowest risk of bias. Supplementary Information The online version contains supplementary material available at 10.1186/s12874-022-01750-2. This is the first empirical study to replicate Jadad algorithm assessments to evaluate discordance across systematic reviews. In 62% (13/21) of cases, we were unable to replicate the Jadad algorithm assessment and ultimately chose a different systematic review than the authors. When assessing systematic reviews using the Jadad algorithm, some steps of the Jadad algorithm were vague in description, making it difficult to operationalise, interpret, and use. The Jadad algorithm has several limitations as it does not account for the last literature search of the systematic review and publication recency of included trials. To assess discordance in the absence of an algorithm, we recommend decision makers consider relevance (objectives that most closely resemble their clinical question), recency (dates of search), comprehensiveness (most trials), and risk of bias (lowest risk of bias SR) when choosing one systematic review across multiple.
Collapse
Affiliation(s)
- C Lunny
- grid.17091.3e0000 0001 2288 9830Unity Health Toronto and the Cochrane Hypertension Review Group, St Michael’s Hospital, University of British Columbia, V6T 1Z3 Vancouver, BC Canada
| | - Sai Surabi Thirugnanasampanthar
- grid.17063.330000 0001 2157 2938Epidemiology Division, Institute for Health Policy, Management, and Evaluation, Dalla Lana School of Public Health, University of Toronto, Toronto, ON Canada
| | - S Kanji
- grid.412687.e0000 0000 9606 5108The Ottawa Hospital, Ottawa Hospital Research Institute, Ottawa, Canada
| | - N Ferri
- grid.6292.f0000 0004 1757 1758Division of Occupational Medicine, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy ,grid.6292.f0000 0004 1757 1758Department of Biomedical and Neuromotor Sciences (DIBINEM), Alma Mater Studiorum, University of Bologna, 40138 Bologna, Italy
| | - D Pieper
- grid.473452.3Faculty of Health Sciences Brandenburg, Brandenburg Medical School Theodor Fontane, Institute for Health Services and Health System Research, Rüdersdorf, Germany ,grid.473452.3Center for Health Services Research, Brandenburg Medical School Theodor Fontane, Rüdersdorf, Germany
| | - S Whitelaw
- grid.14709.3b0000 0004 1936 8649Faculty of Medicine and Health Sciences, McGill University, Montreal, QC Canada
| | - S Tasnim
- grid.17091.3e0000 0001 2288 9830Cochrane Hypertension Review Group, University of British Columbia, 2176 Health Science Mall, Vancouver, BC V6T 1Z3 Canada
| | - H Nelson
- grid.410356.50000 0004 1936 8331Faculty of Health Sciences, Queen’s University, Kingston, ON Canada
| | - EK Reid
- Nova Scotia Health, Halifax, NS Canada
| | - Jia He (Janet) Zhang
- grid.17091.3e0000 0001 2288 9830Faculty of Science, University of British Columbia, Vancouver, BC Canada
| | - Banveer Kalkat
- grid.17091.3e0000 0001 2288 9830Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, BC Canada
| | - Yuan Chi
- Beijing Yealth Technology Co., Ltd, Beijing, China ,Cochrane Campbell Global Ageing Partnership, London, United Kingdom
| | - Reema Abdoulrezzak
- grid.17091.3e0000 0001 2288 9830Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, BC Canada
| | - Di Wen Zheng
- grid.17091.3e0000 0001 2288 9830Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, BC Canada
| | - Lindy R.S. Pangka
- grid.17091.3e0000 0001 2288 9830Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, BC Canada
| | - Dian (Xin Ran) Wang
- grid.17091.3e0000 0001 2288 9830Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, BC Canada
| | - Parisa Safavi
- grid.17091.3e0000 0001 2288 9830Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, BC Canada
| | - Anmol Sooch
- grid.17091.3e0000 0001 2288 9830Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, BC Canada
| | - Kevin T. Kang
- grid.17091.3e0000 0001 2288 9830Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, BC Canada
| | - Andrea C, Tricco
- grid.415502.7Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Unity Health Toronto, 209 Victoria St, M5B 1T8 Toronto, ON Canada ,grid.17063.330000 0001 2157 2938Epidemiology Division, Institute for Health Policy, Management, and Evaluation, Dalla Lana School of Public Health, University of Toronto, 155 College St Room 500, M5T 3M7 Toronto, ON Canada ,grid.410356.50000 0004 1936 8331Queen’s Collaboration for Health Care Quality Joanna Briggs Institute Centre of Excellence, School of Nursing, Queen’s University, 99 University Ave, K7L 3N6 Kingston, ON Canada
| |
Collapse
|
8
|
Gates M, Gates A, Pieper D, Fernandes RM, Tricco AC, Moher D, Brennan SE, Li T, Pollock M, Lunny C, Sepúlveda D, McKenzie JE, Scott SD, Robinson KA, Matthias K, Bougioukas KI, Fusar-Poli P, Whiting P, Moss SJ, Hartling L. Reporting guideline for overviews of reviews of healthcare interventions: development of the PRIOR statement. BMJ 2022; 378:e070849. [PMID: 35944924 PMCID: PMC9361065 DOI: 10.1136/bmj-2022-070849] [Citation(s) in RCA: 139] [Impact Index Per Article: 69.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/30/2022]
Abstract
OBJECTIVE To develop a reporting guideline for overviews of reviews of healthcare interventions. DESIGN Development of the preferred reporting items for overviews of reviews (PRIOR) statement. PARTICIPANTS Core team (seven individuals) led day-to-day operations, and an expert advisory group (three individuals) provided methodological advice. A panel of 100 experts (authors, editors, readers including members of the public or patients) was invited to participate in a modified Delphi exercise. 11 expert panellists (chosen on the basis of expertise, and representing relevant stakeholder groups) were invited to take part in a virtual face-to-face meeting to reach agreement (≥70%) on final checklist items. 21 authors of recently published overviews were invited to pilot test the checklist. SETTING International consensus. INTERVENTION Four stage process established by the EQUATOR Network for developing reporting guidelines in health research: project launch (establish a core team and expert advisory group, register intent), evidence reviews (systematic review of published overviews to describe reporting quality, scoping review of methodological guidance and author reported challenges related to undertaking overviews of reviews), modified Delphi exercise (two online Delphi surveys to reach agreement (≥70%) on relevant reporting items followed by a virtual face-to-face meeting), and development of the reporting guideline. RESULTS From the evidence reviews, we drafted an initial list of 47 potentially relevant reporting items. An international group of 52 experts participated in the first Delphi survey (52% participation rate); agreement was reached for inclusion of 43 (91%) items. 44 experts (85% retention rate) completed the second Delphi survey, which included the four items lacking agreement from the first survey and five new items based on respondent comments. During the second round, agreement was not reached for the inclusion or exclusion of the nine remaining items. 19 individuals (6 core team and 3 expert advisory group members, and 10 expert panellists) attended the virtual face-to-face meeting. Among the nine items discussed, high agreement was reached for the inclusion of three and exclusion of six. Six authors participated in pilot testing, resulting in minor wording changes. The final checklist includes 27 main items (with 19 sub-items) across all stages of an overview of reviews. CONCLUSIONS PRIOR fills an important gap in reporting guidance for overviews of reviews of healthcare interventions. The checklist, along with rationale and example for each item, provides guidance for authors that will facilitate complete and transparent reporting. This will allow readers to assess the methods used in overviews of reviews of healthcare interventions and understand the trustworthiness and applicability of their findings.
Collapse
Affiliation(s)
- Michelle Gates
- Alberta Research Centre for Health Evidence, Department of Pediatrics, University of Alberta, Edmonton, AB, Canada
| | - Allison Gates
- Alberta Research Centre for Health Evidence, Department of Pediatrics, University of Alberta, Edmonton, AB, Canada
| | - Dawid Pieper
- Institute for Research in Operative Medicine, Witten/Herdecke University, Witten, Germany
| | - Ricardo M Fernandes
- Clinical Pharmacology Unit, Faculty of Medicine and Institute of Molecular Medicine, University of Lisbon, Lisbon, Portugal
| | - Andrea C Tricco
- Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, ON, Canada
- Epidemiology Division and Institute of Health Policy, Management, and Evaluation, Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada
- Queen's Collaboration for Health Care Quality Joanna Briggs Institute Centre of Excellence, Queen's University, Kingston, ON, Canada
| | - David Moher
- Clinical Epidemiology Program, Ottawa Hospital Research Institute, and School of Epidemiology and Public Health, University of Ottawa, ON, Canada
| | - Sue E Brennan
- School of Public Health and Preventive Medicine, Monash University, Melbourne, VIC, Australia
| | - Tianjing Li
- Department of Ophthalmology, School of Medicine, University of Colorado Anschutz Medical Campus, Denver, CO, USA
| | | | - Carole Lunny
- Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, ON, Canada
| | - Dino Sepúlveda
- Department of Health Technology Assessment and Evidence Based Healthcare, Ministry of Health, Chile
- School of Medicine, Autonomous University of Chile, Santiago, Chile
| | - Joanne E McKenzie
- School of Public Health and Preventive Medicine, Monash University, Melbourne, VIC, Australia
| | | | | | - Katja Matthias
- Faculty of Electrical Engineering and Computer Science, University of Applied Science Stralsund, Stralsund, Germany
| | - Konstantinos I Bougioukas
- Department of Hygiene, Social-Preventive Medicine, and Medical Statistics, School of Medicine, Faculty of Health Sciences, Aristotle University of Thessaloniki, Thessaloniki, Greece
| | - Paolo Fusar-Poli
- Early Psychosis: Interventions and Clinical-detection Laboratory, Department of Psychosis Studies, Institute of Psychiatry, Psychology, and Neuroscience, King's Collect London, London, UK
- OASIS Service, South London and Maudsley NHS Foundation Trust, London, UK
- Department of Brain and Behavioural Sciences, University of Pavia, Pavia, Italy
- National Institute for Health Research, Maudsley Biomedical Research Centre, South London and Maudsley NHS Foundation Trust, London, UK
| | - Penny Whiting
- Bristol Medical School, University of Bristol, Bristol, UK
| | - Stephana J Moss
- Department of Critical Care Medicine, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
| | - Lisa Hartling
- Alberta Research Centre for Health Evidence, Department of Pediatrics, University of Alberta, Edmonton, AB, Canada
| |
Collapse
|
9
|
Lunny C, Thirugnanasampanthar SS, Kanji S, Ferri N, Thabet P, Pieper D, Tasnim S, Nelson H, Reid E, Zhang JHJ, Kalkat B, Chi Y, Thompson J, Abdoulrezzak R, Zheng DWW, Pangka L, Wang DXR, Safavi P, Sooch A, Kang K, Whitelaw S, Tricco AC. Identifying and addressing conflicting results across multiple discordant systematic reviews on the same question: protocol for a replication study of the Jadad algorithm. BMJ Open 2022; 12:e054223. [PMID: 35443948 PMCID: PMC9021774 DOI: 10.1136/bmjopen-2021-054223] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/04/2023] Open
Abstract
INTRODUCTION An increasing growth of systematic reviews (SRs) presents notable challenges for decision-makers seeking to answer clinical questions. In 1997, an algorithm was created by Jadad to assess discordance in results across SRs on the same question. Our study aims to (1) replicate assessments done in a sample of studies using the Jadad algorithm to determine if the same SR would have been chosen, (2) evaluate the Jadad algorithm in terms of utility, efficiency and comprehensiveness, and (3) describe how authors address discordance in results across multiple SRs. METHODS AND ANALYSIS We will use a database of 1218 overviews (2000-2020) created from a bibliometric study as the basis of our search for studies assessing discordance (called discordant reviews). This bibliometric study searched MEDLINE (Ovid), Epistemonikos and Cochrane Database of Systematic Reviews for overviews. We will include any study using Jadad (1997) or another method to assess discordance. The first 30 studies screened at the full-text stage by two independent reviewers will be included. We will replicate the authors' Jadad assessments. We will compare our outcomes qualitatively and evaluate the differences between our Jadad assessment of discordance and the authors' assessment. ETHICS AND DISSEMINATION No ethics approval was required as no human subjects were involved. In addition to publishing in an open-access journal, we will disseminate evidence summaries through formal and informal conferences, academic websites, and across social media platforms. This is the first study to comprehensively evaluate and replicate Jadad algorithm assessments of discordance across multiple SRs.
Collapse
Affiliation(s)
- Carole Lunny
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, Ontario, Canada
- Epidemiology Division and Institute for Health Policy, Management, and Evaluation, Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada
| | - Sai Surabi Thirugnanasampanthar
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, Ontario, Canada
- Epidemiology Division and Institute for Health Policy, Management, and Evaluation, Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada
| | - Salmaan Kanji
- Department of Pharmacy, The Ottawa Hospital, Ottawa, Ontario, Canada
- Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada
| | - Nicola Ferri
- Department of Biomedical and Neuromotor Sciences, University of Bologna, Bologna, Italy
| | | | - Dawid Pieper
- Institute for Research in Operative Medicine (IFOM), Witten/Herdecke University, Ostmerheimer Str. 200, Cologne, Germany
| | - Sara Tasnim
- Anesthesiology, Pharmacology & Therapeutics, Faculty of Medicine, University of British Columbia, Vancouver, British Columbia, Canada
| | - Harrison Nelson
- Queen's University Faculty of Health Sciences, Kingston, Ontario, Canada
| | - Emma Reid
- Nova Scotia Health Authority, Halifax, Nova Scotia, Canada
| | | | - Banveer Kalkat
- Faculty of Pharmaceutical Sciences, The University of British Columbia, Vancouver, British Columbia, Canada
| | - Yuan Chi
- Yealth Network, Beijing Yealth Technology Co., Ltd, Beijing, China
| | - Jacqueline Thompson
- University of Birmingham Institute of Applied Health Research, Birmingham, UK
| | - Reema Abdoulrezzak
- Faculty of Pharmaceutical Sciences, The University of British Columbia, Vancouver, British Columbia, Canada
| | - Di Wen Wendy Zheng
- Faculty of Pharmaceutical Sciences, The University of British Columbia, Vancouver, British Columbia, Canada
| | - Lindy Pangka
- Faculty of Pharmaceutical Sciences, The University of British Columbia, Vancouver, British Columbia, Canada
| | - Dian Xin Ran Wang
- Faculty of Pharmaceutical Sciences, The University of British Columbia, Vancouver, British Columbia, Canada
| | - Parisa Safavi
- Faculty of Pharmaceutical Sciences, The University of British Columbia, Vancouver, British Columbia, Canada
| | - Anmol Sooch
- Faculty of Pharmaceutical Sciences, The University of British Columbia, Vancouver, British Columbia, Canada
| | - Kevin Kang
- Faculty of Pharmaceutical Sciences, The University of British Columbia, Vancouver, British Columbia, Canada
| | - Sera Whitelaw
- Faculty of Medicine and Health Sciences, McGill University, Montreal, Quebec, Canada
| | - Andrea C Tricco
- Knowledge Translation Program, Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, Ontario, Canada
- Epidemiology Division and Institute for Health Policy, Management, and Evaluation, Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada
| |
Collapse
|