1
|
Doleman B, Mathiesen O, Jakobsen JC, Sutton AJ, Freeman S, Lund JN, Williams JP. Methodologies for systematic reviews with meta-analysis of randomised clinical trials in pain, anaesthesia, and perioperative medicine. Br J Anaesth 2021; 126:903-911. [PMID: 33558052 DOI: 10.1016/j.bja.2021.01.004] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/09/2020] [Revised: 12/16/2020] [Accepted: 01/07/2021] [Indexed: 02/07/2023] Open
Abstract
Systematic reviews and meta-analyses (SRMAs) are increasing in popularity, but should they be used to inform clinical decision-making in anaesthesia? We present evidence that the certainty of evidence from SRMAs in anaesthesia (and in general) may be unacceptably low because of risks of bias exaggerating treatment effects, unexplained heterogeneity reducing certainty in estimates, random errors, and widespread prevalence of publication bias. We also present the latest methodological advances to help improve the certainty of evidence from SRMAs. The target audience includes both review authors and practising clinicians to help with SRMA appraisal. Issues discussed include minimising risks of bias from included trials, trial sequential analysis to reduce random error, updated methods for presenting effect estimates, and novel publication bias tests for commonly used outcome measures. These methods can help to reduce spurious conclusions on clinical significance, explain statistical heterogeneity, and reduce false positives when evaluating small-study effects. By reducing concerns in these domains of Grading of Recommendations, Assessment, Development and Evaluation, it should help improve the certainty of evidence from SRMAs used for decision-making in anaesthesia, pain, and perioperative medicine.
Collapse
Affiliation(s)
- Brett Doleman
- Department of Anaesthesia and Surgery, Graduate Entry Medicine, University of Nottingham, Nottingham, UK.
| | - Ole Mathiesen
- Department of Medicine, University of Copenhagen, Copenhagen, Denmark; Department of Anaesthesia, Zealand University Hospital, Køge, Denmark
| | - Janus C Jakobsen
- Copenhagen Trial Unit, Copenhagen, Denmark; Department of Regional Health Research, Faculty of Heath Sciences, University of Southern Denmark, Odense, Denmark
| | - Alex J Sutton
- Department of Health Sciences, University of Leicester, Leicester, UK
| | - Suzanne Freeman
- Department of Health Sciences, University of Leicester, Leicester, UK
| | - Jonathan N Lund
- Department of Anaesthesia and Surgery, Graduate Entry Medicine, University of Nottingham, Nottingham, UK
| | - John P Williams
- Department of Anaesthesia and Surgery, Graduate Entry Medicine, University of Nottingham, Nottingham, UK
| |
Collapse
|
2
|
Pussegoda K, Turner L, Garritty C, Mayhew A, Skidmore B, Stevens A, Boutron I, Sarkis-Onofre R, Bjerre LM, Hróbjartsson A, Altman DG, Moher D. Systematic review adherence to methodological or reporting quality. Syst Rev 2017; 6:131. [PMID: 28720117 PMCID: PMC5516390 DOI: 10.1186/s13643-017-0527-2] [Citation(s) in RCA: 151] [Impact Index Per Article: 21.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/07/2016] [Accepted: 06/16/2017] [Indexed: 12/12/2022] Open
Abstract
BACKGROUND Guidelines for assessing methodological and reporting quality of systematic reviews (SRs) were developed to contribute to implementing evidence-based health care and the reduction of research waste. As SRs assessing a cohort of SRs is becoming more prevalent in the literature and with the increased uptake of SR evidence for decision-making, methodological quality and standard of reporting of SRs is of interest. The objective of this study is to evaluate SR adherence to the Quality of Reporting of Meta-analyses (QUOROM) and PRISMA reporting guidelines and the A Measurement Tool to Assess Systematic Reviews (AMSTAR) and Overview Quality Assessment Questionnaire (OQAQ) quality assessment tools as evaluated in methodological overviews. METHODS The Cochrane Library, MEDLINE®, and EMBASE® databases were searched from January 1990 to October 2014. Title and abstract screening and full-text screening were conducted independently by two reviewers. Reports assessing the quality or reporting of a cohort of SRs of interventions using PRISMA, QUOROM, OQAQ, or AMSTAR were included. All results are reported as frequencies and percentages of reports and SRs respectively. RESULTS Of the 20,765 independent records retrieved from electronic searching, 1189 reports were reviewed for eligibility at full text, of which 56 reports (5371 SRs in total) evaluating the PRISMA, QUOROM, AMSTAR, and/or OQAQ tools were included. Notable items include the following: of the SRs using PRISMA, over 85% (1532/1741) provided a rationale for the review and less than 6% (102/1741) provided protocol information. For reports using QUOROM, only 9% (40/449) of SRs provided a trial flow diagram. However, 90% (402/449) described the explicit clinical problem and review rationale in the introduction section. Of reports using AMSTAR, 30% (534/1794) used duplicate study selection and data extraction. Conversely, 80% (1439/1794) of SRs provided study characteristics of included studies. In terms of OQAQ, 37% (499/1367) of the SRs assessed risk of bias (validity) in the included studies, while 80% (1112/1387) reported the criteria for study selection. CONCLUSIONS Although reporting guidelines and quality assessment tools exist, reporting and methodological quality of SRs are inconsistent. Mechanisms to improve adherence to established reporting guidelines and methodological assessment tools are needed to improve the quality of SRs.
Collapse
Affiliation(s)
- Kusala Pussegoda
- Ottawa Methods Centre, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Centre for Practice-Changing Research, Ottawa, ON, Canada
| | - Lucy Turner
- Ottawa Methods Centre, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Centre for Practice-Changing Research, Ottawa, ON, Canada
| | - Chantelle Garritty
- Ottawa Methods Centre, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Centre for Practice-Changing Research, Ottawa, ON, Canada
- Translational Research in Biomedicine (TRIBE) Program, University of Split School of Medicine, Split, Croatia
| | - Alain Mayhew
- Ottawa Methods Centre, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Centre for Practice-Changing Research, Ottawa, ON, Canada
- Bruyère Research Institute, Ottawa, ON, Canada
| | - Becky Skidmore
- Ottawa Methods Centre, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Centre for Practice-Changing Research, Ottawa, ON, Canada
| | - Adrienne Stevens
- Ottawa Methods Centre, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Centre for Practice-Changing Research, Ottawa, ON, Canada
- Translational Research in Biomedicine (TRIBE) Program, University of Split School of Medicine, Split, Croatia
| | - Isabelle Boutron
- Paris Descartes University, Centre of Research in Epidemiology and Statistics Sorbonne Paris Cité (CRESS), UMR 1153, INSERM, Paris, France
| | | | - Lise M Bjerre
- Bruyère Research Institute, Ottawa, ON, Canada
- Department of Family Medicine, University of Ottawa, Ottawa, ON, Canada
- School of Epidemiology, Public Health and Preventive Medicine, University of Ottawa, Ottawa, ON, Canada
| | - Asbjørn Hróbjartsson
- Center for Evidence-Based Medicine, University of Southern Denmark/Odense University Hospital, Odense, Denmark
| | - Douglas G Altman
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
| | - David Moher
- Centre for Journalology; Canadian EQUATOR Centre, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Centre for Practice-Changing Research, Ottawa, ON, Canada.
| |
Collapse
|
3
|
Pussegoda K, Turner L, Garritty C, Mayhew A, Skidmore B, Stevens A, Boutron I, Sarkis-Onofre R, Bjerre LM, Hróbjartsson A, Altman DG, Moher D. Identifying approaches for assessing methodological and reporting quality of systematic reviews: a descriptive study. Syst Rev 2017; 6:117. [PMID: 28629396 PMCID: PMC5477124 DOI: 10.1186/s13643-017-0507-6] [Citation(s) in RCA: 38] [Impact Index Per Article: 5.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/20/2016] [Accepted: 05/31/2017] [Indexed: 12/16/2022] Open
Abstract
BACKGROUND The methodological quality and completeness of reporting of the systematic reviews (SRs) is fundamental to optimal implementation of evidence-based health care and the reduction of research waste. Methods exist to appraise SRs yet little is known about how they are used in SRs or where there are potential gaps in research best-practice guidance materials. The aims of this study are to identify reports assessing the methodological quality (MQ) and/or reporting quality (RQ) of a cohort of SRs and to assess their number, general characteristics, and approaches to 'quality' assessment over time. METHODS The Cochrane Library, MEDLINE®, and EMBASE® were searched from January 1990 to October 16, 2014, for reports assessing MQ and/or RQ of SRs. Title, abstract, and full-text screening of all reports were conducted independently by two reviewers. Reports assessing the MQ and/or RQ of a cohort of ten or more SRs of interventions were included. All results are reported as frequencies and percentages of reports. RESULTS Of 20,765 unique records retrieved, 1189 of them were reviewed for full-text review, of which 76 reports were included. Eight previously published approaches to assessing MQ or reporting guidelines used as proxy to assess RQ were used in 80% (61/76) of identified reports. These included two reporting guidelines (PRISMA and QUOROM) and five quality assessment tools (AMSTAR, R-AMSTAR, OQAQ, Mulrow, Sacks) and GRADE criteria. The remaining 24% (18/76) of reports developed their own criteria. PRISMA, OQAQ, and AMSTAR were the most commonly used published tools to assess MQ or RQ. In conjunction with other approaches, published tools were used in 29% (22/76) of reports, with 36% (8/22) assessing adherence to both PRISMA and AMSTAR criteria and 26% (6/22) using QUOROM and OQAQ. CONCLUSIONS The methods used to assess quality of SRs are diverse, and none has become universally accepted. The most commonly used quality assessment tools are AMSTAR, OQAQ, and PRISMA. As new tools and guidelines are developed to improve both the MQ and RQ of SRs, authors of methodological studies are encouraged to put thoughtful consideration into the use of appropriate tools to assess quality and reporting.
Collapse
Affiliation(s)
- Kusala Pussegoda
- Ottawa Methods Centre, Clinical Epidemiology Program, Centre for Practice-Changing Research, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
| | - Lucy Turner
- Ottawa Methods Centre, Clinical Epidemiology Program, Centre for Practice-Changing Research, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
| | - Chantelle Garritty
- Ottawa Methods Centre, Clinical Epidemiology Program, Centre for Practice-Changing Research, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
- Translational Research in Biomedicine (TRIBE) Program, University of Split School of Medicine, Split, Croatia
| | - Alain Mayhew
- Ottawa Methods Centre, Clinical Epidemiology Program, Centre for Practice-Changing Research, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
| | - Becky Skidmore
- Ottawa Methods Centre, Clinical Epidemiology Program, Centre for Practice-Changing Research, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
| | - Adrienne Stevens
- Ottawa Methods Centre, Clinical Epidemiology Program, Centre for Practice-Changing Research, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
- Translational Research in Biomedicine (TRIBE) Program, University of Split School of Medicine, Split, Croatia
| | - Isabelle Boutron
- INSERM, UMR 1153, Centre of Research in Epidemiology and Statistics Sorbonne Paris Cité, University Paris Descartes, Paris, France
| | - Rafael Sarkis-Onofre
- Graduate Program in Dentistry, Federal University of Pelotas, Pelotas, RS, Brazil
| | - Lise M Bjerre
- Department of Family Medicine, University of Ottawa, Ottawa, Ontario, Canada
- Bruyère Research Institute, Ottawa, Ontario, Canada
- School of Epidemiology, Public Health and Preventive Medicine, University of Ottawa, Ottawa, Ontario, Canada
| | - Asbjørn Hróbjartsson
- Center for Evidence-Based Medicine, University of Southern Denmark & Odense University Hospital, Odense, Denmark
| | - Douglas G Altman
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology & Musculoskeletal Sciences, University of Oxford, Oxford, UK
| | - David Moher
- Centre for Journalology, Canadian EQUATOR Centre, Clinical Epidemiology Program, Centre for Practice-Changing Research, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada.
| |
Collapse
|
4
|
Oh JH, Shin WJ, Park S, Chung JS. Reporting and methodologic evaluation of meta-analyses published in the anesthesia literature according to AMSTAR and PRISMA checklists: a preliminary study. Korean J Anesthesiol 2017; 70:446-455. [PMID: 28794841 PMCID: PMC5548948 DOI: 10.4097/kjae.2017.70.4.446] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2016] [Revised: 03/03/2017] [Accepted: 03/04/2017] [Indexed: 11/10/2022] Open
Abstract
BACKGROUND There have been few recent reports on the methodological quality of meta-analysis, despite the enormous number of studies using meta-analytic techniques in the field of anesthesia. The purpose of this study was to evaluate the quality of meta-analyses and systematic reviews according to the Assessment of Multiple Systematic Reviews (AMSTAR) and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines in the anesthesia literature. METHODS A search was conducted to identify all meta-analyses ever been published in the British Journal of Anaesthesia (BJA), Anaesthesia, and Korean Journal of Anesthesiology (KJA) between Jan. 01, 2004 and Nov. 31, 2016. We aimed to apply the AMSTAR and PRISMA checklists to all published meta-analyses. RESULTS We identified 121 meta-analyses in the anesthesia literature from January 2004 through the end of November 2016 (BJA; 75, Anaesthesia; 43, KJA; 3). The number of studies published and percentage of 'Yes' responses for meta-analysis articles published after the year 2010 was significantly increased compared to that of studies published before the year 2009 (P = 0.014 for Anaesthesia). In the anesthesia literature as a whole, participation of statisticians as authors statistically improved average scores of PRISMA items (P = 0.004) especially in the BJA (P = 0.003). CONCLUSIONS Even though there is little variability in the reporting and methodology of meta-analysis in the anesthesia literature, significant quality improvement in the reporting was observed in the Anaesthesia by applying the PRISMA checklist. Participation of a statistician as an author improved the reporting quality of the meta-analysis.
Collapse
Affiliation(s)
- Jae Hoon Oh
- Department of Anesthesiology and Pain Medicine, Hanyang University College of Medicine, Seoul, Korea
| | - Woo Jong Shin
- Department of Anesthesiology and Pain Medicine, Hanyang University College of Medicine, Seoul, Korea
| | - Suin Park
- Department of Anesthesiology and Pain Medicine, Hanyang University College of Medicine, Seoul, Korea
| | - Jae Soon Chung
- Department of Anesthesiology and Pain Medicine, Hanyang University College of Medicine, Seoul, Korea
| |
Collapse
|
5
|
Organizational Strategies for Building Capacity in Evidence-Based Oncology Nursing Practice. Nurs Clin North Am 2017; 52:149-158. [DOI: 10.1016/j.cnur.2016.10.001] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
|
6
|
Golder S, Loke YK, Zorzela L. Comparison of search strategies in systematic reviews of adverse effects to other systematic reviews. Health Info Libr J 2014; 31:92-105. [PMID: 24754741 DOI: 10.1111/hir.12041] [Citation(s) in RCA: 29] [Impact Index Per Article: 2.9] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2012] [Accepted: 07/04/2013] [Indexed: 12/20/2022]
Abstract
BACKGROUND Research indicates that the methods used to identify data for systematic reviews of adverse effects may need to differ from other systematic reviews. OBJECTIVES To compare search methods in systematic reviews of adverse effects with other reviews. METHODS The search methodologies in 849 systematic reviews of adverse effects were compared with other reviews. RESULTS Poor reporting of search strategies is apparent in both systematic reviews of adverse effects and other types of systematic reviews. Systematic reviews of adverse effects are less likely to restrict their searches to MEDLINE or include only randomised controlled trials (RCTs). The use of other databases is largely dependent on the topic area and the year the review was conducted, with more databases searched in more recent reviews. Adverse effects search terms are used by 72% of reviews and despite recommendations only two reviews report using floating subheadings. CONCLUSIONS The poor reporting of search strategies in systematic reviews is universal, as is the dominance of searching MEDLINE. However, reviews of adverse effects are more likely to include a range of study designs (not just RCTs) and search beyond MEDLINE.
Collapse
Affiliation(s)
- Su Golder
- Centre for Reviews and Dissemination (CRD), University of York, York, UK
| | | | | |
Collapse
|
7
|
Klimo P, Thompson CJ, Ragel BT, Boop FA. Methodology and reporting of meta-analyses in the neurosurgical literature. J Neurosurg 2014; 120:796-810. [PMID: 24460488 DOI: 10.3171/2013.11.jns13195] [Citation(s) in RCA: 43] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/29/2022]
Abstract
OBJECT Neurosurgeons are inundated with vast amounts of new clinical research on a daily basis, making it difficult and time-consuming to keep up with the latest literature. Meta-analysis is an extension of a systematic review that employs statistical techniques to pool the data from the literature in order to calculate a cumulative effect size. This is done to answer a clearly defined a priori question. Despite their increasing popularity in the neurosurgery literature, meta-analyses have not been scrutinized in terms of reporting and methodology. METHODS The authors performed a literature search using PubMed/MEDLINE to locate all meta-analyses that have been published in the JNS Publishing Group journals (Journal of Neurosurgery, Journal of Neurosurgery: Pediatrics, Journal of Neurosurgery: Spine, and Neurosurgical Focus) or Neurosurgery. Accepted checklists for reporting (PRISMA) and methodology (AMSTAR) were applied to each meta-analysis, and the number of items within each checklist that were satisfactorily fulfilled was recorded. The authors sought to answer 4 specific questions: Are meta-analyses improving 1) with time; 2) when the study met their definition of a meta-analysis; 3) when clinicians collaborated with a potential expert in meta-analysis; and 4) when the meta-analysis was the only focus of the paper? RESULTS Seventy-two meta-analyses were published in the JNS Publishing Group journals and Neurosurgery between 1990 and 2012. The number of published meta-analyses has increased dramatically in the last several years. The most common topics were vascular, and most were based on observational studies. Only 11 papers were prepared using an established checklist. The average AMSTAR and PRISMA scores (proportion of items satisfactorily fulfilled divided by the total number of eligible items in the respective instrument) were 31% and 55%, respectively. Major deficiencies were identified, including the lack of a comprehensive search strategy, study selection and data extraction, assessment of heterogeneity, publication bias, and study quality. Almost one-third of the papers did not meet our basic definition of a meta-analysis. The quality of reporting and methodology was better 1) when the study met our definition of a meta-analysis; 2) when one or more of the authors had experience or expertise in conducting a meta-analysis; 3) when the meta-analysis was not conducted alongside an evaluation of the authors' own data; and 4) in more recent studies. CONCLUSIONS Reporting and methodology of meta-analyses in the neurosurgery literature is excessively variable and overall poor. As these papers are being published with increasing frequency, neurosurgical journals need to adopt a clear definition of a meta-analysis and insist that they be created using checklists for both reporting and methodology. Standardization will ensure high-quality publications.
Collapse
Affiliation(s)
- Paul Klimo
- Semmes-Murphey Neurologic & Spine Institute
| | | | | | | |
Collapse
|
8
|
Kitsiou S, Paré G, Jaana M. Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality. J Med Internet Res 2013; 15:e150. [PMID: 23880072 PMCID: PMC3785977 DOI: 10.2196/jmir.2770] [Citation(s) in RCA: 82] [Impact Index Per Article: 7.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/13/2013] [Revised: 07/10/2013] [Accepted: 07/10/2013] [Indexed: 01/13/2023] Open
Abstract
BACKGROUND Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases have increased over the past decade and become increasingly important to a wide range of clinicians, policy makers, and other health care stakeholders. While a few criticisms about their methodological rigor and synthesis approaches have recently appeared, no formal appraisal of their quality has been conducted yet. OBJECTIVE The primary aim of this critical review was to evaluate the methodology, quality, and reporting characteristics of prior reviews that have investigated the effects of home telemonitoring interventions in the context of chronic diseases. METHODS Ovid MEDLINE, the Database of Abstract of Reviews of Effects (DARE), and Health Technology Assessment Database (HTA) of the Cochrane Library were electronically searched to find relevant systematic reviews, published between January 1966 and December 2012. Potential reviews were screened and assessed for inclusion independently by three reviewers. Data pertaining to the methods used were extracted from each included review and examined for accuracy by two reviewers. A validated quality assessment instrument, R-AMSTAR, was used as a framework to guide the assessment process. RESULTS Twenty-four reviews, nine of which were meta-analyses, were identified from more than 200 citations. The bibliographic search revealed that the number of published reviews has increased substantially over the years in this area and although most reviews focus on studying the effects of home telemonitoring on patients with congestive heart failure, researcher interest has extended to other chronic diseases as well, such as diabetes, hypertension, chronic obstructive pulmonary disease, and asthma. Nevertheless, an important number of these reviews appear to lack optimal scientific rigor due to intrinsic methodological issues. Also, the overall quality of reviews does not appear to have improved over time. While several criteria were met satisfactorily by either all or nearly all reviews, such as the establishment of an a priori design with inclusion and exclusion criteria, use of electronic searches on multiple databases, and reporting of studies characteristics, there were other important areas that needed improvement. Duplicate data extraction, manual searches of highly relevant journals, inclusion of gray and non-English literature, assessment of the methodological quality of included studies and quality of evidence were key methodological procedures that were performed infrequently. Furthermore, certain methodological limitations identified in the synthesis of study results have affected the results and conclusions of some reviews. CONCLUSIONS Despite the availability of methodological guidelines that can be utilized to guide the proper conduct of systematic reviews and meta-analyses and eliminate potential risks of bias, this knowledge has not yet been fully integrated in the area of home telemonitoring. Further efforts should be made to improve the design, conduct, reporting, and publication of systematic reviews and meta-analyses in this area.
Collapse
Affiliation(s)
- Spyros Kitsiou
- Canada Research Chair in Information Technology in Health Care, HEC Montreal, Montreal, QC, Canada.
| | | | | |
Collapse
|
9
|
Golder S, Loke YK, Zorzela L. Some improvements are apparent in identifying adverse effects in systematic reviews from 1994 to 2011. J Clin Epidemiol 2013; 66:253-60. [PMID: 23347849 DOI: 10.1016/j.jclinepi.2012.09.013] [Citation(s) in RCA: 26] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/08/2011] [Revised: 09/14/2012] [Accepted: 09/19/2012] [Indexed: 11/19/2022]
Abstract
OBJECTIVE An increasing amount of research and guidelines has been published on search methodology and the reporting of search strategies in systematic reviews. This research assessed whether this has lead to any improvements in the reporting and quality of searching in systematic reviews of adverse effects. STUDY DESIGN AND SETTING All records within Cochrane Database of Systematic Reviews and Database of Abstracts of Reviews of Effects were scanned for systematic reviews of adverse effects. Data were then extracted on the methods used for information retrieval in these reviews and a descriptive analysis conducted by publication year. RESULTS A total of 849 reviews published from 1994 to 2011 met the inclusion criteria. There has been a significant increase (P<0.001) in the number of adverse effects reviews per year from 1994 (n=5) to 2010 (n=104). Some improvements were apparent, such as an increase in the number of databases searched and fewer date and language restrictions applied. However, there has been an increase in reviews limited to data from randomized controlled trials, whereas the reporting of search strategies could still be improved further, with only 9% (74/849) of the reviews reporting reproducible searches. CONCLUSION Some improvements in searching systematic reviews of adverse effects are apparent; however, poor reporting of search strategies remains a great obstacle to readers.
Collapse
Affiliation(s)
- Su Golder
- Centre for Reviews and Dissemination, University of York, York YO10 5DD, UK.
| | | | | |
Collapse
|
10
|
Evidence-Based Cancer Nursing. Cancer Nurs 2013; 36:1-2. [DOI: 10.1097/ncc.0b013e3182578a14] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
|
11
|
Calvache JA, Chaparro LE, Chaves A, Delgado MB, Fonseca N, Montes FR, Moyano JR, Rubio J. Estrategias y obstáculos para el desarrollo de la investigación en programas de anestesiología: documento de consenso en Colombia. COLOMBIAN JOURNAL OF ANESTHESIOLOGY 2012. [DOI: 10.1016/j.rca.2012.07.001] [Citation(s) in RCA: 11] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022] Open
|
12
|
Calvache JA, Chaparro LE, Chaves A, Delgado MB, Fonseca N, Montes FR, Moyano JR, Rubio J. Strategies and obstacles to research development in anesthesiology programs: Consensus document in Colombia. COLOMBIAN JOURNAL OF ANESTHESIOLOGY 2012. [DOI: 10.1016/j.rcae.2012.05.008] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022] Open
|
13
|
Hartog CS, Skupin H, Natanson C, Sun J, Reinhart K. Systematic analysis of hydroxyethyl starch (HES) reviews: proliferation of low-quality reviews overwhelms the results of well-performed meta-analyses. Intensive Care Med 2012; 38:1258-71. [PMID: 22790311 PMCID: PMC3783958 DOI: 10.1007/s00134-012-2614-0] [Citation(s) in RCA: 31] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/12/2012] [Accepted: 05/20/2012] [Indexed: 01/09/2023]
Abstract
PURPOSE Hydroxyethyl starch (HES) is a synthetic colloid used widely for resuscitation despite the availability of safer, less costly fluids. Numerous HES reviews have been published that may have influenced clinicians' practice. We have therefore examined the relationship between the methodological quality of published HES reviews, authors' potential conflicts of interest (pCOI) and the recommendations made. METHODS Systematic analysis of reviews on HES use. RESULTS Between 1975 and 2010, 165 reviews were published containing recommendations for or against HES use. From the 1990s onwards, favorable reviews increased from two to eight per year and HES's share of the artificial colloid market tripled from 20 to 60 %. Only 7 % (12/165) of these reviews of HES use contained meta-analyses; these 7 % had higher Overview Quality Assessment Questionnaire (OQAQ) scores [median (range) 6.5 (3-7)] than reviews without meta-analysis [2 (1-4); p < 0.001]. The rates of recommending against HES use are 83 % (10/12) in meta-analyses and 20 % (31/153) in reviews without meta-analysis (p < 0.0001). Fourteen authors published the majority (70/124) of positive reviews, and ten of these 14 had or have since developed a pCOI with various manufacturers of HES. CONCLUSIONS Low-quality HES reviews reached different conclusions than high-quality meta-analyses from independent entities, such as Cochrane Reviews. The majority of these low-quality positive HES reviews were written by a small group of authors, most of whom had or have since established ties to industry. The proliferation of positive HES reviews has been associated with increased utilization of an expensive therapy despite the lack of evidence for meaningful clinical benefit and increased risks. Clinicians need to be more informed that marketing efforts are potentially influencing scientific literature.
Collapse
Affiliation(s)
- Christiane S. Hartog
- Department of Anesthesiology and Intensive Care Medicine, Jena University Hospital, Friedrich-Schiller-University, Erlanger Allee 101, 07747 Jena, Germany
| | - Helga Skupin
- Department of Anesthesiology and Intensive Care Medicine, Jena University Hospital, Friedrich-Schiller-University, Erlanger Allee 101, 07747 Jena, Germany
| | - Charles Natanson
- Critical Care Medicine Department, Clinical Center, National Institutes of Health, Bethesda, MD, USA
| | - Junfeng Sun
- Critical Care Medicine Department, Clinical Center, National Institutes of Health, Bethesda, MD, USA
| | - Konrad Reinhart
- Department of Anesthesiology and Intensive Care Medicine, Jena University Hospital, Friedrich-Schiller-University, Erlanger Allee 101, 07747 Jena, Germany
| |
Collapse
|
14
|
Two decades of exceptional achievements: does the evidence support nurses to favour Cochrane systematic reviews over other systematic reviews? Int J Nurs Stud 2012; 49:773-4. [PMID: 22698732 DOI: 10.1016/j.ijnurstu.2012.05.007] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/22/2012] [Accepted: 05/22/2012] [Indexed: 11/20/2022]
|
15
|
Strategies and obstacles to research development in anesthesiology programs: Consensus document in Colombia☆. COLOMBIAN JOURNAL OF ANESTHESIOLOGY 2012. [DOI: 10.1097/01819236-201240040-00004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022] Open
|
16
|
Evidenzbasierte Anästhesiologie. Anaesthesist 2011; 60:407-10, 412-3. [DOI: 10.1007/s00101-011-1895-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
|
17
|
Hessel EA, Levy JH. Guidelines for Perioperative Blood Transfusion and Conservation in Cardiac Surgery. Anesth Analg 2010; 111:1555-9. [DOI: 10.1213/ane.0b013e3181fbb386] [Citation(s) in RCA: 13] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
|
18
|
Macario A, Royal MA. A literature review of randomized clinical trials of intravenous acetaminophen (paracetamol) for acute postoperative pain. Pain Pract 2010; 11:290-6. [PMID: 21114616 DOI: 10.1111/j.1533-2500.2010.00426.x] [Citation(s) in RCA: 106] [Impact Index Per Article: 7.6] [Reference Citation Analysis] [Abstract] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Abstract
INTRODUCTION This study's objective was to systematically review the literature to assess analgesic outcomes of intravenous (IV) acetaminophen for acute postoperative pain in adults. METHODS We searched Medline and the Cochrane library (January 1, 2000 to January 17, 2010, date of last search) for prospective, randomized, controlled trials (RCTs) of IV acetaminophen vs. either an active comparator or placebo. RESULTS Sixteen articles from 9 countries published between 2005 and 2010 met inclusion criteria and had a total of 1,464 patients. Median sample size=54 patients (range 25 to 165) and median follow-up=1 day (range 1 hour to 7 days). Four of the 16 articles had 3 arms in the study. One article had 4 arms. As a result, 22 study comparisons were analyzed: IV acetaminophen to an active comparator (n=8 studies) and IV acetaminophen to placebo (n=14 studies). The RCTs were of high methodological quality with Jadad median score=5. In 7 of 8 active comparator studies (IV parecoxib [n=3 studies], IV metamizol [n=4], oral ibuprofen [n=1]), IV acetaminophen had similar analgesic outcomes as the active comparator. Twelve of the 14 placebo studies found that IV acetaminophen patients had improved analgesia. Ten of those 14 studies reported less opioid consumption, a lower percentage of patients rescuing, or a longer time to first rescue with IV acetaminophen. Formal meta-analysis pooling was not performed because the studies had different primary end points, and the IV acetaminophen dosing regimens varied in dose, and duration and timing. CONCLUSION In aggregate, these data indicate that IV acetaminophen is an effective analgesic across a variety of surgical procedures.
Collapse
Affiliation(s)
- Alex Macario
- Department of Anesthesia, Stanford University School of Medicine, California 94305-5640, USA.
| | | |
Collapse
|
19
|
Suebnukarn S, Ngamboonsirisingh S, Rattanabanlang A. A Systematic Evaluation of the Quality of Meta-analyses in Endodontics. J Endod 2010; 36:602-8. [DOI: 10.1016/j.joen.2009.12.019] [Citation(s) in RCA: 41] [Impact Index Per Article: 2.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/15/2009] [Accepted: 12/16/2009] [Indexed: 01/08/2023]
|
20
|
Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg 2010; 8:336-41. [PMID: 20171303 DOI: 10.1016/j.ijsu.2010.02.007] [Citation(s) in RCA: 7532] [Impact Index Per Article: 538.0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/17/2010] [Indexed: 10/19/2022]
|
21
|
Lundh A, Knijnenburg SL, Jørgensen AW, van Dalen EC, Kremer LC. Quality of systematic reviews in pediatric oncology – A systematic review. Cancer Treat Rev 2009; 35:645-52. [DOI: 10.1016/j.ctrv.2009.08.010] [Citation(s) in RCA: 55] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/16/2009] [Revised: 08/21/2009] [Accepted: 08/21/2009] [Indexed: 01/10/2023]
|
22
|
Minelli C, Thompson JR, Abrams KR, Thakkinstian A, Attia J. The quality of meta-analyses of genetic association studies: a review with recommendations. Am J Epidemiol 2009; 170:1333-43. [PMID: 19901000 PMCID: PMC2778766 DOI: 10.1093/aje/kwp350] [Citation(s) in RCA: 36] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/27/2022] Open
Abstract
Although there has been a rapid rise in the publication of meta-analyses of genetic association studies, little is known about their methodological quality. The authors reviewed the quality of 120 randomly selected genetic meta-analyses published between 2005 and 2007. Data extracted included issues of general relevance and other issues specific to genetic epidemiology. Quality was markedly poorer in the 26% of the meta-analyses that accompanied a report on a primary study. Such meta-analyses were predominantly published in specialist journals, and their quality was positively associated with the impact factor of the journal. Among the meta-analyses that did not accompany a primary study, Human Genome Epidemiology reviews tended to score better than the others, although the comparison was limited by relatively small numbers. Comparison of the overall quality with that of genetic meta-analyses published before 2000 showed improvement in both conduct and reporting. However, the quality of the handling of specific genetic issues remains disappointingly low. For a few key general quality issues, the authors compared their findings with findings in other fields of medicine and found that general quality was similar. On the basis of this review, the authors provide practical recommendations for the conduct and reporting of genetic meta-analyses.
Collapse
Affiliation(s)
- Cosetta Minelli
- Institute of Genetic Medicine, EURAC Research, Viale Druso 1, 39100 Bolzano, Italy.
| | | | | | | | | |
Collapse
|
23
|
Moher D, Liberati A, Tetzlaff J, Altman DG. Reprint--preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Phys Ther 2009. [PMID: 19723669 DOI: 10.1093/ptj/89.9.873] [Citation(s) in RCA: 1240] [Impact Index Per Article: 82.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/07/2023]
Abstract
Editor's Note: PTJ's Editorial Board has adopted PRISMA to help PTJ better communicate research to physical therapists. For more, read Chris Maher's editorial starting on page 870.
Membership of the PRISMA Group is provided in the Acknowledgments.
This article has been reprinted with permission from the Annals of Internal Medicine from Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. Ann Intern Med. Available at: http://www.annals.org/cgi/content/full/151/4/264. The authors jointly hold copyright of this article. This article has also been published in PLoS Medicine, BMJ, Journal of Clinical Epidemiology, and Open Medicine.
Copyright © 2009 Moher et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Collapse
|
24
|
Abstract
Meta-analyses are seen as representing the pinnacle of a hierarchy of evidence used to inform clinical practice. Therefore, the potential importance of differences in the rigor with which they are conducted and reported warrants consideration. In this review, we use standardized instruments to describe the scientific and reporting quality of meta-analyses of randomized controlled trials of the treatment of anxiety disorders. We also use traditional and novel metrics of article impact to assess the influence of meta-analyses across a range of research fields in the anxiety disorders. Overall, although the meta-analyses that we examined had some flaws, their quality of reporting was generally acceptable. Neither the scientific nor reporting quality of the meta-analyses was predicted by any of the impact metrics. The finding that treatment meta-analyses were cited less frequently than quantitative reviews of studies in current "hot spots" of research (ie, genetics, imaging) points to the multifactorial nature of citation patterns. A list of the meta-analyses included in this review is available on an evidence-based website of anxiety and trauma-related disorders.
Collapse
|
25
|
Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol 2009; 62:1006-12. [PMID: 19631508 DOI: 10.1016/j.jclinepi.2009.06.005] [Citation(s) in RCA: 8573] [Impact Index Per Article: 571.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2008] [Accepted: 06/22/2009] [Indexed: 02/06/2023]
Affiliation(s)
- David Moher
- Ottawa Methods Centre, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada.
| | | | | | | | | |
Collapse
|
26
|
Affiliation(s)
- David Moher
- Ottawa Methods Centre, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada.
| | | | | | | |
Collapse
|
27
|
Al Faleh K, Al-Omran M. Reporting and methodologic quality of Cochrane Neonatal review group systematic reviews. BMC Pediatr 2009; 9:38. [PMID: 19534780 PMCID: PMC2709610 DOI: 10.1186/1471-2431-9-38] [Citation(s) in RCA: 29] [Impact Index Per Article: 1.9] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/02/2008] [Accepted: 06/17/2009] [Indexed: 11/21/2022] Open
Abstract
Background The Cochrane Neonatal Review Group (CNRG) has achieved a lot with limited resources in producing high quality systematic reviews to assist clinicians in evidence-based decision-making. A formal assessment of published CNRG systematic reviews has not been undertaken; we sought to provide a comprehensive assessment of the quality of systematic reviews (both methodologic and reporting quality) published in CNRG. Methods We selected a random sample of published CNRG systematic reviews. Items of the QUOROM statement were utilized to assess quality of reporting, while items and total scores of the Oxman-Guyatt Overview Quality Assessment Questionnaire (OQAQ) were used to assess methodologic quality. Two reviewers independently extracted data and assessed quality. A Student t-test was used to compare quality scores pre- and post-publication of the QUOROM statement. Results Sixty-one systematic reviews were assessed. Overall, the included reviews had good quality with minor flaws based on OQAQ total scores (mean, 4.5 [0.9]; 95% CI, 4.27–4.77). However, room for improvement was noted in some areas, such as the title, abstract reporting, a priori plan for heterogeneity assessment and how to handle heterogeneity in case it exists, and assessment of publication bias. In addition, reporting of agreement among reviewers, documentation of trials flow, and discussion of possible biases were addressed in the review process. Reviews published post the QUOROM statement had a significantly higher quality scores. Conclusion The systematic reviews published in the CNRG are generally of good quality with minor flaws. However, efforts should be made to improve the quality of reports. Readers must continue to assess the quality of published reports on an individual basis prior to implementing the recommendations.
Collapse
Affiliation(s)
- Khalid Al Faleh
- Department of Pediatrics, King Khalid University Hospital, College of Medicine, King Saud University, Riyadh, Saudi Arabia.
| | | |
Collapse
|
28
|
Improving safety in the operating room: a systematic literature review of retained surgical sponges. Curr Opin Anaesthesiol 2009; 22:207-14. [DOI: 10.1097/aco.0b013e328324f82d] [Citation(s) in RCA: 113] [Impact Index Per Article: 7.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/31/2023]
|
29
|
Feng ZZ, Shi J, Zhao XW, Xu ZF. Meta-Analysis of On-Pump and Off-Pump Coronary Arterial Revascularization. Ann Thorac Surg 2009; 87:757-65. [DOI: 10.1016/j.athoracsur.2008.11.042] [Citation(s) in RCA: 38] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/04/2008] [Revised: 11/11/2008] [Accepted: 11/17/2008] [Indexed: 10/21/2022]
|
30
|
Macario A, Dexter F, Sypal J, Cosgriff N, Heniford BT. Operative Time and Other Outcomes of the Electrothermal Bipolar Vessel Sealing System (LigaSure™) Versus Other Methods for Surgical Hemostasis: A Meta-Analysis. Surg Innov 2008; 15:284-91. [DOI: 10.1177/1553350608324933] [Citation(s) in RCA: 70] [Impact Index Per Article: 4.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
A meta-analysis was performed of 29 prospective, randomized trials (published January 1, 2000, to August 14, 2007) comparing an electrothermal bipolar vessel sealing system (EBVS-LigaSure™, Covidien) (total n = 1107 patients) with either clamping with suture ligation/ electrocauterization (n = 1079 patients) or ultrasonic energy (eg, Harmonic Scalpel®, Johnson & Johnson). Hemorrhoidectomy (12 articles), hysterectomy (4 articles), and thyroidectomy (3 articles) were the most common procedures. For 15 of 26 studies reporting standard deviations, the normalized mean operative time reduction for EBVS equaled 28% (95% confidence interval [CI] 18%-39%, P < .0001) compared with conventional surgical hemostasis. Operative time was reduced with EBVS in 24 of 26 studies ( P < .0001). EBVS was associated with 43 mL (95% CI 14-73 mL, P = .0021) less blood loss, fewer complications (odds ratio 0.66, 95% CI 0.47-0.92, P = .02), and mean reduction in postoperative pain of 2.8 units (95% CI 1.5-4.1, P < .0001). Five studies used ultrasonic energy as the comparator, but none reported standard deviation so data could not be pooled.
Collapse
Affiliation(s)
- Alex Macario
- Departments of Anesthesia and Health Research & Policy, Stanford University School of Medicine, Stanford, California,
| | - Franklin Dexter
- Departments of Anesthesia and Health Management & Policy, University of Iowa
| | | | - Ned Cosgriff
- Covidien, Energy-based Devices, Boulder, Colorado
| | - B. Todd Heniford
- Division of Gastrointestinal and Minimally Invasive Surgery, Carolinas Medical Center, Charlotte, North Carolina
| |
Collapse
|
31
|
Abstract
A systematic review uses an explicitly defined process to comprehensively identify all studies pertaining to a specific focused question, appraise the methods of the studies, summarize the results, identify reasons for different findings across studies, and cite limitations of current knowledge. Meta-analyses usually combine aggregate-level data reported in each primary study, which may provide a more precise estimate of the "true effect" than any individual study. However, the conclusions may be limited by between-trial heterogeneity, publication bias, or deficits in the conduct or reporting of individual primary studies.
Collapse
|
32
|
Boluyt N, van der Lee JH, Moyer VA, Brand PLP, Offringa M. State of the evidence on acute asthma management in children: a critical appraisal of systematic reviews. Pediatrics 2007; 120:1334-43. [PMID: 18055684 DOI: 10.1542/peds.2006-3381] [Citation(s) in RCA: 16] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Abstract
OBJECTIVE Our goal was to evaluate clinical, methodologic, and reporting aspects of systematic reviews on the management of acute asthma in children. METHODS We undertook a systematic review of systematic reviews on acute asthma management in children. We identified eligible reviews by searching the Cochrane Database of Systematic Reviews, the Database of Abstracts of Reviews of Effects, Medline, and Embase 1990 to March 2006. Data were extracted on clinical issues, methodologic characteristics, and results of the reviews. Methodologic quality was assessed with the Overview Quality Assessment Questionnaire and with additional questions on heterogeneity. Separate reporting on children in mixed adult-pediatric population reviews was assessed. Methodologic quality of systematic reviews published in peer-reviewed journals was compared with Cochrane reviews. RESULTS A total of 23 systematic reviews were included: 14 were published in the Cochrane Library, and 9 were published in peer-reviewed journals. Eight reviews included children only, and 15 were mixed-population reviews. The majority of reviews defined the study population as having "acute asthma" without a more precise definition, and 16 different health outcomes were reported. The overall quality according to the Overview Quality Assessment Questionnaire was good, with Cochrane reviews showing minimal flaws and journal reviews showing minor flaws (median scores: 7 vs 5). Results on children were reported separately in 8 of 15 mixed-population reviews. Clinical heterogeneity was explored in only 2 of 23 reviews, and the methods used to identify and address heterogeneity were diverse. CONCLUSIONS The methodologic quality of both the Cochrane and journal reviews on the management of acute asthma in children seems good, with Cochrane reviews being more rigorous. However, their usefulness for clinical practice is hampered by a lack of clear definitions of included populations, clinically important health outcomes, and separate reporting on children in mixed reviews. A major threat to these reviews' validity is the insufficient identification and handling of heterogeneity.
Collapse
Affiliation(s)
- Nicole Boluyt
- Emma Children's Hospital/Department of Pediatric Clinical Epidemiology, Room H3-145, Academic Medical Center, PO Box 22700, 1100 DD Amsterdam, The Netherlands.
| | | | | | | | | |
Collapse
|
33
|
Gerber S, Tallon D, Trelle S, Schneider M, Jüni P, Egger M. Bibliographic study showed improving methodology of meta-analyses published in leading journals 1993–2002. J Clin Epidemiol 2007; 60:773-80. [PMID: 17606172 DOI: 10.1016/j.jclinepi.2006.10.022] [Citation(s) in RCA: 17] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/19/2006] [Revised: 10/12/2006] [Accepted: 10/19/2006] [Indexed: 01/18/2023]
Abstract
OBJECTIVE To assess the methodology of meta-analyses published in leading general and specialist medical journals over a 10-year period. STUDY DESIGN AND SETTING Volumes 1993-2002 of four general medicine journals and four specialist journals were searched by hand for meta-analyses including at least five controlled trials. Characteristics were assessed using a standardized questionnaire. RESULTS A total of 272 meta-analyses, which included a median of 11 trials (range 5-195), were assessed. Most (81%) were published in general medicine journals. The median (range) number of databases searched increased from 1 (1-9) in 1993/1994 to 3.5 (1-21) in 2001/2002, P<0.0001. The proportion of meta-analyses including searches by hand (10% in 1993/1994, 25% in 2001/2002, P=0.005), searches of the grey literature (29%, 51%, P=0.010 by chi-square test), and of trial registers (10%, 32%, P=0.025) also increased. Assessments of the quality of trials also became more common (45%, 70%, P=0.008), including whether allocation of patients to treatment groups had been concealed (24%, 60%, P=0.001). The methodological and reporting quality was consistently higher in general medicine compared to specialist journals. CONCLUSION Many meta-analyses published in leading journals have important methodological limitations. The situation has improved in recent years but considerable room for further improvements remains.
Collapse
Affiliation(s)
- Stefan Gerber
- Department of Social and Preventive Medicine, University of Bern, Finkenhubelweg 11, CH-3012 Berne, Switzerland
| | | | | | | | | | | |
Collapse
|
34
|
Gebel K, Bauman AE, Petticrew M. The physical environment and physical activity: a critical appraisal of review articles. Am J Prev Med 2007; 32:361-369. [PMID: 17478260 DOI: 10.1016/j.amepre.2007.01.020] [Citation(s) in RCA: 136] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/05/2006] [Revised: 12/11/2006] [Accepted: 01/10/2007] [Indexed: 11/22/2022]
Abstract
BACKGROUND Over the last few years an increasing number of studies investigating the association between the physical environment and physical activity have been published. Many reviews have also summarized this emerging body of research, and such review papers are frequently used by public health policymakers and researchers themselves to inform decision making. METHODS This paper systematically appraises methodologic aspects of literature reviews examining the relationship between physical activity and the physical environment published in peer-reviewed journals between 2000 and 2005. Eleven reviews and their antecedent source papers were examined. RESULTS The majority of these reviews omitted between one third and two thirds of the studies that could have been eligible for inclusion at the time they conducted the review. Methodologic information on how the review was conducted was not always provided. Furthermore, in some cases results of a study were reported incorrectly, or physical environmental aspects were conflated with social environmental or cognitive factors. Moreover, when results were reported incorrectly, physical environmental variables were almost always reported as significantly associated with physical activity, when these associations were nonsignificant, or were not assessed as part of the primary study. CONCLUSIONS Users of reviews in this field should be aware that there are significant methodologic variations among them, and that some reviews may include only a sample of the relevant primary studies. However, this is difficult to determine given the frequent incompleteness of review method reporting. Greater standardization in the reporting of review methods may assist with future efforts to summarize studies of the relationship between physical environments and physical activity.
Collapse
Affiliation(s)
- Klaus Gebel
- Centre for Physical Activity and Health, School of Public Health, University of Sydney, Sydney, New South Wales, Australia.
| | | | | |
Collapse
|
35
|
Delaney A, Bagshaw SM, Ferland A, Laupland K, Manns B, Doig C. The quality of reports of critical care meta-analyses in the Cochrane Database of Systematic Reviews: an independent appraisal. Crit Care Med 2007; 35:589-94. [PMID: 17205029 DOI: 10.1097/01.ccm.0000253394.15628.fd] [Citation(s) in RCA: 79] [Impact Index Per Article: 4.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Abstract
OBJECTIVE To independently appraise the methodological quality of a sample of reports of meta-analyses that address critical care topics in the Cochrane Database of Systematic Reviews compared with the quality of reports published in regular journals, using a validated assessment instrument, the Overview Quality Assessment Questionnaire (OQAQ). DATA SOURCE Studies were selected from a search of MEDLINE, EMBASE, and the Cochrane Database of Systematic Reviews from 1994 to 2003, using multiple search terms for critical care and sensitive filters to identify meta-analyses. STUDY SELECTION Two authors independently selected meta-analyses that addressed topics pertinent to critical care medicine. DATA EXTRACTION Two authors independently extracted the data. The proportion of reports that met each component of the OQAQ was determined, as was the overall quality score. Meta-analyses published in the Cochrane Database of Systematic Reviews were compared with those published in regular journals. DATA SYNTHESIS There were 36 reports of meta-analyses in the Cochrane Database of Systematic Reviews and 103 reports of meta-analyses published in regular journals; 11 of these were reports of Cochrane reviews. The meta-analyses published in the Cochrane Database of Systematic Reviews were more likely to fulfill most components of the OQAQ. The median overall OQAQ scores indicated significant methodological problems in the reports regardless of the source of publication, although the reports in the Cochrane database scored higher than those in regular journals (five compared with two, p<.001). Major methodological flaws, notably failure to appropriately refer to the validity of included studies, were found in meta-analyses in both the Cochrane Database of Systematic Reviews and regular journals (44.4% and 79.3%, respectively). CONCLUSIONS Although the quality of reports of meta-analyses published in the Cochrane Database of Systematic Reviews is superior to the quality of reports of meta-analyses published in regular journals, there is significant room for improvement. Clinicians should critically appraise all reports of meta-analyses before considering the results, regardless of the source of publication.
Collapse
Affiliation(s)
- Anthony Delaney
- Department of Intensive Care Medicine, Royal North Shore Hospital, and Northern Clinical School, University of Sydney, St. Leonards, NSW, Australia
| | | | | | | | | | | |
Collapse
|
36
|
Needham DM, Desai SV, Robinson KA. Critically appraise before you believe: the quality of meta-analyses in critical care medicine. Crit Care Med 2007; 35:666-7. [PMID: 17251726 DOI: 10.1097/01.ccm.0000251635.70798.10] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
|
37
|
van de Laar FA, Akkermans RP, van Binsbergen JJ. Limited evidence for effects of diet for type 2 diabetes from systematic reviews. Eur J Clin Nutr 2007; 61:929-37. [PMID: 17251927 DOI: 10.1038/sj.ejcn.1602611] [Citation(s) in RCA: 8] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/15/2022]
Abstract
OBJECTIVE Systematic reviews are an appraised method to summarize research in a concise and transparent way, and may enable to draw conclusions beyond the sum of results of individual studies. We assessed the results, quality and external validity of systematic reviews on diet in patients with type 2 diabetes. DESIGN, SETTING, SUBJECTS We systematically searched for systematic reviews on nutritional interventions in patients with type 2 diabetes that used a reproducible search strategy in at least one major database that applied some form of quality assessment. We assessed quality and the external validity of the retrieved systematic reviews. Outcomes were defined as statistical meta-analyses or narrative results using a predefined and reproducible method. RESULTS Six systematic reviews met the inclusion criteria, investigating dietary interventions in general (n=3), chromium supplementation (n=1), fish-oil (n=1) or herbs and nutrition supplements (n=1). Quality assessment showed minimal/minor flaws in four cases and major/extensive flaws in two cases. All reviews had insufficient data needed to judge external validity. In reviews with minimal/minor flaws, we found beneficial effects of very-low-calorie diets and fish-oil supplements. However, the external validity of these results could not be assessed sufficiently. CONCLUSIONS Systematic reviews largely failed to produce knowledge beyond the sum of the original studies. Furthermore, judgment of external validity was hampered in most cases owing to missing data. To improve the quality and usefulness of systematic reviews of dietary interventions, we recommend the application of more focused research questions, but with broader inclusion criteria, for example, the use of observational studies. SPONSORSHIP Internal funding Radboud University Nijmegen Medical Centre.
Collapse
Affiliation(s)
- F A van de Laar
- Department of General Practice, Radboud University Nijmegen Medical Centre, 6500 HB Nijmegen, The Netherlands.
| | | | | |
Collapse
|
38
|
Abstract
PURPOSE OF REVIEW The purpose of this review is to present the latest knowledge and research on the definition and distribution of clinically relevant articles in anesthesia journals. It will also discuss the importance of the chosen methodology and outcome of articles. RECENT FINDINGS In the last few years, more attention has been paid to evidence-based medicine in anesthesia. Several articles on the subject have focused on the need to base clinical decisions on sound research employing both methodological rigor and clinically relevant outcomes. The number of systematic reviews in anesthesia literature is increasing as well as the focus on diminishing the number of surrogate outcomes. It has been shown that the impact factor is not a valid measure of establishing the level of clinical relevance to a journal. SUMMARY This review presents definitions of clinically relevant anesthesia articles. A clinically relevant article employs both methodological rigor and a clinically relevant outcome. The terms methodological rigor and clinical outcomes are fully discussed in the review as well as problems with journal impact factors.
Collapse
Affiliation(s)
- Jakob Lauritsen
- Department of Anaesthesiology, Herlev University Hospital, Herlev, Denmark.
| | | |
Collapse
|
39
|
Lawson ML, Pham B, Klassen TP, Moher D. Systematic reviews involving complementary and alternative medicine interventions had higher quality of reporting than conventional medicine reviews. J Clin Epidemiol 2005; 58:777-84. [PMID: 16018912 DOI: 10.1016/j.jclinepi.2004.08.022] [Citation(s) in RCA: 32] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/14/2004] [Revised: 07/26/2004] [Accepted: 08/30/2004] [Indexed: 12/01/2022]
Abstract
OBJECTIVE To compare the quality of systematic reviews reported in English and in languages other than English, and to determine whether there are differences between conventional medicine (CM) and complementary and alternative medicine (CAM) reports. STUDY DESIGN AND SETTING We used the Oxman and Guyatt (OG) scale to assess the quality of reporting in 130 systematic reviews: 50 were language-restricted, 32 were language-inclusive but only English-language (EL) trials contained (inclusive-EL), and 48 were language-inclusive and included trials published in languages other than English (inclusive-LOE). Of the 130 reviews, 105 addressed CM interventions and 25 addressed CAM interventions. RESULTS Comparison of the systematic reviews showed that the quality of reporting and reporting characteristics are not affected by inclusion or exclusion of LOE; however, the quality of reporting of systematic reviews involving CAM interventions is higher than that of reviews focusing on CM interventions. CONCLUSION Informal comparison of the OG scale with the data collected on quality assessments showed that the OG scale performs well overall but may not identify important differences in comprehensiveness of the search strategy and avoidance of bias in study selection. Further research is required to determine the best methods for assessing quality of systematic reviews and whether the effect of language restrictions is dependent on the type of intervention (CM or CAM).
Collapse
Affiliation(s)
- Margaret L Lawson
- Department of Pediatrics, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada
| | | | | | | |
Collapse
|
40
|
Dixon E, Hameed M, Sutherland F, Cook DJ, Doig C. Evaluating meta-analyses in the general surgical literature: a critical appraisal. Ann Surg 2005; 241:450-9. [PMID: 15729067 PMCID: PMC1356983 DOI: 10.1097/01.sla.0000154258.30305.df] [Citation(s) in RCA: 103] [Impact Index Per Article: 5.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
Abstract
OBJECTIVE To assess the methodologic quality of meta-analyses of general surgery topics published in peer-reviewed journals. SUMMARY BACKGROUND DATA Systematic reviews and meta-analysis are used to seek, summarize, and interpret primary studies on a given topic. Accordingly, systematic reviews and meta-analyses of high-quality primary studies may be the highest level of evidence for issues of prevention and treatment in evidence-based medicine. However, not all published meta-analyses are rigorously performed. METHODS We searched MEDLINE (from January 1, 1997, to September 1, 2002) and reference lists and solicited general surgery specialists to identify relevant meta-analyses. Inclusion criteria were use of meta-analytic methods to pool the results of primary studies in general surgery on issues of diagnosis, causation, prognosis, or treatment. Our search strategies identified 487 potentially relevant articles. After excluding articles based on a priori criteria, 51 meta-analyses fulfilled eligibility criteria. In duplicate and independently, 2 reviewers assessed the quality of these meta-analyses using a 10-item index called the Overview Quality Assessment Questionnaire. RESULTS Overall concordance between 2 independent reviewers was good (interobserver agreement 81%, and a kappa of 0.62 (95% CI 0.55-0.69). Of 51 relevant articles, 38 were published in surgical journals. Most studies had major methodologic flaws (median score of 3.3, scale of 1-7). Factors associated with low overall scientific quality included the absence of any prior meta-analyses publications by authors and meta-analyses produced by surgical department members without external collaboration. CONCLUSIONS This critical appraisal of meta-analyses published in the general surgery literature demonstrates frequent methodologic flaws. The quality of these reports limits the validity of the findings and the inferences that can be made about the primary studies reviewed. To improve the quality of future meta-analyses, we recommend following guidelines for the optimal conduct and reporting of meta-analyses in general surgery.
Collapse
Affiliation(s)
- Elijah Dixon
- Department of Surgery, University of Calgary, Calgary, Alberta, Canada
| | | | | | | | | |
Collapse
|
41
|
Abstract
Clinicians performing evidence-based anesthesia rely on anesthesia journals for clinically relevant information. The objective of this study was to analyze the proportion of clinically relevant articles in five high impact anesthesia journals. We evaluated all articles published in Anesthesiology, Anesthesia & Analgesia, British Journal of Anesthesia, Anesthesia, and Acta Anaesthesiologica Scandinavica from January to June, 2000. Articles were assessed and classified according to type, outcome, and design; 1379 articles consisting of 5468 pages were evaluated and categorized. The most common types of article were animal and laboratory research (31.2%) and randomized clinical trial (20.4%). A clinically relevant article was defined as an article that used a statistically valid method and had a clinically relevant end-point. Altogether 18.6% of the pages had as their subject matter clinically relevant trials. We compared the Journal Impact Factor (a measure of the number of citations per article in a journal) and the proportion of clinically relevant pages and found that they were inversely proportional to each other.
Collapse
Affiliation(s)
- Jakob Lauritsen
- Department of Anesthesia, Herlev University Hospital, Herlev, Denmark
| | | |
Collapse
|
42
|
Pedersen T, Møller AM, Cracknell J. The Mission of the Cochrane Anesthesia Review Group: Preparing and Disseminating Systematic Reviews of the Effect of Health Care in Anesthesiology. Anesth Analg 2002. [DOI: 10.1213/00000539-200210000-00041] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
|
43
|
Pedersen T, Møller AM, Cracknell J. The mission of the cochrane anesthesia review group: preparing and disseminating systematic reviews of the effect of health care in anesthesiology. Anesth Analg 2002; 95:1012-8, table of contents. [PMID: 12351286 DOI: 10.1097/00000539-200210000-00041] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
Abstract
IMPLICATIONS This article illustrates the basic principles of evidence-based medicine and the work within the Cochrane Collaboration and the Cochrane Anesthesia Review Group. It describes how important randomized controlled trials and systematic reviews are in providing the best evidence to answer clinically relevant questions.
Collapse
Affiliation(s)
- Tom Pedersen
- Department of Anesthesiology, Bispebjerg University Hospital, Copenhagen, Denmark.
| | | | | |
Collapse
|
44
|
Choi PTL, Galinski SE, Lucas S, Takeuchi L, Jadad AR. Examining the evidence in anesthesia literature: a survey and evaluation of obstetrical postdural puncture headache reports. Can J Anaesth 2002; 49:49-56. [PMID: 11782328 DOI: 10.1007/bf03020418] [Citation(s) in RCA: 19] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/20/2022] Open
Abstract
PURPOSE To describe a bibliographic database on the literature of postdural puncture headache (PDPH) in the obstetrical population, to describe the research architecture in this field, and to evaluate the quality of case-control studies, cohort studies, and controlled clinical trials on PDPH. METHODS Computerized bibliographic searches, citation review, and hand searches were conducted to find all relevant citations on incidence, clinical course, prevention, or treatment of PDPH in parturients. The study design and topic(s) covered by each study were evaluated. Case-control studies and cohort studies were evaluated using the Quality Index; clinical trials were evaluated using the Jadad scale. RESULTS One hundred ninety-six relevant citations were published between 1949 and 1999. Research on PDPH has been increasing rapidly with the majority of studies published in the 1990's. Incidence and prevention were the focus of over half of all citations. Optimal study designs were infrequently utilized. The methodological quality was poor for observational studies (Quality Index 10/29) and clinical trials (Jadad scale 2/5). CONCLUSION Although the amount of research on PDPH in parturients is increasing, use of optimal study designs and improvement in methodology is needed.
Collapse
Affiliation(s)
- Peter T-L Choi
- Department of Anesthesia, St. Joseph's Healthcare, Hamilton, Ontario, Canada.
| | | | | | | | | |
Collapse
|
45
|
|
46
|
Walder B, Tramèr MR. Evidence-based practice in peri-operative medicine. Best Pract Res Clin Anaesthesiol 2001. [DOI: 10.1053/bean.2002.0188] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
|
47
|
|