Examining the Evidence in Anesthesia Literature: A Critical Appraisal of Systematic Reviews

Methodologies for systematic reviews with meta-analysis of randomised clinical trials in pain, anaesthesia, and perioperative medicine

Systematic reviews and meta-analyses (SRMAs) are increasing in popularity, but should they be used to inform clinical decision-making in anaesthesia? We present evidence that the certainty of evidence from SRMAs in anaesthesia (and in general) may be unacceptably low because of risks of bias exaggerating treatment effects, unexplained heterogeneity reducing certainty in estimates, random errors, and widespread prevalence of publication bias. We also present the latest methodological advances to help improve the certainty of evidence from SRMAs. The target audience includes both review authors and practising clinicians to help with SRMA appraisal. Issues discussed include minimising risks of bias from included trials, trial sequential analysis to reduce random error, updated methods for presenting effect estimates, and novel publication bias tests for commonly used outcome measures. These methods can help to reduce spurious conclusions on clinical significance, explain statistical heterogeneity, and reduce false positives when evaluating small-study effects. By reducing concerns in these domains of Grading of Recommendations, Assessment, Development and Evaluation, it should help improve the certainty of evidence from SRMAs used for decision-making in anaesthesia, pain, and perioperative medicine.

Systematic review adherence to methodological or reporting quality

BACKGROUND

Guidelines for assessing methodological and reporting quality of systematic reviews (SRs) were developed to contribute to implementing evidence-based health care and the reduction of research waste. As SRs assessing a cohort of SRs is becoming more prevalent in the literature and with the increased uptake of SR evidence for decision-making, methodological quality and standard of reporting of SRs is of interest. The objective of this study is to evaluate SR adherence to the Quality of Reporting of Meta-analyses (QUOROM) and PRISMA reporting guidelines and the A Measurement Tool to Assess Systematic Reviews (AMSTAR) and Overview Quality Assessment Questionnaire (OQAQ) quality assessment tools as evaluated in methodological overviews.

METHODS

The Cochrane Library, MEDLINE®, and EMBASE® databases were searched from January 1990 to October 2014. Title and abstract screening and full-text screening were conducted independently by two reviewers. Reports assessing the quality or reporting of a cohort of SRs of interventions using PRISMA, QUOROM, OQAQ, or AMSTAR were included. All results are reported as frequencies and percentages of reports and SRs respectively.

RESULTS

Of the 20,765 independent records retrieved from electronic searching, 1189 reports were reviewed for eligibility at full text, of which 56 reports (5371 SRs in total) evaluating the PRISMA, QUOROM, AMSTAR, and/or OQAQ tools were included. Notable items include the following: of the SRs using PRISMA, over 85% (1532/1741) provided a rationale for the review and less than 6% (102/1741) provided protocol information. For reports using QUOROM, only 9% (40/449) of SRs provided a trial flow diagram. However, 90% (402/449) described the explicit clinical problem and review rationale in the introduction section. Of reports using AMSTAR, 30% (534/1794) used duplicate study selection and data extraction. Conversely, 80% (1439/1794) of SRs provided study characteristics of included studies. In terms of OQAQ, 37% (499/1367) of the SRs assessed risk of bias (validity) in the included studies, while 80% (1112/1387) reported the criteria for study selection.

CONCLUSIONS

Although reporting guidelines and quality assessment tools exist, reporting and methodological quality of SRs are inconsistent. Mechanisms to improve adherence to established reporting guidelines and methodological assessment tools are needed to improve the quality of SRs.

Identifying approaches for assessing methodological and reporting quality of systematic reviews: a descriptive study

BACKGROUND

The methodological quality and completeness of reporting of the systematic reviews (SRs) is fundamental to optimal implementation of evidence-based health care and the reduction of research waste. Methods exist to appraise SRs yet little is known about how they are used in SRs or where there are potential gaps in research best-practice guidance materials. The aims of this study are to identify reports assessing the methodological quality (MQ) and/or reporting quality (RQ) of a cohort of SRs and to assess their number, general characteristics, and approaches to 'quality' assessment over time.

METHODS

The Cochrane Library, MEDLINE®, and EMBASE® were searched from January 1990 to October 16, 2014, for reports assessing MQ and/or RQ of SRs. Title, abstract, and full-text screening of all reports were conducted independently by two reviewers. Reports assessing the MQ and/or RQ of a cohort of ten or more SRs of interventions were included. All results are reported as frequencies and percentages of reports.

RESULTS

Of 20,765 unique records retrieved, 1189 of them were reviewed for full-text review, of which 76 reports were included. Eight previously published approaches to assessing MQ or reporting guidelines used as proxy to assess RQ were used in 80% (61/76) of identified reports. These included two reporting guidelines (PRISMA and QUOROM) and five quality assessment tools (AMSTAR, R-AMSTAR, OQAQ, Mulrow, Sacks) and GRADE criteria. The remaining 24% (18/76) of reports developed their own criteria. PRISMA, OQAQ, and AMSTAR were the most commonly used published tools to assess MQ or RQ. In conjunction with other approaches, published tools were used in 29% (22/76) of reports, with 36% (8/22) assessing adherence to both PRISMA and AMSTAR criteria and 26% (6/22) using QUOROM and OQAQ.

CONCLUSIONS

The methods used to assess quality of SRs are diverse, and none has become universally accepted. The most commonly used quality assessment tools are AMSTAR, OQAQ, and PRISMA. As new tools and guidelines are developed to improve both the MQ and RQ of SRs, authors of methodological studies are encouraged to put thoughtful consideration into the use of appropriate tools to assess quality and reporting.

Reporting and methodologic evaluation of meta-analyses published in the anesthesia literature according to AMSTAR and PRISMA checklists: a preliminary study

BACKGROUND

There have been few recent reports on the methodological quality of meta-analysis, despite the enormous number of studies using meta-analytic techniques in the field of anesthesia. The purpose of this study was to evaluate the quality of meta-analyses and systematic reviews according to the Assessment of Multiple Systematic Reviews (AMSTAR) and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines in the anesthesia literature.

METHODS

A search was conducted to identify all meta-analyses ever been published in the British Journal of Anaesthesia (BJA), Anaesthesia, and Korean Journal of Anesthesiology (KJA) between Jan. 01, 2004 and Nov. 31, 2016. We aimed to apply the AMSTAR and PRISMA checklists to all published meta-analyses.

RESULTS

We identified 121 meta-analyses in the anesthesia literature from January 2004 through the end of November 2016 (BJA; 75, Anaesthesia; 43, KJA; 3). The number of studies published and percentage of 'Yes' responses for meta-analysis articles published after the year 2010 was significantly increased compared to that of studies published before the year 2009 (P = 0.014 for Anaesthesia). In the anesthesia literature as a whole, participation of statisticians as authors statistically improved average scores of PRISMA items (P = 0.004) especially in the BJA (P = 0.003).

CONCLUSIONS

Even though there is little variability in the reporting and methodology of meta-analysis in the anesthesia literature, significant quality improvement in the reporting was observed in the Anaesthesia by applying the PRISMA checklist. Participation of a statistician as an author improved the reporting quality of the meta-analysis.

Comparison of search strategies in systematic reviews of adverse effects to other systematic reviews

BACKGROUND

Research indicates that the methods used to identify data for systematic reviews of adverse effects may need to differ from other systematic reviews.

OBJECTIVES

To compare search methods in systematic reviews of adverse effects with other reviews.

METHODS

The search methodologies in 849 systematic reviews of adverse effects were compared with other reviews.

RESULTS

Poor reporting of search strategies is apparent in both systematic reviews of adverse effects and other types of systematic reviews. Systematic reviews of adverse effects are less likely to restrict their searches to MEDLINE or include only randomised controlled trials (RCTs). The use of other databases is largely dependent on the topic area and the year the review was conducted, with more databases searched in more recent reviews. Adverse effects search terms are used by 72% of reviews and despite recommendations only two reviews report using floating subheadings.

CONCLUSIONS

The poor reporting of search strategies in systematic reviews is universal, as is the dominance of searching MEDLINE. However, reviews of adverse effects are more likely to include a range of study designs (not just RCTs) and search beyond MEDLINE.

Methodology and reporting of meta-analyses in the neurosurgical literature

OBJECT

Neurosurgeons are inundated with vast amounts of new clinical research on a daily basis, making it difficult and time-consuming to keep up with the latest literature. Meta-analysis is an extension of a systematic review that employs statistical techniques to pool the data from the literature in order to calculate a cumulative effect size. This is done to answer a clearly defined a priori question. Despite their increasing popularity in the neurosurgery literature, meta-analyses have not been scrutinized in terms of reporting and methodology.

METHODS

The authors performed a literature search using PubMed/MEDLINE to locate all meta-analyses that have been published in the JNS Publishing Group journals (Journal of Neurosurgery, Journal of Neurosurgery: Pediatrics, Journal of Neurosurgery: Spine, and Neurosurgical Focus) or Neurosurgery. Accepted checklists for reporting (PRISMA) and methodology (AMSTAR) were applied to each meta-analysis, and the number of items within each checklist that were satisfactorily fulfilled was recorded. The authors sought to answer 4 specific questions: Are meta-analyses improving 1) with time; 2) when the study met their definition of a meta-analysis; 3) when clinicians collaborated with a potential expert in meta-analysis; and 4) when the meta-analysis was the only focus of the paper?

RESULTS

Seventy-two meta-analyses were published in the JNS Publishing Group journals and Neurosurgery between 1990 and 2012. The number of published meta-analyses has increased dramatically in the last several years. The most common topics were vascular, and most were based on observational studies. Only 11 papers were prepared using an established checklist. The average AMSTAR and PRISMA scores (proportion of items satisfactorily fulfilled divided by the total number of eligible items in the respective instrument) were 31% and 55%, respectively. Major deficiencies were identified, including the lack of a comprehensive search strategy, study selection and data extraction, assessment of heterogeneity, publication bias, and study quality. Almost one-third of the papers did not meet our basic definition of a meta-analysis. The quality of reporting and methodology was better 1) when the study met our definition of a meta-analysis; 2) when one or more of the authors had experience or expertise in conducting a meta-analysis; 3) when the meta-analysis was not conducted alongside an evaluation of the authors' own data; and 4) in more recent studies.

CONCLUSIONS

Reporting and methodology of meta-analyses in the neurosurgery literature is excessively variable and overall poor. As these papers are being published with increasing frequency, neurosurgical journals need to adopt a clear definition of a meta-analysis and insist that they be created using checklists for both reporting and methodology. Standardization will ensure high-quality publications.

Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality

BACKGROUND

Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases have increased over the past decade and become increasingly important to a wide range of clinicians, policy makers, and other health care stakeholders. While a few criticisms about their methodological rigor and synthesis approaches have recently appeared, no formal appraisal of their quality has been conducted yet.

OBJECTIVE

The primary aim of this critical review was to evaluate the methodology, quality, and reporting characteristics of prior reviews that have investigated the effects of home telemonitoring interventions in the context of chronic diseases.

METHODS

Ovid MEDLINE, the Database of Abstract of Reviews of Effects (DARE), and Health Technology Assessment Database (HTA) of the Cochrane Library were electronically searched to find relevant systematic reviews, published between January 1966 and December 2012. Potential reviews were screened and assessed for inclusion independently by three reviewers. Data pertaining to the methods used were extracted from each included review and examined for accuracy by two reviewers. A validated quality assessment instrument, R-AMSTAR, was used as a framework to guide the assessment process.

RESULTS

Twenty-four reviews, nine of which were meta-analyses, were identified from more than 200 citations. The bibliographic search revealed that the number of published reviews has increased substantially over the years in this area and although most reviews focus on studying the effects of home telemonitoring on patients with congestive heart failure, researcher interest has extended to other chronic diseases as well, such as diabetes, hypertension, chronic obstructive pulmonary disease, and asthma. Nevertheless, an important number of these reviews appear to lack optimal scientific rigor due to intrinsic methodological issues. Also, the overall quality of reviews does not appear to have improved over time. While several criteria were met satisfactorily by either all or nearly all reviews, such as the establishment of an a priori design with inclusion and exclusion criteria, use of electronic searches on multiple databases, and reporting of studies characteristics, there were other important areas that needed improvement. Duplicate data extraction, manual searches of highly relevant journals, inclusion of gray and non-English literature, assessment of the methodological quality of included studies and quality of evidence were key methodological procedures that were performed infrequently. Furthermore, certain methodological limitations identified in the synthesis of study results have affected the results and conclusions of some reviews.

CONCLUSIONS

Despite the availability of methodological guidelines that can be utilized to guide the proper conduct of systematic reviews and meta-analyses and eliminate potential risks of bias, this knowledge has not yet been fully integrated in the area of home telemonitoring. Further efforts should be made to improve the design, conduct, reporting, and publication of systematic reviews and meta-analyses in this area.

Some improvements are apparent in identifying adverse effects in systematic reviews from 1994 to 2011

OBJECTIVE

An increasing amount of research and guidelines has been published on search methodology and the reporting of search strategies in systematic reviews. This research assessed whether this has lead to any improvements in the reporting and quality of searching in systematic reviews of adverse effects.

STUDY DESIGN AND SETTING

All records within Cochrane Database of Systematic Reviews and Database of Abstracts of Reviews of Effects were scanned for systematic reviews of adverse effects. Data were then extracted on the methods used for information retrieval in these reviews and a descriptive analysis conducted by publication year.

RESULTS

A total of 849 reviews published from 1994 to 2011 met the inclusion criteria. There has been a significant increase (P<0.001) in the number of adverse effects reviews per year from 1994 (n=5) to 2010 (n=104). Some improvements were apparent, such as an increase in the number of databases searched and fewer date and language restrictions applied. However, there has been an increase in reviews limited to data from randomized controlled trials, whereas the reporting of search strategies could still be improved further, with only 9% (74/849) of the reviews reporting reproducible searches.

CONCLUSION

Some improvements in searching systematic reviews of adverse effects are apparent; however, poor reporting of search strategies remains a great obstacle to readers.

Systematic analysis of hydroxyethyl starch (HES) reviews: proliferation of low-quality reviews overwhelms the results of well-performed meta-analyses

PURPOSE

Hydroxyethyl starch (HES) is a synthetic colloid used widely for resuscitation despite the availability of safer, less costly fluids. Numerous HES reviews have been published that may have influenced clinicians' practice. We have therefore examined the relationship between the methodological quality of published HES reviews, authors' potential conflicts of interest (pCOI) and the recommendations made.

METHODS

Systematic analysis of reviews on HES use.

RESULTS

Between 1975 and 2010, 165 reviews were published containing recommendations for or against HES use. From the 1990s onwards, favorable reviews increased from two to eight per year and HES's share of the artificial colloid market tripled from 20 to 60 %. Only 7 % (12/165) of these reviews of HES use contained meta-analyses; these 7 % had higher Overview Quality Assessment Questionnaire (OQAQ) scores [median (range) 6.5 (3-7)] than reviews without meta-analysis [2 (1-4); p < 0.001]. The rates of recommending against HES use are 83 % (10/12) in meta-analyses and 20 % (31/153) in reviews without meta-analysis (p < 0.0001). Fourteen authors published the majority (70/124) of positive reviews, and ten of these 14 had or have since developed a pCOI with various manufacturers of HES.

CONCLUSIONS

Low-quality HES reviews reached different conclusions than high-quality meta-analyses from independent entities, such as Cochrane Reviews. The majority of these low-quality positive HES reviews were written by a small group of authors, most of whom had or have since established ties to industry. The proliferation of positive HES reviews has been associated with increased utilization of an expensive therapy despite the lack of evidence for meaningful clinical benefit and increased risks. Clinicians need to be more informed that marketing efforts are potentially influencing scientific literature.

A literature review of randomized clinical trials of intravenous acetaminophen (paracetamol) for acute postoperative pain

INTRODUCTION

This study's objective was to systematically review the literature to assess analgesic outcomes of intravenous (IV) acetaminophen for acute postoperative pain in adults.

METHODS

We searched Medline and the Cochrane library (January 1, 2000 to January 17, 2010, date of last search) for prospective, randomized, controlled trials (RCTs) of IV acetaminophen vs. either an active comparator or placebo.

RESULTS

Sixteen articles from 9 countries published between 2005 and 2010 met inclusion criteria and had a total of 1,464 patients. Median sample size=54 patients (range 25 to 165) and median follow-up=1 day (range 1 hour to 7 days). Four of the 16 articles had 3 arms in the study. One article had 4 arms. As a result, 22 study comparisons were analyzed: IV acetaminophen to an active comparator (n=8 studies) and IV acetaminophen to placebo (n=14 studies). The RCTs were of high methodological quality with Jadad median score=5. In 7 of 8 active comparator studies (IV parecoxib [n=3 studies], IV metamizol [n=4], oral ibuprofen [n=1]), IV acetaminophen had similar analgesic outcomes as the active comparator. Twelve of the 14 placebo studies found that IV acetaminophen patients had improved analgesia. Ten of those 14 studies reported less opioid consumption, a lower percentage of patients rescuing, or a longer time to first rescue with IV acetaminophen. Formal meta-analysis pooling was not performed because the studies had different primary end points, and the IV acetaminophen dosing regimens varied in dose, and duration and timing.

CONCLUSION

In aggregate, these data indicate that IV acetaminophen is an effective analgesic across a variety of surgical procedures.

The quality of meta-analyses of genetic association studies: a review with recommendations

Although there has been a rapid rise in the publication of meta-analyses of genetic association studies, little is known about their methodological quality. The authors reviewed the quality of 120 randomly selected genetic meta-analyses published between 2005 and 2007. Data extracted included issues of general relevance and other issues specific to genetic epidemiology. Quality was markedly poorer in the 26% of the meta-analyses that accompanied a report on a primary study. Such meta-analyses were predominantly published in specialist journals, and their quality was positively associated with the impact factor of the journal. Among the meta-analyses that did not accompany a primary study, Human Genome Epidemiology reviews tended to score better than the others, although the comparison was limited by relatively small numbers. Comparison of the overall quality with that of genetic meta-analyses published before 2000 showed improvement in both conduct and reporting. However, the quality of the handling of specific genetic issues remains disappointingly low. For a few key general quality issues, the authors compared their findings with findings in other fields of medicine and found that general quality was similar. On the basis of this review, the authors provide practical recommendations for the conduct and reporting of genetic meta-analyses.

Reprint--preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement

Editor's Note: PTJ's Editorial Board has adopted PRISMA to help PTJ better communicate research to physical therapists. For more, read Chris Maher's editorial starting on page 870.

Membership of the PRISMA Group is provided in the Acknowledgments.

This article has been reprinted with permission from the Annals of Internal Medicine from Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. Ann Intern Med. Available at: http://www.annals.org/cgi/content/full/151/4/264. The authors jointly hold copyright of this article. This article has also been published in PLoS Medicine, BMJ, Journal of Clinical Epidemiology, and Open Medicine.

Copyright © 2009 Moher et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

A systematic review of the quality and impact of anxiety disorder meta-analyses

Meta-analyses are seen as representing the pinnacle of a hierarchy of evidence used to inform clinical practice. Therefore, the potential importance of differences in the rigor with which they are conducted and reported warrants consideration. In this review, we use standardized instruments to describe the scientific and reporting quality of meta-analyses of randomized controlled trials of the treatment of anxiety disorders. We also use traditional and novel metrics of article impact to assess the influence of meta-analyses across a range of research fields in the anxiety disorders. Overall, although the meta-analyses that we examined had some flaws, their quality of reporting was generally acceptable. Neither the scientific nor reporting quality of the meta-analyses was predicted by any of the impact metrics. The finding that treatment meta-analyses were cited less frequently than quantitative reviews of studies in current "hot spots" of research (ie, genetics, imaging) points to the multifactorial nature of citation patterns. A list of the meta-analyses included in this review is available on an evidence-based website of anxiety and trauma-related disorders.

Reporting and methodologic quality of Cochrane Neonatal review group systematic reviews

Background

The Cochrane Neonatal Review Group (CNRG) has achieved a lot with limited resources in producing high quality systematic reviews to assist clinicians in evidence-based decision-making. A formal assessment of published CNRG systematic reviews has not been undertaken; we sought to provide a comprehensive assessment of the quality of systematic reviews (both methodologic and reporting quality) published in CNRG.

Methods

We selected a random sample of published CNRG systematic reviews. Items of the QUOROM statement were utilized to assess quality of reporting, while items and total scores of the Oxman-Guyatt Overview Quality Assessment Questionnaire (OQAQ) were used to assess methodologic quality. Two reviewers independently extracted data and assessed quality. A Student t-test was used to compare quality scores pre- and post-publication of the QUOROM statement.

Results

Sixty-one systematic reviews were assessed. Overall, the included reviews had good quality with minor flaws based on OQAQ total scores (mean, 4.5 [0.9]; 95% CI, 4.27–4.77). However, room for improvement was noted in some areas, such as the title, abstract reporting, a priori plan for heterogeneity assessment and how to handle heterogeneity in case it exists, and assessment of publication bias. In addition, reporting of agreement among reviewers, documentation of trials flow, and discussion of possible biases were addressed in the review process. Reviews published post the QUOROM statement had a significantly higher quality scores.

Conclusion

The systematic reviews published in the CNRG are generally of good quality with minor flaws. However, efforts should be made to improve the quality of reports. Readers must continue to assess the quality of published reports on an individual basis prior to implementing the recommendations.

Operative Time and Other Outcomes of the Electrothermal Bipolar Vessel Sealing System (LigaSure™) Versus Other Methods for Surgical Hemostasis: A Meta-Analysis

A meta-analysis was performed of 29 prospective, randomized trials (published January 1, 2000, to August 14, 2007) comparing an electrothermal bipolar vessel sealing system (EBVS-LigaSure™, Covidien) (total n = 1107 patients) with either clamping with suture ligation/ electrocauterization (n = 1079 patients) or ultrasonic energy (eg, Harmonic Scalpel®, Johnson & Johnson). Hemorrhoidectomy (12 articles), hysterectomy (4 articles), and thyroidectomy (3 articles) were the most common procedures. For 15 of 26 studies reporting standard deviations, the normalized mean operative time reduction for EBVS equaled 28% (95% confidence interval [CI] 18%-39%, P < .0001) compared with conventional surgical hemostasis. Operative time was reduced with EBVS in 24 of 26 studies ( P < .0001). EBVS was associated with 43 mL (95% CI 14-73 mL, P = .0021) less blood loss, fewer complications (odds ratio 0.66, 95% CI 0.47-0.92, P = .02), and mean reduction in postoperative pain of 2.8 units (95% CI 1.5-4.1, P < .0001). Five studies used ultrasonic energy as the comparator, but none reported standard deviation so data could not be pooled.

Primer on systematic review and meta-analysis

A systematic review uses an explicitly defined process to comprehensively identify all studies pertaining to a specific focused question, appraise the methods of the studies, summarize the results, identify reasons for different findings across studies, and cite limitations of current knowledge. Meta-analyses usually combine aggregate-level data reported in each primary study, which may provide a more precise estimate of the "true effect" than any individual study. However, the conclusions may be limited by between-trial heterogeneity, publication bias, or deficits in the conduct or reporting of individual primary studies.

State of the evidence on acute asthma management in children: a critical appraisal of systematic reviews

OBJECTIVE

Our goal was to evaluate clinical, methodologic, and reporting aspects of systematic reviews on the management of acute asthma in children.

METHODS

We undertook a systematic review of systematic reviews on acute asthma management in children. We identified eligible reviews by searching the Cochrane Database of Systematic Reviews, the Database of Abstracts of Reviews of Effects, Medline, and Embase 1990 to March 2006. Data were extracted on clinical issues, methodologic characteristics, and results of the reviews. Methodologic quality was assessed with the Overview Quality Assessment Questionnaire and with additional questions on heterogeneity. Separate reporting on children in mixed adult-pediatric population reviews was assessed. Methodologic quality of systematic reviews published in peer-reviewed journals was compared with Cochrane reviews.

RESULTS

A total of 23 systematic reviews were included: 14 were published in the Cochrane Library, and 9 were published in peer-reviewed journals. Eight reviews included children only, and 15 were mixed-population reviews. The majority of reviews defined the study population as having "acute asthma" without a more precise definition, and 16 different health outcomes were reported. The overall quality according to the Overview Quality Assessment Questionnaire was good, with Cochrane reviews showing minimal flaws and journal reviews showing minor flaws (median scores: 7 vs 5). Results on children were reported separately in 8 of 15 mixed-population reviews. Clinical heterogeneity was explored in only 2 of 23 reviews, and the methods used to identify and address heterogeneity were diverse.

CONCLUSIONS

The methodologic quality of both the Cochrane and journal reviews on the management of acute asthma in children seems good, with Cochrane reviews being more rigorous. However, their usefulness for clinical practice is hampered by a lack of clear definitions of included populations, clinically important health outcomes, and separate reporting on children in mixed reviews. A major threat to these reviews' validity is the insufficient identification and handling of heterogeneity.

Bibliographic study showed improving methodology of meta-analyses published in leading journals 1993–2002

OBJECTIVE

To assess the methodology of meta-analyses published in leading general and specialist medical journals over a 10-year period.

STUDY DESIGN AND SETTING

Volumes 1993-2002 of four general medicine journals and four specialist journals were searched by hand for meta-analyses including at least five controlled trials. Characteristics were assessed using a standardized questionnaire.

RESULTS

A total of 272 meta-analyses, which included a median of 11 trials (range 5-195), were assessed. Most (81%) were published in general medicine journals. The median (range) number of databases searched increased from 1 (1-9) in 1993/1994 to 3.5 (1-21) in 2001/2002, P<0.0001. The proportion of meta-analyses including searches by hand (10% in 1993/1994, 25% in 2001/2002, P=0.005), searches of the grey literature (29%, 51%, P=0.010 by chi-square test), and of trial registers (10%, 32%, P=0.025) also increased. Assessments of the quality of trials also became more common (45%, 70%, P=0.008), including whether allocation of patients to treatment groups had been concealed (24%, 60%, P=0.001). The methodological and reporting quality was consistently higher in general medicine compared to specialist journals.

CONCLUSION

Many meta-analyses published in leading journals have important methodological limitations. The situation has improved in recent years but considerable room for further improvements remains.

The physical environment and physical activity: a critical appraisal of review articles

BACKGROUND

Over the last few years an increasing number of studies investigating the association between the physical environment and physical activity have been published. Many reviews have also summarized this emerging body of research, and such review papers are frequently used by public health policymakers and researchers themselves to inform decision making.

METHODS

This paper systematically appraises methodologic aspects of literature reviews examining the relationship between physical activity and the physical environment published in peer-reviewed journals between 2000 and 2005. Eleven reviews and their antecedent source papers were examined.

RESULTS

The majority of these reviews omitted between one third and two thirds of the studies that could have been eligible for inclusion at the time they conducted the review. Methodologic information on how the review was conducted was not always provided. Furthermore, in some cases results of a study were reported incorrectly, or physical environmental aspects were conflated with social environmental or cognitive factors. Moreover, when results were reported incorrectly, physical environmental variables were almost always reported as significantly associated with physical activity, when these associations were nonsignificant, or were not assessed as part of the primary study.

CONCLUSIONS

Users of reviews in this field should be aware that there are significant methodologic variations among them, and that some reviews may include only a sample of the relevant primary studies. However, this is difficult to determine given the frequent incompleteness of review method reporting. Greater standardization in the reporting of review methods may assist with future efforts to summarize studies of the relationship between physical environments and physical activity.

The quality of reports of critical care meta-analyses in the Cochrane Database of Systematic Reviews: an independent appraisal

OBJECTIVE

To independently appraise the methodological quality of a sample of reports of meta-analyses that address critical care topics in the Cochrane Database of Systematic Reviews compared with the quality of reports published in regular journals, using a validated assessment instrument, the Overview Quality Assessment Questionnaire (OQAQ).

DATA SOURCE

Studies were selected from a search of MEDLINE, EMBASE, and the Cochrane Database of Systematic Reviews from 1994 to 2003, using multiple search terms for critical care and sensitive filters to identify meta-analyses.

STUDY SELECTION

Two authors independently selected meta-analyses that addressed topics pertinent to critical care medicine.

DATA EXTRACTION

Two authors independently extracted the data. The proportion of reports that met each component of the OQAQ was determined, as was the overall quality score. Meta-analyses published in the Cochrane Database of Systematic Reviews were compared with those published in regular journals.

DATA SYNTHESIS

There were 36 reports of meta-analyses in the Cochrane Database of Systematic Reviews and 103 reports of meta-analyses published in regular journals; 11 of these were reports of Cochrane reviews. The meta-analyses published in the Cochrane Database of Systematic Reviews were more likely to fulfill most components of the OQAQ. The median overall OQAQ scores indicated significant methodological problems in the reports regardless of the source of publication, although the reports in the Cochrane database scored higher than those in regular journals (five compared with two, p<.001). Major methodological flaws, notably failure to appropriately refer to the validity of included studies, were found in meta-analyses in both the Cochrane Database of Systematic Reviews and regular journals (44.4% and 79.3%, respectively).

CONCLUSIONS

Although the quality of reports of meta-analyses published in the Cochrane Database of Systematic Reviews is superior to the quality of reports of meta-analyses published in regular journals, there is significant room for improvement. Clinicians should critically appraise all reports of meta-analyses before considering the results, regardless of the source of publication.

Limited evidence for effects of diet for type 2 diabetes from systematic reviews

OBJECTIVE

Systematic reviews are an appraised method to summarize research in a concise and transparent way, and may enable to draw conclusions beyond the sum of results of individual studies. We assessed the results, quality and external validity of systematic reviews on diet in patients with type 2 diabetes.

DESIGN, SETTING, SUBJECTS

We systematically searched for systematic reviews on nutritional interventions in patients with type 2 diabetes that used a reproducible search strategy in at least one major database that applied some form of quality assessment. We assessed quality and the external validity of the retrieved systematic reviews. Outcomes were defined as statistical meta-analyses or narrative results using a predefined and reproducible method.

RESULTS

Six systematic reviews met the inclusion criteria, investigating dietary interventions in general (n=3), chromium supplementation (n=1), fish-oil (n=1) or herbs and nutrition supplements (n=1). Quality assessment showed minimal/minor flaws in four cases and major/extensive flaws in two cases. All reviews had insufficient data needed to judge external validity. In reviews with minimal/minor flaws, we found beneficial effects of very-low-calorie diets and fish-oil supplements. However, the external validity of these results could not be assessed sufficiently.

CONCLUSIONS

Systematic reviews largely failed to produce knowledge beyond the sum of the original studies. Furthermore, judgment of external validity was hampered in most cases owing to missing data. To improve the quality and usefulness of systematic reviews of dietary interventions, we recommend the application of more focused research questions, but with broader inclusion criteria, for example, the use of observational studies.

SPONSORSHIP

Internal funding Radboud University Nijmegen Medical Centre.

Clinical relevance in anesthesia journals

PURPOSE OF REVIEW

The purpose of this review is to present the latest knowledge and research on the definition and distribution of clinically relevant articles in anesthesia journals. It will also discuss the importance of the chosen methodology and outcome of articles.

RECENT FINDINGS

In the last few years, more attention has been paid to evidence-based medicine in anesthesia. Several articles on the subject have focused on the need to base clinical decisions on sound research employing both methodological rigor and clinically relevant outcomes. The number of systematic reviews in anesthesia literature is increasing as well as the focus on diminishing the number of surrogate outcomes. It has been shown that the impact factor is not a valid measure of establishing the level of clinical relevance to a journal.

SUMMARY

This review presents definitions of clinically relevant anesthesia articles. A clinically relevant article employs both methodological rigor and a clinically relevant outcome. The terms methodological rigor and clinical outcomes are fully discussed in the review as well as problems with journal impact factors.

Systematic reviews involving complementary and alternative medicine interventions had higher quality of reporting than conventional medicine reviews

OBJECTIVE

To compare the quality of systematic reviews reported in English and in languages other than English, and to determine whether there are differences between conventional medicine (CM) and complementary and alternative medicine (CAM) reports.

STUDY DESIGN AND SETTING

We used the Oxman and Guyatt (OG) scale to assess the quality of reporting in 130 systematic reviews: 50 were language-restricted, 32 were language-inclusive but only English-language (EL) trials contained (inclusive-EL), and 48 were language-inclusive and included trials published in languages other than English (inclusive-LOE). Of the 130 reviews, 105 addressed CM interventions and 25 addressed CAM interventions.

RESULTS

Comparison of the systematic reviews showed that the quality of reporting and reporting characteristics are not affected by inclusion or exclusion of LOE; however, the quality of reporting of systematic reviews involving CAM interventions is higher than that of reviews focusing on CM interventions.

CONCLUSION

Informal comparison of the OG scale with the data collected on quality assessments showed that the OG scale performs well overall but may not identify important differences in comprehensiveness of the search strategy and avoidance of bias in study selection. Further research is required to determine the best methods for assessing quality of systematic reviews and whether the effect of language restrictions is dependent on the type of intervention (CM or CAM).

Evaluating meta-analyses in the general surgical literature: a critical appraisal

OBJECTIVE

To assess the methodologic quality of meta-analyses of general surgery topics published in peer-reviewed journals.

SUMMARY BACKGROUND DATA

Systematic reviews and meta-analysis are used to seek, summarize, and interpret primary studies on a given topic. Accordingly, systematic reviews and meta-analyses of high-quality primary studies may be the highest level of evidence for issues of prevention and treatment in evidence-based medicine. However, not all published meta-analyses are rigorously performed.

METHODS

We searched MEDLINE (from January 1, 1997, to September 1, 2002) and reference lists and solicited general surgery specialists to identify relevant meta-analyses. Inclusion criteria were use of meta-analytic methods to pool the results of primary studies in general surgery on issues of diagnosis, causation, prognosis, or treatment. Our search strategies identified 487 potentially relevant articles. After excluding articles based on a priori criteria, 51 meta-analyses fulfilled eligibility criteria. In duplicate and independently, 2 reviewers assessed the quality of these meta-analyses using a 10-item index called the Overview Quality Assessment Questionnaire.

RESULTS

Overall concordance between 2 independent reviewers was good (interobserver agreement 81%, and a kappa of 0.62 (95% CI 0.55-0.69). Of 51 relevant articles, 38 were published in surgical journals. Most studies had major methodologic flaws (median score of 3.3, scale of 1-7). Factors associated with low overall scientific quality included the absence of any prior meta-analyses publications by authors and meta-analyses produced by surgical department members without external collaboration.

CONCLUSIONS

This critical appraisal of meta-analyses published in the general surgery literature demonstrates frequent methodologic flaws. The quality of these reports limits the validity of the findings and the inferences that can be made about the primary studies reviewed. To improve the quality of future meta-analyses, we recommend following guidelines for the optimal conduct and reporting of meta-analyses in general surgery.

Publications in Anesthesia Journals: Quality and Clinical Relevance

Clinicians performing evidence-based anesthesia rely on anesthesia journals for clinically relevant information. The objective of this study was to analyze the proportion of clinically relevant articles in five high impact anesthesia journals. We evaluated all articles published in Anesthesiology, Anesthesia & Analgesia, British Journal of Anesthesia, Anesthesia, and Acta Anaesthesiologica Scandinavica from January to June, 2000. Articles were assessed and classified according to type, outcome, and design; 1379 articles consisting of 5468 pages were evaluated and categorized. The most common types of article were animal and laboratory research (31.2%) and randomized clinical trial (20.4%). A clinically relevant article was defined as an article that used a statistically valid method and had a clinically relevant end-point. Altogether 18.6% of the pages had as their subject matter clinically relevant trials. We compared the Journal Impact Factor (a measure of the number of citations per article in a journal) and the proportion of clinically relevant pages and found that they were inversely proportional to each other.

The mission of the cochrane anesthesia review group: preparing and disseminating systematic reviews of the effect of health care in anesthesiology

IMPLICATIONS

This article illustrates the basic principles of evidence-based medicine and the work within the Cochrane Collaboration and the Cochrane Anesthesia Review Group. It describes how important randomized controlled trials and systematic reviews are in providing the best evidence to answer clinically relevant questions.

Examining the evidence in anesthesia literature: a survey and evaluation of obstetrical postdural puncture headache reports

PURPOSE

To describe a bibliographic database on the literature of postdural puncture headache (PDPH) in the obstetrical population, to describe the research architecture in this field, and to evaluate the quality of case-control studies, cohort studies, and controlled clinical trials on PDPH.

METHODS

Computerized bibliographic searches, citation review, and hand searches were conducted to find all relevant citations on incidence, clinical course, prevention, or treatment of PDPH in parturients. The study design and topic(s) covered by each study were evaluated. Case-control studies and cohort studies were evaluated using the Quality Index; clinical trials were evaluated using the Jadad scale.

RESULTS

One hundred ninety-six relevant citations were published between 1949 and 1999. Research on PDPH has been increasing rapidly with the majority of studies published in the 1990's. Incidence and prevention were the focus of over half of all citations. Optimal study designs were infrequently utilized. The methodological quality was poor for observational studies (Quality Index 10/29) and clinical trials (Jadad scale 2/5).

CONCLUSION

Although the amount of research on PDPH in parturients is increasing, use of optimal study designs and improvement in methodology is needed.