Examining the evidence in anesthesia literature: a critical appraisal of systematic reviews

Systematic review adherence to methodological or reporting quality

BACKGROUND

Guidelines for assessing methodological and reporting quality of systematic reviews (SRs) were developed to contribute to implementing evidence-based health care and the reduction of research waste. As SRs assessing a cohort of SRs is becoming more prevalent in the literature and with the increased uptake of SR evidence for decision-making, methodological quality and standard of reporting of SRs is of interest. The objective of this study is to evaluate SR adherence to the Quality of Reporting of Meta-analyses (QUOROM) and PRISMA reporting guidelines and the A Measurement Tool to Assess Systematic Reviews (AMSTAR) and Overview Quality Assessment Questionnaire (OQAQ) quality assessment tools as evaluated in methodological overviews.

METHODS

The Cochrane Library, MEDLINE®, and EMBASE® databases were searched from January 1990 to October 2014. Title and abstract screening and full-text screening were conducted independently by two reviewers. Reports assessing the quality or reporting of a cohort of SRs of interventions using PRISMA, QUOROM, OQAQ, or AMSTAR were included. All results are reported as frequencies and percentages of reports and SRs respectively.

RESULTS

Of the 20,765 independent records retrieved from electronic searching, 1189 reports were reviewed for eligibility at full text, of which 56 reports (5371 SRs in total) evaluating the PRISMA, QUOROM, AMSTAR, and/or OQAQ tools were included. Notable items include the following: of the SRs using PRISMA, over 85% (1532/1741) provided a rationale for the review and less than 6% (102/1741) provided protocol information. For reports using QUOROM, only 9% (40/449) of SRs provided a trial flow diagram. However, 90% (402/449) described the explicit clinical problem and review rationale in the introduction section. Of reports using AMSTAR, 30% (534/1794) used duplicate study selection and data extraction. Conversely, 80% (1439/1794) of SRs provided study characteristics of included studies. In terms of OQAQ, 37% (499/1367) of the SRs assessed risk of bias (validity) in the included studies, while 80% (1112/1387) reported the criteria for study selection.

CONCLUSIONS

Although reporting guidelines and quality assessment tools exist, reporting and methodological quality of SRs are inconsistent. Mechanisms to improve adherence to established reporting guidelines and methodological assessment tools are needed to improve the quality of SRs.

Identifying approaches for assessing methodological and reporting quality of systematic reviews: a descriptive study

BACKGROUND

The methodological quality and completeness of reporting of the systematic reviews (SRs) is fundamental to optimal implementation of evidence-based health care and the reduction of research waste. Methods exist to appraise SRs yet little is known about how they are used in SRs or where there are potential gaps in research best-practice guidance materials. The aims of this study are to identify reports assessing the methodological quality (MQ) and/or reporting quality (RQ) of a cohort of SRs and to assess their number, general characteristics, and approaches to 'quality' assessment over time.

METHODS

The Cochrane Library, MEDLINE®, and EMBASE® were searched from January 1990 to October 16, 2014, for reports assessing MQ and/or RQ of SRs. Title, abstract, and full-text screening of all reports were conducted independently by two reviewers. Reports assessing the MQ and/or RQ of a cohort of ten or more SRs of interventions were included. All results are reported as frequencies and percentages of reports.

RESULTS

Of 20,765 unique records retrieved, 1189 of them were reviewed for full-text review, of which 76 reports were included. Eight previously published approaches to assessing MQ or reporting guidelines used as proxy to assess RQ were used in 80% (61/76) of identified reports. These included two reporting guidelines (PRISMA and QUOROM) and five quality assessment tools (AMSTAR, R-AMSTAR, OQAQ, Mulrow, Sacks) and GRADE criteria. The remaining 24% (18/76) of reports developed their own criteria. PRISMA, OQAQ, and AMSTAR were the most commonly used published tools to assess MQ or RQ. In conjunction with other approaches, published tools were used in 29% (22/76) of reports, with 36% (8/22) assessing adherence to both PRISMA and AMSTAR criteria and 26% (6/22) using QUOROM and OQAQ.

CONCLUSIONS

The methods used to assess quality of SRs are diverse, and none has become universally accepted. The most commonly used quality assessment tools are AMSTAR, OQAQ, and PRISMA. As new tools and guidelines are developed to improve both the MQ and RQ of SRs, authors of methodological studies are encouraged to put thoughtful consideration into the use of appropriate tools to assess quality and reporting.

Reporting and methodologic evaluation of meta-analyses published in the anesthesia literature according to AMSTAR and PRISMA checklists: a preliminary study

BACKGROUND

There have been few recent reports on the methodological quality of meta-analysis, despite the enormous number of studies using meta-analytic techniques in the field of anesthesia. The purpose of this study was to evaluate the quality of meta-analyses and systematic reviews according to the Assessment of Multiple Systematic Reviews (AMSTAR) and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines in the anesthesia literature.

METHODS

A search was conducted to identify all meta-analyses ever been published in the British Journal of Anaesthesia (BJA), Anaesthesia, and Korean Journal of Anesthesiology (KJA) between Jan. 01, 2004 and Nov. 31, 2016. We aimed to apply the AMSTAR and PRISMA checklists to all published meta-analyses.

RESULTS

We identified 121 meta-analyses in the anesthesia literature from January 2004 through the end of November 2016 (BJA; 75, Anaesthesia; 43, KJA; 3). The number of studies published and percentage of 'Yes' responses for meta-analysis articles published after the year 2010 was significantly increased compared to that of studies published before the year 2009 (P = 0.014 for Anaesthesia). In the anesthesia literature as a whole, participation of statisticians as authors statistically improved average scores of PRISMA items (P = 0.004) especially in the BJA (P = 0.003).

CONCLUSIONS

Even though there is little variability in the reporting and methodology of meta-analysis in the anesthesia literature, significant quality improvement in the reporting was observed in the Anaesthesia by applying the PRISMA checklist. Participation of a statistician as an author improved the reporting quality of the meta-analysis.

Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions

BACKGROUND

Systematic reviews may be compromised by selective inclusion and reporting of outcomes and analyses. Selective inclusion occurs when there are multiple effect estimates in a trial report that could be included in a particular meta-analysis (e.g. from multiple measurement scales and time points) and the choice of effect estimate to include in the meta-analysis is based on the results (e.g. statistical significance, magnitude or direction of effect). Selective reporting occurs when the reporting of a subset of outcomes and analyses in the systematic review is based on the results (e.g. a protocol-defined outcome is omitted from the published systematic review).

OBJECTIVES

To summarise the characteristics and synthesise the results of empirical studies that have investigated the prevalence of selective inclusion or reporting in systematic reviews of randomised controlled trials (RCTs), investigated the factors (e.g. statistical significance or direction of effect) associated with the prevalence and quantified the bias.

SEARCH METHODS

We searched the Cochrane Methodology Register (to July 2012), Ovid MEDLINE, Ovid EMBASE, Ovid PsycINFO and ISI Web of Science (each up to May 2013), and the US Agency for Healthcare Research and Quality (AHRQ) Effective Healthcare Program's Scientific Resource Center (SRC) Methods Library (to June 2013). We also searched the abstract books of the 2011 and 2012 Cochrane Colloquia and the article alerts for methodological work in research synthesis published from 2009 to 2011 and compiled in Research Synthesis Methods.

SELECTION CRITERIA

We included both published and unpublished empirical studies that investigated the prevalence and factors associated with selective inclusion or reporting, or both, in systematic reviews of RCTs of healthcare interventions. We included empirical studies assessing any type of selective inclusion or reporting, such as investigations of how frequently RCT outcome data is selectively included in systematic reviews based on the results, outcomes and analyses are discrepant between protocol and published review or non-significant outcomes are partially reported in the full text or summary within systematic reviews.

DATA COLLECTION AND ANALYSIS

Two review authors independently selected empirical studies for inclusion, extracted the data and performed a risk of bias assessment. A third review author resolved any disagreements about inclusion or exclusion of empirical studies, data extraction and risk of bias. We contacted authors of included studies for additional unpublished data. Primary outcomes included overall prevalence of selective inclusion or reporting, association between selective inclusion or reporting and the statistical significance of the effect estimate, and association between selective inclusion or reporting and the direction of the effect estimate. We combined prevalence estimates and risk ratios (RRs) using a random-effects meta-analysis model.

MAIN RESULTS

Seven studies met the inclusion criteria. No studies had investigated selective inclusion of results in systematic reviews, or discrepancies in outcomes and analyses between systematic review registry entries and published systematic reviews. Based on a meta-analysis of four studies (including 485 Cochrane Reviews), 38% (95% confidence interval (CI) 23% to 54%) of systematic reviews added, omitted, upgraded or downgraded at least one outcome between the protocol and published systematic review. The association between statistical significance and discrepant outcome reporting between protocol and published systematic review was uncertain. The meta-analytic estimate suggested an increased risk of adding or upgrading (i.e. changing a secondary outcome to primary) when the outcome was statistically significant, although the 95% CI included no association and a decreased risk as plausible estimates (RR 1.43, 95% CI 0.71 to 2.85; two studies, n = 552 meta-analyses). Also, the meta-analytic estimate suggested an increased risk of downgrading (i.e. changing a primary outcome to secondary) when the outcome was statistically significant, although the 95% CI included no association and a decreased risk as plausible estimates (RR 1.26, 95% CI 0.60 to 2.62; two studies, n = 484 meta-analyses). None of the included studies had investigated whether the association between statistical significance and adding, upgrading or downgrading of outcomes was modified by the type of comparison, direction of effect or type of outcome; or whether there is an association between direction of the effect estimate and discrepant outcome reporting.Several secondary outcomes were reported in the included studies. Two studies found that reasons for discrepant outcome reporting were infrequently reported in published systematic reviews (6% in one study and 22% in the other). One study (including 62 Cochrane Reviews) found that 32% (95% CI 21% to 45%) of systematic reviews did not report all primary outcomes in the abstract. Another study (including 64 Cochrane and 118 non-Cochrane reviews) found that statistically significant primary outcomes were more likely to be completely reported in the systematic review abstract than non-significant primary outcomes (RR 2.66, 95% CI 1.81 to 3.90). None of the studies included systematic reviews published after 2009 when reporting standards for systematic reviews (Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) Statement, and Methodological Expectations of Cochrane Intervention Reviews (MECIR)) were disseminated, so the results might not be generalisable to more recent systematic reviews.

AUTHORS' CONCLUSIONS

Discrepant outcome reporting between the protocol and published systematic review is fairly common, although the association between statistical significance and discrepant outcome reporting is uncertain. Complete reporting of outcomes in systematic review abstracts is associated with statistical significance of the results for those outcomes. Systematic review outcomes and analysis plans should be specified prior to seeing the results of included studies to minimise post-hoc decisions that may be based on the observed results. Modifications that occur once the review has commenced, along with their justification, should be clearly reported. Effect estimates and CIs should be reported for all systematic review outcomes regardless of the results. The lack of research on selective inclusion of results in systematic reviews needs to be addressed and studies that avoid the methodological weaknesses of existing research are also needed.

Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality

BACKGROUND

Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases have increased over the past decade and become increasingly important to a wide range of clinicians, policy makers, and other health care stakeholders. While a few criticisms about their methodological rigor and synthesis approaches have recently appeared, no formal appraisal of their quality has been conducted yet.

OBJECTIVE

The primary aim of this critical review was to evaluate the methodology, quality, and reporting characteristics of prior reviews that have investigated the effects of home telemonitoring interventions in the context of chronic diseases.

METHODS

Ovid MEDLINE, the Database of Abstract of Reviews of Effects (DARE), and Health Technology Assessment Database (HTA) of the Cochrane Library were electronically searched to find relevant systematic reviews, published between January 1966 and December 2012. Potential reviews were screened and assessed for inclusion independently by three reviewers. Data pertaining to the methods used were extracted from each included review and examined for accuracy by two reviewers. A validated quality assessment instrument, R-AMSTAR, was used as a framework to guide the assessment process.

RESULTS

Twenty-four reviews, nine of which were meta-analyses, were identified from more than 200 citations. The bibliographic search revealed that the number of published reviews has increased substantially over the years in this area and although most reviews focus on studying the effects of home telemonitoring on patients with congestive heart failure, researcher interest has extended to other chronic diseases as well, such as diabetes, hypertension, chronic obstructive pulmonary disease, and asthma. Nevertheless, an important number of these reviews appear to lack optimal scientific rigor due to intrinsic methodological issues. Also, the overall quality of reviews does not appear to have improved over time. While several criteria were met satisfactorily by either all or nearly all reviews, such as the establishment of an a priori design with inclusion and exclusion criteria, use of electronic searches on multiple databases, and reporting of studies characteristics, there were other important areas that needed improvement. Duplicate data extraction, manual searches of highly relevant journals, inclusion of gray and non-English literature, assessment of the methodological quality of included studies and quality of evidence were key methodological procedures that were performed infrequently. Furthermore, certain methodological limitations identified in the synthesis of study results have affected the results and conclusions of some reviews.

CONCLUSIONS

Despite the availability of methodological guidelines that can be utilized to guide the proper conduct of systematic reviews and meta-analyses and eliminate potential risks of bias, this knowledge has not yet been fully integrated in the area of home telemonitoring. Further efforts should be made to improve the design, conduct, reporting, and publication of systematic reviews and meta-analyses in this area.

Systematic analysis of hydroxyethyl starch (HES) reviews: proliferation of low-quality reviews overwhelms the results of well-performed meta-analyses

PURPOSE

Hydroxyethyl starch (HES) is a synthetic colloid used widely for resuscitation despite the availability of safer, less costly fluids. Numerous HES reviews have been published that may have influenced clinicians' practice. We have therefore examined the relationship between the methodological quality of published HES reviews, authors' potential conflicts of interest (pCOI) and the recommendations made.

METHODS

Systematic analysis of reviews on HES use.

RESULTS

Between 1975 and 2010, 165 reviews were published containing recommendations for or against HES use. From the 1990s onwards, favorable reviews increased from two to eight per year and HES's share of the artificial colloid market tripled from 20 to 60 %. Only 7 % (12/165) of these reviews of HES use contained meta-analyses; these 7 % had higher Overview Quality Assessment Questionnaire (OQAQ) scores [median (range) 6.5 (3-7)] than reviews without meta-analysis [2 (1-4); p < 0.001]. The rates of recommending against HES use are 83 % (10/12) in meta-analyses and 20 % (31/153) in reviews without meta-analysis (p < 0.0001). Fourteen authors published the majority (70/124) of positive reviews, and ten of these 14 had or have since developed a pCOI with various manufacturers of HES.

CONCLUSIONS

Low-quality HES reviews reached different conclusions than high-quality meta-analyses from independent entities, such as Cochrane Reviews. The majority of these low-quality positive HES reviews were written by a small group of authors, most of whom had or have since established ties to industry. The proliferation of positive HES reviews has been associated with increased utilization of an expensive therapy despite the lack of evidence for meaningful clinical benefit and increased risks. Clinicians need to be more informed that marketing efforts are potentially influencing scientific literature.

The quality of meta-analyses of genetic association studies: a review with recommendations

Although there has been a rapid rise in the publication of meta-analyses of genetic association studies, little is known about their methodological quality. The authors reviewed the quality of 120 randomly selected genetic meta-analyses published between 2005 and 2007. Data extracted included issues of general relevance and other issues specific to genetic epidemiology. Quality was markedly poorer in the 26% of the meta-analyses that accompanied a report on a primary study. Such meta-analyses were predominantly published in specialist journals, and their quality was positively associated with the impact factor of the journal. Among the meta-analyses that did not accompany a primary study, Human Genome Epidemiology reviews tended to score better than the others, although the comparison was limited by relatively small numbers. Comparison of the overall quality with that of genetic meta-analyses published before 2000 showed improvement in both conduct and reporting. However, the quality of the handling of specific genetic issues remains disappointingly low. For a few key general quality issues, the authors compared their findings with findings in other fields of medicine and found that general quality was similar. On the basis of this review, the authors provide practical recommendations for the conduct and reporting of genetic meta-analyses.

A systematic review of the quality and impact of anxiety disorder meta-analyses

Meta-analyses are seen as representing the pinnacle of a hierarchy of evidence used to inform clinical practice. Therefore, the potential importance of differences in the rigor with which they are conducted and reported warrants consideration. In this review, we use standardized instruments to describe the scientific and reporting quality of meta-analyses of randomized controlled trials of the treatment of anxiety disorders. We also use traditional and novel metrics of article impact to assess the influence of meta-analyses across a range of research fields in the anxiety disorders. Overall, although the meta-analyses that we examined had some flaws, their quality of reporting was generally acceptable. Neither the scientific nor reporting quality of the meta-analyses was predicted by any of the impact metrics. The finding that treatment meta-analyses were cited less frequently than quantitative reviews of studies in current "hot spots" of research (ie, genetics, imaging) points to the multifactorial nature of citation patterns. A list of the meta-analyses included in this review is available on an evidence-based website of anxiety and trauma-related disorders.

Reporting and methodologic quality of Cochrane Neonatal review group systematic reviews

Background

The Cochrane Neonatal Review Group (CNRG) has achieved a lot with limited resources in producing high quality systematic reviews to assist clinicians in evidence-based decision-making. A formal assessment of published CNRG systematic reviews has not been undertaken; we sought to provide a comprehensive assessment of the quality of systematic reviews (both methodologic and reporting quality) published in CNRG.

Methods

We selected a random sample of published CNRG systematic reviews. Items of the QUOROM statement were utilized to assess quality of reporting, while items and total scores of the Oxman-Guyatt Overview Quality Assessment Questionnaire (OQAQ) were used to assess methodologic quality. Two reviewers independently extracted data and assessed quality. A Student t-test was used to compare quality scores pre- and post-publication of the QUOROM statement.

Results

Sixty-one systematic reviews were assessed. Overall, the included reviews had good quality with minor flaws based on OQAQ total scores (mean, 4.5 [0.9]; 95% CI, 4.27–4.77). However, room for improvement was noted in some areas, such as the title, abstract reporting, a priori plan for heterogeneity assessment and how to handle heterogeneity in case it exists, and assessment of publication bias. In addition, reporting of agreement among reviewers, documentation of trials flow, and discussion of possible biases were addressed in the review process. Reviews published post the QUOROM statement had a significantly higher quality scores.

Conclusion

The systematic reviews published in the CNRG are generally of good quality with minor flaws. However, efforts should be made to improve the quality of reports. Readers must continue to assess the quality of published reports on an individual basis prior to implementing the recommendations.

Need for quality improvement in renal systematic reviews

BACKGROUND AND OBJECTIVES

Systematic reviews of clinical studies aim to compile best available evidence for various diagnosis and treatment options. This study assessed the methodologic quality of all systematic reviews relevant to the practice of nephrology published in 2005.

DESIGN, SETTING, PARTICIPANTS, & MEASUREMENTS

We searched electronic databases (Medline, Embase, American College of Physicians Journal Club, Cochrane) and hand searched Cochrane renal group records. Clinical practice guidelines, case reports, narrative reviews, and pooled individual patient data meta-analyses were excluded. Methodologic quality was measured using a validated questionnaire (Overview Quality Assessment Questionnaire). For reviews of randomized trials, we also evaluated adherence to recommended reporting guidelines (Quality of Reporting of Meta-Analyses).

RESULTS

Ninety renal systematic reviews were published in year 2005, 60 of which focused on therapy. Many systematic reviews (54%) had major methodologic flaws. The most common review flaws were failure to assess the methodologic quality of included primary studies and failure to minimize bias in study inclusion. Only 2% of reviews of randomized trials fully adhered to reporting guidelines. A minority of journals (four of 48) endorsed adherence to consensus guidelines for review reporting, and these journals published systematic reviews of higher methodologic quality (P < 0.001).

CONCLUSIONS

The majority of systematic reviews had major methodologic flaws. The majority of journals do not endorse consensus guidelines for review reporting in their instructions to authors; however, journals that recommended such adherence published systemic reviews of higher methodologic quality.

Evidence-based medicine in anesthesiology

PURPOSE OF REVIEW

The principles of evidence-based medicine are now being applied in anesthesiology and intensive care medicine. The purpose of this review is to acquaint practitioners with the fundamentals of evidence-based medicine and to provide examples of how these principles can be incorporated into clinicians' practice.

RECENT FINDINGS

Outcomes research utilizing evidence-based approaches are manifest in several areas of anesthesiology and intensive care medicine. Several examples are cited from various sub-specialty areas, including pediatric, obstetric, and general anesthesia, as well as intensive care medicine.

SUMMARY

Evidence-based medicine is the term used to describe a practice paradigm that emphasizes the use of the best evidence available in the medical literature in making treatment decisions about the care of patients. Evidence-based approaches to care integrate individual expertise with data from externally conducted systematic research. Although evidence-based medicine has its origins in the 'treating' and 'diagnosing' specialty of internal medicine, its tenets are applicable to 'non-therapeutic' specialties such as anesthesiology and intensive care medicine. This review illustrates how evidence-based principles can be incorporated into the practice of perioperative medicine.

Systematic Literature Review of Spinal Decompression Via Motorized Traction for Chronic Discogenic Low Back Pain

OBJECTIVE

The objective of this study was to systematically review the literature to assess the efficacy of nonsurgical spinal decompression achieved with motorized traction for chronic discogenic lumbosacral back pain.

DESIGN

Computer-aided systematic literature search of MEDLINE and the Cochrane collaboration for prospective clinical trials on adults with low back pain in the English literature from 1975 to October 2005. Methodologic quality for each study was assessed. Studies were included if the intervention group received motorized spinal decompression and the comparison group received sham or another type of nonsurgical treatment.

RESULTS

Data from 10 studies were fully analyzed. Seven studies were randomized controlled trials using various apparatus types. Because of this low number, we also analyzed three nonrandomized case series studies of spinal decompression systems. As the overall quality of studies was low and the patient groups heterogeneous, a meta-analysis was not appropriate and a qualitative review was undertaken. Sample sizes averaged 121 patients (range 27-292), with six of the seven randomized studies reporting no difference with motorized spinal decompression and one study reporting reduced pain but not disability. The three unrandomized studies (no control group) of motorized spinal decompression found a 77% to 86% reduction in pain.

CONCLUSIONS

These data suggest that the efficacy of spinal decompression achieved with motorized traction for chronic discogenic low back pain remains unproved. This may be, in part, due to heterogeneous patient groups and the difficulties involved in properly blinding patients to the mechanical pulling mechanism. Scientifically more rigorous studies with better randomization, control groups, and standardized outcome measures are needed to overcome the limitations of past studies.

A systematic evaluation of the quality of meta-analyses in the critical care literature

INTRODUCTION

Meta-analyses have been suggested to be the highest form of evidence available to clinicians to guide clinical practice in critical care. The purpose of this study was to systematically evaluate the quality of meta-analyses that address topics pertinent to critical care.

METHODS

To identify potentially eligible meta-analyses for inclusion, a systematic search of Medline, EMBASE and the Cochrane Database of Systematic Reviews was undertaken, using broad search terms relevant to intensive care, including: intensive care, critical care, shock, resuscitation, inotropes and mechanical ventilation. Predetermined inclusion criteria were applied to each identified meta-analysis independently by two authors. To assess report quality, the included meta-analyses were assessed using the component and overall scores from the Overview Quality Assessment Questionnaire (OQAQ). The quality of reports published before and after the publication of the QUOROM statement was compared.

RESULTS

A total of 139 reports of meta-analyses were included (kappa = 0.93). The overall quality of reports of meta-analyses was found to be poor, with an estimated mean overall OQAQ score of 3.3 (95% CI; 3.0-3.6). Only 43 (30.9%) were scored as having minimal or minor flaws (>5). We noted problems with the reporting of key characteristics of meta-analyses, such as performing a thorough literature search, avoidance of bias in the inclusion of studies and appropriately referring to the validity of the included studies. After the release of the QUOROM statement, however, an improvement in the overall quality of published meta-analyses was noted.

CONCLUSION

The overall quality of the reports of meta-analyses available to critical care physicians is poor. Physicians should critically evaluate these studies prior to considering applying the results of these studies in their clinical practice.

Evaluating meta-analyses in the general surgical literature: a critical appraisal

OBJECTIVE

To assess the methodologic quality of meta-analyses of general surgery topics published in peer-reviewed journals.

SUMMARY BACKGROUND DATA

Systematic reviews and meta-analysis are used to seek, summarize, and interpret primary studies on a given topic. Accordingly, systematic reviews and meta-analyses of high-quality primary studies may be the highest level of evidence for issues of prevention and treatment in evidence-based medicine. However, not all published meta-analyses are rigorously performed.

METHODS

We searched MEDLINE (from January 1, 1997, to September 1, 2002) and reference lists and solicited general surgery specialists to identify relevant meta-analyses. Inclusion criteria were use of meta-analytic methods to pool the results of primary studies in general surgery on issues of diagnosis, causation, prognosis, or treatment. Our search strategies identified 487 potentially relevant articles. After excluding articles based on a priori criteria, 51 meta-analyses fulfilled eligibility criteria. In duplicate and independently, 2 reviewers assessed the quality of these meta-analyses using a 10-item index called the Overview Quality Assessment Questionnaire.

RESULTS

Overall concordance between 2 independent reviewers was good (interobserver agreement 81%, and a kappa of 0.62 (95% CI 0.55-0.69). Of 51 relevant articles, 38 were published in surgical journals. Most studies had major methodologic flaws (median score of 3.3, scale of 1-7). Factors associated with low overall scientific quality included the absence of any prior meta-analyses publications by authors and meta-analyses produced by surgical department members without external collaboration.

CONCLUSIONS

This critical appraisal of meta-analyses published in the general surgery literature demonstrates frequent methodologic flaws. The quality of these reports limits the validity of the findings and the inferences that can be made about the primary studies reviewed. To improve the quality of future meta-analyses, we recommend following guidelines for the optimal conduct and reporting of meta-analyses in general surgery.