Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals

Efficacy and safety of pharmacological interventions for the treatment of the Alcohol Withdrawal Syndrome

BACKGROUND

Alcohol abuse and dependence represents a very serious health problem worldwide with major social, interpersonal and legal interpolations. Pharmacological treatments presently used are of uncertain effectiveness and there is even more doubt on the comparative effects and value for money.

OBJECTIVES

To summarize Cochrane reviews that assess the effectiveness and safety of pharmacological interventions in the treatment of alcohol withdrawal.

METHODS

We searched the Cochrane Database of Systematic Reviews (30 November 2010). Two authors independently screened, extracted data, summarised key characteristics of the included reviews and assessed their quality using AMSTAR; the quality of the evidence was summarised according to the GRADE methodology.

MAIN RESULTS

Five reviews, 114 studies, 7333 participants, satisfied criteria for inclusions. The outcomes considered were alcohol withdrawal seizures, adverse events and dropouts. Comparing the five treatments with placebo, benzodiazepines performed better for seizures, three studies, 324 participants, RR 0.16 (95% CI 0.04 to 0.69), moderate quality of evidence. Comparing each of the five treatments versus specific class of drugs, benzodiazepines performed better than antipsychotics for seizures, 4 studies, 633 participants, RR 0.24 (95% CI 0.07 to 0.88) high quality of the evidence. Comparing different benzodiazepines and anticonvulsants among themselves, 28 comparisons, results never reached statistical significance but chlordiazepoxide performed better. The quality of evidence was high for 3% of the results, moderate for 28%, low for 48% and very low for 20%.

AUTHORS' CONCLUSIONS

Among the treatments considered, benzodiazepines showed a protective benefit against seizures, when compared to placebo and a potentially protective benefit for many outcomes when compared with antipsychotics. Nevertheless, no definite conclusions about the effectiveness and safety of benzodiazepines were possible, because of the heterogeneity of the trials both in interventions and in the assessment of outcomes. Data on potential harms are sparse and fragmented. Results do not provide sufficient evidence in favour of anticonvulsants for the treatment of AWS, but anticonvulsants seem to have limited side effects. There is also not enough evidence of effectiveness and safety of baclofen, because only one study consider this treatment and of GHB for which no strong differences were observed in the comparisons with placebo, benzodiazepines and anticonvulsants.

Basics of systematic reviews and meta-analyses for the nephrologist

Renal practitioners are expected to apply the best available evidence from rigorous scientific research to clinical decision-making and also for policy-making for those involved. Advances in information technology and unprecedented access to data have simplified the process for the search of best available evidence to guide practice. However, it is challenging to cope with the increasing volume of publications in nephrology and other areas of medicine. Accordingly, systematic reviews and meta-analysis have greatly facilitated best practice and effective clinical decision-making. Conducting a systematic review/meta-analysis involves a number of steps that start with protocol development and research question formulation, design and study selection criteria, followed by retrieval of potentially relevant studies, selection of those studies to be included and evaluation of a study's risk of bias. Systematic reviews and meta-analyses have both strengths and weaknesses. Many of the perceived limitations of meta-analysis are not inherent in the methodology, but actually represent deficits in the conduct or reporting of individual primary studies. With the continuous proliferation of published renal clinical studies, such publications will continue to be an important resource for clinicians and researchers in nephrology. It is therefore important for nephrologists to keep abreast of developments in this field, which requires some knowledge about how these studies are conducted, reported and how to appraise them for application to clinical practice or policy-making.

Chronic fatigue syndrome in the media: a content analysis of newspaper articles

OBJECTIVES

Although cognitive behavioural therapy and graded exercise treatment are recognized evidence-based treatments for chronic fatigue syndrome/myalgic encephalomyelitis (ME), their use is still considered controversial by some patient groups. This debate has been reflected in the media, where many patients gather health information. The aim of this study was to examine how treatment for chronic fatigue syndrome/ME is described in the newspaper media.

DESIGN

Content analysis of newspaper articles.

SETTING

The digitalized media archive Atekst was used to identify Norwegian newspaper articles where chronic fatigue syndrome/ME was mentioned.

PARTICIPANTS

Norwegian newspaper articles published over a 20-month period, from 1 January 2008 to 31 August 2009.

MAIN OUTCOME MEASURES

Statements regarding efficiency of various types of treatment for chronic fatigue syndrome/ME and the related source of the treatment advice. Statements were categorized as being either positive or negative towards evidence-based or alternative treatment.

RESULTS

One hundred and twenty-two statements regarding treatment of chronic fatigue syndrome/ME were identified among 123 newspaper articles. The most frequent statements were positive statements towards alternative treatment Lightning Process (26.2%), negative statements towards evidence-based treatments (22.1%), and positive statements towards other alternative treatment interventions (22.1%). Only 14.8% of the statements were positive towards evidence-based treatment. Case-subjects were the most frequently cited sources, accounting for 35.2% of the statements, followed by physicians and the Norwegian ME association.

CONCLUSIONS

Statements regarding treatment for chronic fatigue syndrome/ME in newspapers are mainly pro-alternative treatment and against evidence-based treatment. The media has great potential to influence individual choices. The unbalanced reporting of treatment options for chronic fatigue syndrome/ME in the media is potentially harmful.

Hepato-biliary clinical trials and their inclusion in the Cochrane Hepato-Biliary Group register and reviews

BACKGROUND AND AIMS

The Cochrane Hepato-Biliary Group (CHBG) is one of the 52 collaborative review groups within The Cochrane Collaboration. The activities of the CHBG focus on collecting hepato-biliary randomized clinical trials (RCT) and controlled clinical trials (CCT), and including them in systematic reviews with meta-analyses of the trials. In this overview, we present the growth of The CHBG Controlled Trials Register, as well as the systematic reviews that have been produced since March 1996.

RESULTS

The CHBG register includes almost 11,000 RCT and 700 CCT publications. The earliest RCT in the register were published in 1955, and the earliest CCT in 1945. From 1945 to 1980, there were less than 100 publications each year. From 1981 to 1997, their number increased from over 100 to 600 a year. After 1997, the number of publications seems to have been decreasing. The CHBG has published 199 protocols for systematic reviews and 107 systematic reviews through to August 2009 in which 21% of the RCT and CCT were included. The CHBG reviews have been cited approximately 1200 times.

CONCLUSIONS

A large amount of work has been carried out since 1996. However, there is still much to do, as the CHBG register contains a great number of RCT and CCT on topics that have not yet been systematically reviewed.

Interpreting systematic reviews: are we ready to make our own conclusions? A cross-sectional study

BACKGROUND

Independent evaluation of clinical evidence is advocated in evidence-based medicine (EBM). However, authors' conclusions are often appealing for readers who look for quick messages. We assessed how well a group of Malaysian hospital practitioners and medical students derived their own conclusions from systematic reviews (SRs) and to what extent these were influenced by their prior beliefs and the direction of the study results.

METHODS

We conducted two cross-sectional studies: one with hospital practitioners (n = 150) attending an EBM course in June 2008 in a tertiary hospital and one with final-year medical students (n = 35) in November 2008. We showed our participants four Cochrane SR abstracts without the authors' conclusions. For each article, the participants chose a conclusion from among six options comprising different combinations of the direction of effect and the strength of the evidence. We predetermined the single option that best reflected the actual authors' conclusions and labelled this as our best conclusion. We compared the participants' choices with our predetermined best conclusions. Two chosen reviews demonstrated that the intervention was beneficial ("positive"), and two others did not ("negative"). We also asked the participants their prior beliefs about the intervention.

RESULTS

Overall, 60.3% correctly identified the direction of effect, and 30.1% chose the best conclusions, having identified both the direction of effect and the strength of evidence. More students (48.2%) than practitioners (22.2%) chose the best conclusions (P < 0.001). Fewer than one-half (47%) correctly identified the direction of effect against their prior beliefs. "Positive" SRs were more likely than "negative" SRs to change the participants' beliefs about the effect of the intervention (relative risk (RR) 1.8, 95% confidence interval 1.3 to 2.6) and "convert" those who were previously unsure by making them choose the appropriate direction of effect (RR 1.9, 95% confidence interval 1.3 to 2.8).

CONCLUSIONS

The majority of our participants could not generate appropriate conclusions from SRs independently. Judicious direction from the authors' conclusions still appears crucial to guiding our health care practitioners in identifying appropriate messages from research. Authors, editors and reviewers should ensure that the conclusions of a paper accurately reflect the results. Similar studies should be conducted in other settings where awareness and application of EBM are different.

How useful are meta-analyses in orthopedic trauma?

BACKGROUND

Meta-analysis has become accepted as a methodically rigorous research tool, and as a result, many meta-analyses have been undertaken in orthopedic trauma, the implication being that their conclusions have improved surgeons' knowledge and facilitated improved clinical care. There have been criticisms of the methodology used in a number of meta-analyses; however, there has been no previous study of the clinical usefulness of their conclusions.

METHODS

A study of 60 orthopedic trauma meta-analyses published during or after 2000 was undertaken. We divided the conclusions of the meta-analyses into four types depending on their innovation and potential for improving clinical care. We also separated the meta-analyses according to whether they were undertaken by the Cochrane Collaboration or by independent surgeons.

RESULTS

Only 4 (6.7%) of the meta-analyses contained new, clinically useful, information, and 22 (36.7%) had no conclusions at all. A further 25 (41.7%) contained conclusions that were already in the standard orthopedic trauma literature. Approximately 70% of the Cochrane Collaborations had no conclusions.

CONCLUSIONS

We found considerable variability in the clinical usefulness of orthopedic trauma meta-analyses, and we question the clinical usefulness of this type of research.

The prevalence and effect of publication bias in orthopaedic meta-analyses

BACKGROUND

Meta-analyses of randomized, controlled trials are considered the highest level-of-evidence, thus strongest source of information. However, questions concerning the validity of meta-analyses in orthopaedic surgery emerged recently. Among the most common sources for errors is publication bias. This describes the fact that studies with small or non-significant outcomes are less likely to be published, thus less likely to be identified and included in systematic reviews and meta-analyses. In this study we asked three questions: (1) Whether publication bias is assessed in orthopaedic meta-analyses, (2) What the actual prevalence of publication bias is, and (3) what effect publication bias has on the outcomes of orthopaedic meta-analyses.

METHODS

Using the estimate of 35 ± 20% for the prevalence of publication bias in meta-analyses obtained from earlier research we calculated a required samples size of 22 (plus 20% to account for attrition) and randomly selected 26 orthopaedic meta-analyses. To answer our first question we calculated the percentage of papers that report on formal assessment of publication bias. For our second question we obtained all primary studies (n = 321) from the included meta-analyses and used Egger's regression to search for evidence for publication bias. Third, we used the trim-and-fill method to assess the impact of publication bias, if present. This method estimates publication bias in a meta-analysis and adds hypothetical studies to reduce this bias; it thus produces an estimate of adjusted, unbiased outcomes that can be compared with the unadjusted, publication-biased outcomes to assess the effect of publication bias.

RESULTS

We found that only 35% (95% CI 20-57) of all orthopaedic meta-analyses published between 1992 and 2008 in English and German assessed publication bias. Most studies used funnel plots, which are rather insensitive. The prevalence of publication bias, based on a sensitivity analysis, ranged between 12 and 19%. Adjustment for publication bias did not produce significantly different results, but the magnitude of the pooled estimates in the affected meta-analyses changed by 29% (95% CI 0-63) on average.

CONCLUSION

We found a rather low prevalence of publication bias in orthopaedic meta-analyses, but recommend assessing for it and its effects, which might be substantial.

Practicing evidence-based psychiatry- 2. Interpreting integrative literature: Systematic reviews and meta-analyses

The practice of evidence-based medicine (EBM) requires the ability to adequately appraise relevant clinical evidence with the intention of making an evidence-based treatment decision. Broadly, there are three available sources of information available to the practicing clinician: individual studies, reviews of the literature, and practice guidelines. In the first article in the series, we described the threats to validity (T2V) approach in evaluating evidence and specifically discussed the different threats to validity in applying the findings of a single study to an individual patient. In this article, we describe the five steps of a systematic review and evaluate threats to validity at each of these steps. We illustrate the method by evaluating a review relevant to the treatment question presented in a clinical vignette. We discuss results of the meta-analytic review about comparative antipsychotic effectiveness in patients with schizophrenia and consider application of its conclusions to the question of which antipsychotic to select for the particular patient with schizophrenia utilizing the T2V approach.

Effectiveness of CAM therapy: understanding the evidence

By definition, complementary and alternative medicine (CAM) attempts to diagnose and treat illnesses in unconventional ways. CAM has been classified as: (1) alternative medical systems (eg, traditional Chinese medicine [including acupuncture], naturopathic medicine, ayurvedic medicine, and homeopathy); (2) biologic-based therapies (eg, herbal, special dietary, and individual biologic treatments); (3) energy therapies (eg, Reiki, therapeutic touch, magnet therapy, Qi Gong, and intercessory prayer); (4) manipulative and body-based systems (eg, chiropractic, osteopathy, and massage); and (5) mind-body interventions (eg, meditation, biofeedback, hypnotherapy, and the relaxation response). This review focuses on how to assess the effectiveness of CAM therapies for chronic musculoskeletal pains, emphasizing the role of specific and nonspecific analgesic mechanisms, including placebo.

How to write a Cochrane systematic review

The Cochrane Collaboration is a global network whose aim is to improve health-care decision making through systematic reviews of the effects of health-care interventions. Cochrane systematic reviews are published in the Cochrane Database of Systematic Reviews within The Cochrane Library ( http://www.thecochranelibrary.com), and regularly updated as new evidence arises. Cochrane Reviews are undertaken by teams of volunteer authors, who have access to free training resources, reference texts and software for preparing and maintaining their review. Here we aim to describe the steps involved to undertake a new or update an existing Cochrane Review.

Emotional intelligence in medicine: a systematic review through the context of the ACGME competencies

OBJECTIVES

Emotional intelligence (EI) involves the perception, processing, regulation and management of emotions. This article aims to systematically review the evidence for EI in medicine through the context of the Accreditation Council for Graduate Medical Education (ACGME) competencies.

METHODS

MEDLINE, EMBASE, PsycINFO, the Cochrane Database of Systematic Reviews and the Cochrane Central Register of Controlled Trials were searched for English-language articles published between January 1980 and March 2009. The grey literature was also searched and experts in the field contacted for additional studies. Two independent reviewers selected articles which reported empirical research studies about clinicians or medical students. Conceptual articles and opinion pieces and commentaries were excluded. Information about the measure used to assess EI, the study parameter or domain, and the educational or clinical outcome (with specific relation to the ACGME competencies) was extracted.

RESULTS

The literature search identified 485 citations. An abstract review led to the retrieval of 24 articles for full-text assessment, of which 16 articles were included in the final review. Eleven studies focused on postgraduates, four on undergraduates and one on medical school applicants. Six out of seven studies found women to have higher EI than men. Higher EI was reported to positively contribute to the doctor-patient relationship (three studies), increased empathy (five studies), teamwork and communication skills (six studies), and stress management, organisational commitment and leadership (three studies).

CONCLUSIONS

Measures of EI correlate with many of the competencies that modern medical curricula seek to deliver. Further research is required to determine whether training can improve EI and thus augment educational and clinical outcomes.

Expediting systematic reviews: methods and implications of rapid reviews

BACKGROUND

Policy makers and others often require synthesis of knowledge in an area within six months or less. Traditional systematic reviews typically take at least 12 months to conduct. Rapid reviews streamline traditional systematic review methods in order to synthesize evidence within a shortened timeframe. There is great variation in the process of conducting rapid reviews. This review sought to examine methods used for rapid reviews, as well as implications of methodological streamlining in terms of rigour, bias, and results.

METHODS

A comprehensive search strategy--including five electronic databases, grey literature, hand searching of relevant journals, and contacting key informants--was undertaken. All titles and abstracts (n = 1,989) were reviewed independently by two reviewers. Relevance criteria included articles published between 1995 and 2009 about conducting rapid reviews or addressing comparisons of rapid reviews versus traditional reviews. Full articles were retrieved for any titles deemed relevant by either reviewer (n = 70). Data were extracted from all relevant methodological articles (n = 45) and from exemplars of rapid review methods (n = 25).

RESULTS

Rapid reviews varied from three weeks to six months; various methods for speeding up the process were employed. Some limited searching by years, databases, language, and sources beyond electronic searches. Several employed one reviewer for title and abstract reviewing, full text review, methodological quality assessment, and/or data extraction phases. Within rapid review studies, accelerating the data extraction process may lead to missing some relevant information. Biases may be introduced due to shortened timeframes for literature searching, article retrieval, and appraisal.

CONCLUSIONS

This review examined the continuum between diverse rapid review methods and traditional systematic reviews. It also examines potential implications of streamlined review methods. More of these rapid reviews need to be published in the peer-reviewed literature with an emphasis on articulating methods employed. While one consistent methodological approach may not be optimal or appropriate, it is important that researchers undertaking reviews within the rapid to systematic continuum provide detailed descriptions of methods used and discuss the implications of their chosen methods in terms of potential bias introduced. Further research comparing full systematic reviews with rapid reviews will enhance understanding of the limitations of these methods.

Applicability and generalisability of the results of systematic reviews to public health practice and policy: a systematic review

BACKGROUND

The purpose of the study was to evaluate systematic reviews of research into two public health priorities, tobacco consumption and HIV infection, in terms of the reporting of data related to the applicability of trial results (i.e., whether the results of a trial can be reasonably applied or generalized to a definable group of patients in a particular setting in routine practice, also called external validity or generalisability).

METHODS

All systematic reviews of interventions aimed at reducing or stopping tobacco use and treating or preventing HIV infection published in the Cochrane database of systematic reviews and in journals indexed in MEDLINE between January 1997 and December 2007 were selected. We used a standardized data abstraction form to extract data related to applicability in terms of the context of the trial, (country, centres, settings), participants (recruitment, inclusion and exclusion criteria, baseline characteristics of participants such as age, sex, ethnicity, coexisting diseases or co-morbidities, and socioeconomic status), treatment (duration, intensity/dose of treatment, timing and delivery format), and the outcomes assessment from selected reviews.

RESULTS

A total of 98 systematic reviews were selected (57 Cochrane reviews and 41 non-Cochrane reviews); 49 evaluated interventions aimed at reducing or stopping tobacco use and 49 treating or preventing HIV infection. The setting of the individual studies was reported in 45 (46%) of the systematic reviews, the number of centres in 21 (21%), and the country where the trial took place in 62 (63%). Inclusion and exclusion criteria of the included studies were reported in 16 (16%) and 13 (13%) of the reviews, respectively. Baseline characteristics of participants in the included studies were described in 59 (60%) of the reviews. These characteristics concerned age in about half of the reviews, sex in 46 (47%), and ethnicity in 9 (9%).Applicability of results was discussed in 13 (13%) of the systematic reviews. The reporting was better in systematic reviews by the Cochrane Collaboration than by non-Cochrane groups.

CONCLUSIONS

Our study highlighted the lack of consideration of applicability of results in systematic reviews of research into 2 public health priorities: tobacco consumption and HIV infection.

Evidence-based medicine in psychopharmacotherapy: possibilities, problems and limitations

Psychopharmacotherapy should now be regulated in the sense of evidence-based medicine, as is the case in other areas of clinical treatment in medicine. In general this is a meaningful development, which principally will have a positive impact on routine health care in psychiatry. But several related problems should not be ignored. So far consensus on an internationally accepted evidence graduation could not be reached due to several difficulties related to this. For example, focussing on the results of meta-analyses instead of considering relevant single studies results in a decision-making logic which is in conflict with the rationale applied by drug authorities in the licensing process. Another example is the relevance of placebo-controlled trials: if randomized placebo-controlled phase-III studies are prioritized in the evidence grading, the evidence possibly deviates too far from the conditions of routine clinical care due to the special selection of patients in those studies. However, a grading primarily based on active comparator trials could lead to wrong conclusions about efficacy. This concerns especially the so-called "effectiveness" studies and other forms of phase-IV studies with their less restrictive methodological rigidity. Attempts to regulate psychopharmacotherapy in the sense of evidence-based medicine come closer to their limits the more complex the clinical situation and the respective decision-making logic are. Even in times of evidence-based medicine a large part of complex clinical decision-making in psychopharmacotherapy still relies more on clinical experience and a consensus on clinical experience, traditions and belief systems than on results of efficacy oriented phase-III and effectiveness-oriented phase-IV clinical studies.

Errors in the conduct of systematic reviews of pharmacological interventions for irritable bowel syndrome

OBJECTIVES

Systematic reviews and meta-analyses are integral to evidence-based clinical decision making. Although flawed systematic reviews could compromise optimal decision making, their accuracy has received limited investigation. We assessed conduct of systematic reviews of pharmaceutical interventions for irritable bowel syndrome (IBS).

METHODS

We searched MEDLINE, EMBASE, and the Cochrane Database of Systematic Reviews (up to June 2008) to identify and replicate all published systematic reviews and meta-analyses that examined pharmacological interventions for IBS. We identified trials appropriately and inappropriately included according to the investigators' own eligibility criteria and eligible trials the investigators failed to include, and assessed the accuracy of dichotomous data extraction from all truly eligible trials. We conducted meta-analyses of accurate data from all truly eligible trials, and examined the differences between these accurate estimates and those reported by the authors.

RESULTS

The search strategy identified 120 citations, and 13 appeared to be relevant. Five systematic reviews did not extract dichotomous data, leaving eight eligible for inclusion. In five of the eight meta-analyses 13-29% of included trials were ineligible according to investigators' criteria, constituting 8-26% of included patients. Six of the meta-analyses missed 17 separate published eligible trials; 3-11% of eligible patients were, as a result, not included. All eight meta-analyses contained errors in dichotomous data extraction, in 29-100% of truly eligible trials, leading to errors in 15 of 16 reported pooled treatment effects. There was a > or =10% relative difference in treatment effects between the reported and recalculated summary statistic in five (31%) cases, and a change in the statistical significance of the recalculated summary statistic in a further four (25%) cases.

CONCLUSIONS

We found many errors in both application of eligibility criteria and dichotomous data extraction in the eight meta-analyses studied. Independent verification of systematic reviews and meta-analyses may be required for full confidence in their results.

Cochrane reviews of randomized trials of fluoride therapies for preventing dental caries

AIM

To present the evidence summarized in the Cochrane fluoride reviews.

STUDY DESIGN

An overview of the results of selected systematic reviews.

METHODS

Relevant systematic reviews published in the Cochrane Database of Systematic Reviews (CDSR) were identified by searching 'The Cochrane Library issue 4, 2008', using the terms 'Fluoride' and 'Caries'. Complete Cochrane reviews assessing the effectiveness of any fluoride-based intervention for preventing caries were selected, and their main features and findings were reviewed.

RESULTS

14 papers were identified of which 11 were relevant full-text reviews. The results were assessed of 7 reviews published from 2002 to 2004 concerning the relative effectiveness of 4 topical fluoride treatments (toothpastes, gels, varnishes and mouthrinses) in preventing caries in children and adolescents. Comparisons in these reviews were made against non-fluoride controls, against each other, and against different combinations. Findings from 4 reviews published between 2004 and 2006, assessing other fluoride modalities (slow release devices, milk), specific comparison/site (fluoride varnishes versus sealants in occlusal surfaces), and particular population and caries outcome (fluorides for white spot lesions in orthodontic patients) were also assessed. The 7 reviews confirm a clear and similar effectiveness of topical fluoride toothpastes, mouthrinses, gels and varnishes for preventing caries, and show that additional caries reduction can be expected when another topical fluoride is combined with fluoride toothpaste. Evidence is insufficient to confirm the effectiveness of slow release fluoride devices and fluoridated milk. The comparative effectiveness of other modes of delivering fluoride, such as to orthodontic patients is also as yet unclear. Fissure sealants appear more effective than fluoride varnish for preventing occlusal caries but the size of the difference is unclear.

CONCLUSIONS

The benefits of topical fluorides are firmly established based on a sizeable body of evidence from randomized controlled trials. The size of the reductions in caries increment in both the permanent and the primary dentitions emphasizes the importance of including topical fluoride delivered through toothpastes, rinses, gels or varnishes in any caries preventive program. However, trials to discern potential adverse effects are required, and data on acceptability. Better quality research is needed to reach clearer conclusions on the effects of slow release fluoride devices, milk fluoridation, sealants in comparison with fluoride varnishes, and of different modes of delivering fluoride to orthodontic patients.

Systematic reviews on tobacco control from Cochrane and the Community Guide: different methods, similar findings

OBJECTIVES

To compare the methods and findings of systematic reviews (SRs) on common tobacco control interventions from two organizations: the Cochrane Collaboration ("Cochrane") and the US Task Force for Community Preventive Services ("the Guide").

STUDY DESIGN AND SETTING

Literature review. We retrieved all reviews pertaining to tobacco control produced by the Cochrane and the Guide. We identified seven common topics and compared methods and findings of the retrieved reviews.

RESULTS

There was considerable variability in the designs of included studies and methods of data synthesis. On average, Cochrane identified more studies than did the Guide (Mean 43.7 vs. 19.0), with only limited overlap between sets of included studies. Most Cochrane reviews (71.4%) were synthesized narratively, whereas most Guide reviews (85.7%) were synthesized using a median of effect size. Despite these differences, findings of the reviews yielded substantial agreement.

CONCLUSION

Cochrane and the Guide conduct SRs on similar tobacco control-related topics differently. The SRs of the two organizations include overlapping, but nonidentical sets, of studies. Still, they usually reach similar conclusions. Identification of all pertinent original studies seems to be a weak point in the SR process. Policy makers should use reviews from both organizations in formulating tobacco control policy.

Twenty years of meta-analyses in orthopaedic surgery: has quality kept up with quantity?

BACKGROUND

As the number of studies in the literature is increasing, orthopaedic surgeons highly depend on meta-analyses as their primary source of scientific evidence. The objectives of this review were to assess the scientific quality and number of published meta-analyses on orthopaedics-related topics over time.

METHODS

We conducted, in duplicate and independently, a systematic review of published meta-analyses in orthopaedics in the years 2005 and 2008 and compared them with a previous systematic review of meta-analyses from 1969 to 1999. A search of electronic databases (MEDLINE, EMBASE, and the Cochrane Database of Systematic Reviews) was performed to identify meta-analyses published in 2005 and 2008. We searched bibliographies and contacted content experts to identify additional relevant studies. Two investigators independently assessed the quality of the studies, using the Oxman and Guyatt index, and abstracted relevant data.

RESULTS

We included forty-five and forty-four meta-analyses from 2005 and 2008, respectively. While the number of meta-analyses increased fivefold from 1999 to 2008, the mean quality score did not change significantly over time (p = 0.067). In the later years, a significantly lower proportion of meta-analyses had methodological flaws (56% in 2005 and 68% in 2008) compared with meta-analyses published prior to 2000 (88%) (p = 0.006). In 2005 and 2008, respectively, 18% and 30% of the meta-analyses had major to extensive flaws in their methodology. Studies from 2008 with positive conclusions used and described appropriate criteria for the validity assessment less often than did those with negative results. The use of random-effects and fixed-effects models as pooling methods became more popular toward 2008.

CONCLUSIONS

Although the methodological quality of orthopaedic meta-analyses has increased in the past twenty years, a substantial proportion continues to show major to extensive flaws. As the number of published meta-analyses is increasing, a routine checklist for scientific quality should be used in the peer-review process to ensure methodological standards for publication.

Why are Cochrane hepato-biliary reviews undervalued by physicians as an aid for clinical decision-making?

BACKGROUND

Cochrane systematic reviews are of higher quality than reviews published in scientific journals, yet are used less than other sources for clinical decision-making.

AIM

To assess whether the characteristics of the Cochrane systematic reviews can account for their scant use by physicians.

MATERIALS AND METHODS

We analysed the 87 Cochrane hepato-biliary reviews dealing with therapeutic topics posted in the Cochrane Database of Systematic Reviews through December 2008, which we classified according to four characteristics: empty reviews; outdated reviews; content of reviews; implications for practice.

RESULTS

Six empty reviews found no eligible randomised trials and six found one trial, precluding a systematic review; some empty reviews investigated irrelevant topics. Twenty-one reviews investigated outdated interventions, and thirteen of them were posted ten or more years after the publication of the most recent trial included. Most reviews were too lengthy (median: 40 pages) and their consultation was time-consuming with respect to clinical content. They generally compared two treatments, disregarding other options, and usually did not report any non-randomised (although convincing) evidence of potential use in clinical decision-making.

CONCLUSIONS

If generalized to the entire Cochrane Database of Systematic Reviews, these characteristics may largely explain why physicians undervalue the Cochrane reviews as a source of evidence for clinical decision-making.

SUPPORT Tools for evidence-informed health Policymaking (STP) 8: Deciding how much confidence to place in a systematic review

This article is part of a series written for people responsible for making decisions about health policies and programmes and for those who support these decision makers. The reliability of systematic reviews of the effects of health interventions is variable. Consequently, policymakers and others need to assess how much confidence can be placed in such evidence. The use of systematic and transparent processes to determine such decisions can help to prevent the introduction of errors and bias in these judgements. In this article, we suggest five questions that can be considered when deciding how much confidence to place in the findings of a systematic review of the effects of an intervention. These are: 1. Did the review explicitly address an appropriate policy or management question? 2. Were appropriate criteria used when considering studies for the review? 3. Was the search for relevant studies detailed and reasonably comprehensive? 4. Were assessments of the studies' relevance to the review topic and of their risk of bias reproducible? 5. Were the results similar from study to study?

The quality of meta-analyses of genetic association studies: a review with recommendations

Although there has been a rapid rise in the publication of meta-analyses of genetic association studies, little is known about their methodological quality. The authors reviewed the quality of 120 randomly selected genetic meta-analyses published between 2005 and 2007. Data extracted included issues of general relevance and other issues specific to genetic epidemiology. Quality was markedly poorer in the 26% of the meta-analyses that accompanied a report on a primary study. Such meta-analyses were predominantly published in specialist journals, and their quality was positively associated with the impact factor of the journal. Among the meta-analyses that did not accompany a primary study, Human Genome Epidemiology reviews tended to score better than the others, although the comparison was limited by relatively small numbers. Comparison of the overall quality with that of genetic meta-analyses published before 2000 showed improvement in both conduct and reporting. However, the quality of the handling of specific genetic issues remains disappointingly low. For a few key general quality issues, the authors compared their findings with findings in other fields of medicine and found that general quality was similar. On the basis of this review, the authors provide practical recommendations for the conduct and reporting of genetic meta-analyses.

Systematic reviews to support evidence-based psychiatry: what about schizophrenia?

OBJECTIVE

To assess whether systematic reviews (SRs), the gold standard for scientific research, can offer valuable support in evidence-based psychiatry in the treatment of schizophrenia.

METHODS

We used three database services (Ovid, PubMed and Cochrane) to identify SRs related to schizophrenia, found 163 reviews and grouped them by topic. We then evaluated each study's conclusions and divided them into three groups based on results (ranging from certain to null conclusions).

RESULTS

SRs of pharmacological treatments represented 59% of the studies sampled, only 23% of which had reached certain conclusions. Other clinical topics were less frequently represented and had achieved lower degrees of certainty.

CONCLUSIONS

Only 40 SRs (22 studies investigating pharmacological treatment) provided clear-cut answers to clinical questions examined. Results therefore showed that SRs provide a certain but rather limited contribution to scientific evidence in the field of schizophrenia.

Why be an evidence-based dentistry champion?

Evidence-based dentistry champions are committed to improving the quality, effectiveness, and appropriateness of dental care through the application of evidence-based principles and tools. They share knowledge and skills to promote evidence-based dentistry (EBD) in practice, guiding colleagues, patients and policy makers in the application of critical thinking skills and evidence-based decision-making. Being an evidence-based champion requires furthering an understanding of the full process and the challenges of evidence based dentistry, including the development of an evidence base, evidence synthesis and summary, the creation of best practice guidelines, as well as evidence implementation. Efforts to improve the quality, effectiveness, and appropriateness of dental care need to occur at, and be coordinated across, multiple levels of dentistry, including the patient, clinician, team, organization, and policy.

Assessment of the Methodologic Quality of Medical and Surgical Clinical Trials in Patients with Arthroplasty

Objective.

To assess the methodological quality of randomized controlled trials (RCT) of medical and surgical therapy in patients with arthroplasty.

Methods.

We conducted a Medline database search for all arthroplasty RCT from 1997 and 2006. The quality of the methods of all eligible RCT was assessed by a trained abstractor. We used a checklist of trial quality characteristics, and the overall trial quality was assessed by 3 scales: Jadad (range 0–5), Delphi list (range 0–9), and numeric rating scale (NRS; range 1–10), based on User’s Guides to the Medical Literature.

Results.

A total of 196 articles were included in the analysis; most included hip (n = 81) or knee (n = 80) or both hip/knee arthroplasty (n = 19); 66 (34%) assessed pharmacological treatments, 117 (60%) nonpharmacological treatments, and 13 (7%) both. Mean (SEM) overall quality scores of arthroplasty RCT were low: Jadad score 2.36 (1.4), Delphi list 5.33 (1.6), and NRS score 4.30 (2.6). Multivariable analyses revealed that nonpharmacological intervention RCT had lower odds (odds ratio 0.28–0.39; p = 0.008–0.033) and those with no funding had lower odds (OR 0.28–0.50; p = 0.014–0.119) of being in the highest quartiles of the 3 overall quality scores. In contrast, multicenter RCT had 1.8–4.7 times higher odds of being in highest tertiles of quality scores (p = 0.017–0.185).

Conclusion.

Methodological deficiencies in reporting of hip/knee arthroplasty RCT offer an opportunity for improvement. Type of intervention, number of trial centers, and presence of funding were independently associated with overall trial quality. In future, multicenter RCT (rather than single-center) and modeling protocols of single-center RCT similar in rigor to multicenter RCT may improve the quality of arthroplasty RCT.

Archie Cochrane and his vision for evidence-based medicine

Archibald (Archie) Cochrane's most influential mark on health care was his 1971 publication, Effectiveness and Efficiency. This book strongly criticized the lack of reliable evidence behind many of the commonly accepted health care interventions at the time. His criticisms spurred rigorous evaluations of health care interventions and highlighted the need for evidence in medicine. His call for a collection of systematic reviews led to the creation of The Cochrane Collaboration. Archie Cochrane was a visionary person who helped lay down much of the foundation for evidence-based medicine. This article introduces evidence-based medicine to plastic surgery by tracing its history to the seminal efforts by Archie Cochrane.

Venous thromboembolism prophylaxis after major orthopaedic surgery: a pooled analysis of randomized controlled trials

The use of aspirin for venous thromboembolism (VTE) prophylaxis after major orthopaedic surgery is controversial. The hypothesis of the present study is that aspirin will decrease the rate of operative site bleeding without increasing thromboembolic events when aspirin is used for VTE prophylaxis after major orthopaedic surgery. A pooled analysis of 14 randomized controlled trials (RCTs) cited by the American College of Chest Physicians (ACCP) guidelines was performed to determine pooled rates of symptomatic deep vein thromboses, pulmonary emboli (PE), fatal PE, and operative site bleeding rates. The VTE rates with aspirin were not significantly different than the rates for vitamin K antagonists (VKA), low molecular weight heparins (LMWH), and pentasaccharides. The operative site bleeding relative risks of VKA, LMWH, and pentasaccharides versus aspirin, are 4.9, 6.4, and 4.2, respectively. A pooled analysis of RCTs supports the use of aspirin for VTE prophylaxis after major orthopaedic surgery.

Angiotensin-Converting Enzyme Inhibitors and Angiotensin Receptor Blockers in Peritoneal Dialysis: Systematic Review and Meta-Analysis of Randomized Controlled Trials

Background

Angiotensin-converting enzyme inhibitors (ACEIs) and angiotensin receptor blockers (ARBs) are widely used in clinical practice. The safety and efficacy of these agents in peritoneal dialysis (PD) patients are unclear.

Objectives

We conducted a systematic review to study the safety and efficacy of ACEI and ARB use in PD patients. Primary outcome measures were mortality and cardiovascular (CV) events; secondary outcome measures were renal function, proteinuria, hyperkalemia, and erythropoietin requirement at 3 months.

Methods

We searched Medline, EMBASE, Cochrane Central Register of Controlled Trials, trial registry Web sites, reference lists of eligible and review articles, as well as abstracts from the American Society of Nephrology and Canadian Society of Nephrology meetings. To be eligible, studies had to be randomized controlled trials that allocated PD patients to ACEI and ARB use or to placebo or other antihypertensive medications, included adult patients, and reported on at least one of the outcome measures.

Results

418 citations were identified. Four met the eligibility criteria. Three examined CV events and mortality, of which two studies did not have any events. The third showed no statistically significant difference between control and treatment groups in either CV events or mortality: odds ratio 1.56 [95% confidence interval (CI) 0.24 – 10.05] for mortality and odds ratio 1.00 (95% CI 0.19 – 5.40) for CV events. Two studies reported renal function at 12 months and the weighted mean difference was 0.91 mL/minute/1.73 m2 (95% CI 0.14 – 1.68), favoring ACEI and ARB use.

Conclusions

In PD patients, evidence for the use of ACEIs and ARBs for reduction of mortality and CV events is lacking. Limited data suggest that they slow the loss of residual renal function.

Evidence from the Cochrane Collaboration for Traditional Chinese Medicine therapies

BACKGROUND

The Cochrane Collaboration, an international not-for-profit organization that prepares and maintains systematic reviews of randomized trials of health care therapies, has produced reviews summarizing much of the evidence on Traditional Chinese Medicine (TCM). Our objective was to review the evidence base according to Cochrane systematic reviews.

METHODS

In order to detect reviews focusing on TCM, we searched the titles and abstracts of all reviews in Issue 4, 2008 of the Cochrane Database of Systematic Reviews. For each review, we extracted data on the number of trials included and the total number of participants. We provided an indication of the strength of the review findings by assessing the reviewers' abstract conclusions statement. We supplemented our assessment of the abstract conclusions statements with a listing of the comparisons and outcomes showing statistically significant meta-analyses results.

RESULTS

We identified 70 Cochrane systematic reviews of TCM, primarily acupuncture (n = 26) and Chinese herbal medicine (n = 42), and 1 each of moxibustion and t'ai chi. Nineteen (19) of 26 acupuncture reviews and 22/42 herbal medicine reviews concluded that there was not enough good quality trial evidence to make any conclusion about the efficacy of the evaluated treatment, while the remaining 7 acupuncture and 20 herbal medicine reviews and each of the moxibustion and t'ai chi reviews indicated a suggestion of benefit, which was qualified by a caveat about the poor quality and quantity of studies. Most reviews included many distinct interventions, controls, outcomes, and populations, and a large number of different comparisons were made, each with a distinct forest plot.

CONCLUSIONS

Most Cochrane systematic reviews of TCM are inconclusive, due specifically to the poor methodology and heterogeneity of the studies reviewed. Some systematic reviews provide preliminary evidence of Chinese medicine's benefits to certain patient populations, underscoring the importance and appropriateness of further research. These preliminary findings should be considered tentative and need to be confirmed with rigorous randomized controlled trials.

Trial sample size, but not trial quality, is associated with positive study outcome

OBJECTIVE

To assess whether the reported trial quality or trial characteristics are associated with the trial outcome.

STUDY DESIGN AND SETTING

We identified all eligible randomized controlled trials (RCTs) of arthroplasty from 1997 and 2006. Trials were classified based on whether the main trial outcome was reported to be positive (n = 90) or negative (n = 94). Multivariable logistic regression analyses studied the association of reporting of trial-quality measures (blinding, placebo use, allocation procedure, overall quality) and trial characteristics (intervention type, number of patients/centers, funding) with positive trial outcome.

RESULTS

RCTs that used placebo or blinded care providers, used pharmacological interventions, had higher Jadad quality scores or sample size of more than 100 patients were significantly more likely to report positive result in univariate analyses. Multivariable regression did not identify methodological quality of RCTs, but rather found that sample size was associated with trial outcome. Studies with more than 100 patients were 2.2 times more likely to report a positive result than smaller studies (P = 0.04).

CONCLUSIONS

Lack of association of reported trial quality with positive outcome in multivariable analyses suggests that previously observed association of reported study quality with study outcome in univariate analyses may be mediated by other study characteristics, such as study sample size.

The origins, evolution, and future of The Cochrane Database of Systematic Reviews

The Cochrane Database of Systematic Reviews (CDSR) evolved in response to Archie Cochrane's challenge to the medical profession to assemble "a critical summary, adapted periodically, of all ... relevant randomized controlled trials". CDSR has been an electronic publication from its inception and this has meant that Cochrane reviews (i) need not be constrained by lack of space; (ii) can be updated as new information becomes available and when mistakes or other ways of improving them are identified; and (iii) can be cross-linked to other, related sources of relevant information. Although CDSR has become widely cited, it must continue to evolve in the light of technological and methodological developments, and in response to the needs of people making decisions about health care.

Reporting and methodologic quality of Cochrane Neonatal review group systematic reviews

Background

The Cochrane Neonatal Review Group (CNRG) has achieved a lot with limited resources in producing high quality systematic reviews to assist clinicians in evidence-based decision-making. A formal assessment of published CNRG systematic reviews has not been undertaken; we sought to provide a comprehensive assessment of the quality of systematic reviews (both methodologic and reporting quality) published in CNRG.

Methods

We selected a random sample of published CNRG systematic reviews. Items of the QUOROM statement were utilized to assess quality of reporting, while items and total scores of the Oxman-Guyatt Overview Quality Assessment Questionnaire (OQAQ) were used to assess methodologic quality. Two reviewers independently extracted data and assessed quality. A Student t-test was used to compare quality scores pre- and post-publication of the QUOROM statement.

Results

Sixty-one systematic reviews were assessed. Overall, the included reviews had good quality with minor flaws based on OQAQ total scores (mean, 4.5 [0.9]; 95% CI, 4.27–4.77). However, room for improvement was noted in some areas, such as the title, abstract reporting, a priori plan for heterogeneity assessment and how to handle heterogeneity in case it exists, and assessment of publication bias. In addition, reporting of agreement among reviewers, documentation of trials flow, and discussion of possible biases were addressed in the review process. Reviews published post the QUOROM statement had a significantly higher quality scores.

Conclusion

The systematic reviews published in the CNRG are generally of good quality with minor flaws. However, efforts should be made to improve the quality of reports. Readers must continue to assess the quality of published reports on an individual basis prior to implementing the recommendations.

Spinal cord stimulation in the treatment of refractory angina: systematic review and meta-analysis of randomised controlled trials

Background

The aim of this paper was undertake a systematic review and meta-analysis of the use of spinal cord stimulation (SCS) in the management of refractory angina.

Methods

We searched a number of electronic databases including Medline, Embase and Cochrane Library up to February 2008 to identify randomised controlled trials (RCTs) reporting exercise capacity, ischemic burden, functional class, quality of life, usage of anti-anginal medication, costs and adverse events including mortality. Results were reported both descriptively for each study and using random effects meta-analysis. Given the variety in outcomes reported, some outcome results were pooled as standardised mean differences (SMD) and reported in standard deviation units.

Results

Seven RCTs were identified in a total of 270 refractory angina patients. The outcomes of SCS were found to be similar when directly compared to coronary artery bypass grafting (CABG) and percutaneous myocardial laser revascularisation (PMR). Compared to a 'no stimulation' control, there was some evidence of improvement in all outcomes following SCS implantation with significant gains observed in pooled exercise capacity (SMD: 0.76, 0.07 to 1.46, p = 0.03) and health-related quality of life (SMD: 0.83, 95% CI: 0.32 to 1.34, p = 0.001). Trials were small and were judged to range considerably in their quality. The healthcare costs of SCS appeared to be lower than CABG at 2-years follow up.

Conclusion

SCS appears to be an effective and safe treatment option in the management of refractory angina patients and of similar efficacy and safety to PMR, a potential alternative treatment. Further high quality RCT and cost effectiveness evidence is needed before SCS can be accepted as a routine treatment for refractory angina.

Cochrane reviews used more rigorous methods than non-Cochrane reviews: survey of systematic reviews in physiotherapy

OBJECTIVE

To describe the quality and methods of systematic reviews of physiotherapy interventions, compare Cochrane and non-Cochrane reviews, and establish the interrater reliability of the Overview Quality Assessment Questionnaire (OQAQ) quality assessment tool.

STUDY DESIGN AND SETTING

A survey of 200 published systematic reviews was done. Two independent raters assessed the search strategy, assessment of trial quality, outcomes, pooling, conclusions, and overall quality (OQAQ). The study was carried out in the University research center.

RESULTS

In these reviews, the five most common databases searched were MEDLINE, EMBASE, Cochrane Library, CINAHL, and Cochrane Review Group Registers. The Cochrane allocation concealment system and Jadad Scale were most frequently used to assess trial quality. Cochrane reviews searched more databases and were more likely to assess trial quality, report dichotomous outcomes for individual trials, and conduct a meta-analysis than non-Cochrane reviews. Non-Cochrane reviews were more likely to conclude that there was a beneficial effect of treatment. Cochrane reviews were of higher quality than non-Cochrane reviews. There has been an increase in the quality of systematic reviews over time. The OQAQ has fair to good interrater reliability.

CONCLUSION

The quality of systematic reviews in physiotherapy is improving, and the use of Cochrane Collaboration procedures appears to improve the methods and quality.