Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals

Comparison of search strategies in systematic reviews of adverse effects to other systematic reviews

BACKGROUND

Research indicates that the methods used to identify data for systematic reviews of adverse effects may need to differ from other systematic reviews.

OBJECTIVES

To compare search methods in systematic reviews of adverse effects with other reviews.

METHODS

The search methodologies in 849 systematic reviews of adverse effects were compared with other reviews.

RESULTS

Poor reporting of search strategies is apparent in both systematic reviews of adverse effects and other types of systematic reviews. Systematic reviews of adverse effects are less likely to restrict their searches to MEDLINE or include only randomised controlled trials (RCTs). The use of other databases is largely dependent on the topic area and the year the review was conducted, with more databases searched in more recent reviews. Adverse effects search terms are used by 72% of reviews and despite recommendations only two reviews report using floating subheadings.

CONCLUSIONS

The poor reporting of search strategies in systematic reviews is universal, as is the dominance of searching MEDLINE. However, reviews of adverse effects are more likely to include a range of study designs (not just RCTs) and search beyond MEDLINE.

Strengths and limitations of evidence-based dermatology

The need for understanding and reflecting on evidence-based dermatology (EBD) has never been greater given the exponential growth of new external evidence to inform clinical practice. Like any other branch of medicine, dermatologists need to acquire new skills in constructing answerable questions, efficiently searching electronic bibliographic databases, and critically appraising different types of studies. Secondary summaries of evidence in the form of systematic reviews (SR), that is, reviews that are conducted in a systematic, unbiased and explicit manner, reside at the top of the evidence hierarchy, because they are less prone to bias than traditional expert reviews. In addition to providing summaries of the best external evidence, systematic reviews and randomized controlled trials (RCTs) are also powerful ways of identifying research gaps and ultimately setting the agenda of future clinical research in dermatology. But like any paradigm, EBD can have its limitations. Wrong application, misuse and overuse of EBD can have serious consequences. For example, mindless pooling together of data from dissimilar studies in a meta-analysis may render it a form of reductionism that does not make any sense. Similarly, even highly protocolised study designs such as SRs and RCTs are still susceptible to some degree of dishonesty and bias. Over-reliance on randomized controlled trials (RCT) may be inappropriate, as RCTs are not a good source for picking up rare but important adverse effects such as lupus syndrome with minocycline. A common criticism leveled against SRs is that these frequently conclude that there is lack of sufficient evidence to inform current clinical practice, but arguably, such a perception is grounded more on the interpretation of the SRs than anything else. The apparent absence of evidence should not paralyze the dermatologist to adopt a state of therapeutic nihilism. Poor primary data and an SR based on evidence that is not up-to-date are also limitations that can only improve with better primary studies and updated reviews such as those done by the Cochrane Collaboration. Most dermatologists are interested in integrating the best external evidence with the care of individual patients and have been practicing good EBD without realizing it.

Which is more generalizable, powerful and interpretable in meta-analyses, mean difference or standardized mean difference?

Background

To examine empirically whether the mean difference (MD) or the standardised mean difference (SMD) is more generalizable and statistically powerful in meta-analyses of continuous outcomes when the same unit is used.

Methods

From all the Cochrane Database (March 2013), we identified systematic reviews that combined 3 or more randomised controlled trials (RCT) using the same continuous outcome. Generalizability was assessed using the I-squared (I²) and the percentage agreement. The percentage agreement was calculated by comparing the MD or SMD of each RCT with the corresponding MD or SMD from the meta-analysis of all the other RCTs. The statistical power was estimated using Z-scores. Meta-analyses were conducted using both random-effects and fixed-effect models.

Results

1068 meta-analyses were included. The I² index was significantly smaller for the SMD than for the MD (P < 0.0001, sign test). For continuous outcomes, the current Cochrane reviews pooled some extremely heterogeneous results. When all these or less heterogeneous subsets of the reviews were examined, the SMD always showed a greater percentage agreement than the MD. When the I² index was less than 30%, the percentage agreement was 55.3% for MD and 59.8% for SMD in the random-effects model and 53.0% and 59.8%, respectively, in the fixed effect model (both P < 0.0001, sign test). Although the Z-scores were larger for MD than for SMD, there were no differences in the percentage of statistical significance between MD and SMD in either model.

Conclusions

The SMD was more generalizable than the MD. The MD had a greater statistical power than the SMD but did not result in material differences.

Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap

The current difficulties in keeping systematic reviews up to date leads to considerable inaccuracy, hampering the translation of knowledge into action. Incremental advances in conventional review updating are unlikely to lead to substantial improvements in review currency. A new approach is needed. We propose living systematic review as a contribution to evidence synthesis that combines currency with rigour to enhance the accuracy and utility of health evidence. Living systematic reviews are high quality, up-to-date online summaries of health research, updated as new research becomes available, and enabled by improved production efficiency and adherence to the norms of scholarly communication. Together with innovations in primary research reporting and the creation and use of evidence in health systems, living systematic review contributes to an emerging evidence ecosystem.

Is sustained virological response a marker of treatment efficacy in patients with chronic hepatitis C viral infection with no response or relapse to previous antiviral intervention?

BACKGROUND

Randomised clinical trials (RCTs) of antiviral interventions in patients with chronic hepatitis C virus (HCV) infection use sustained virological response (SVR) as the main outcome. There is sparse information on long-term mortality from RCTs.

METHODS

We created a decision tree model based on a Cochrane systematic review on interferon retreatment for patients who did not respond to initial therapy or who relapsed following SVR. Extrapolating data to 20 years, we modelled the outcome from three scenarios: (1) observed medium-term (5 year) annual mortality rates continue to the long term (20 years); (2) long-term annual mortality in retreatment responders falls to that of the general population while retreatment non-responders continue at the medium-term mortality; (3) long-term annual mortality in retreatment non-responders is the same as control group non-responders (i.e., the increased treatment-related medium mortality "wears off").

RESULTS

The mean differences in life expectancy over 20 years with interferon versus control in the first, second, and third scenarios were -0.34 years (95% confidence interval (CI) -0.71 to 0.03), -0.23 years (95% CI -0.69 to 0.24), and -0.01 (95% CI -0.3 to 0.27), respectively. The life expectancy was always lower in the interferon group than in the control group in scenario 1. In scenario 3, the interferon group had a longer life expectancy than the control group only when more than 7% in the interferon group achieved SVR.

CONCLUSIONS

SVR may be a good prognostic marker but does not seem to be a valid surrogate marker for assessing HCV treatment efficacy of interferon retreatment. The SVR threshold at which retreatment increases life expectancy may be different for different drugs depending upon the adverse event profile and treatment efficacy. This has to be determined for each drug by RCTs and appropriate modelling before SVR can be accepted as a surrogate marker.

Magnesium sulfate treatment improves outcome in patients with subarachnoid hemorrhage: a meta-analysis study

OBJECTIVE

To elucidate the role of magnesium sulfate in patients with subarachnoid hemorrhagic (SAH) brain injury

METHOD

Studies for the meta-analysis were identified from PubMed (1966 to 2009), Embase (1980 to 2009), and two Chinese journals (1989 to 2009). Paper selection was based on randomized controlled trials comparing magnesium sulfate to placebo treatment in patients with SAH. Two independent review authors extracted the data and assessed trial quality. Meta-analysis was performed using the Cochrane Review Manger software.

RESULTS

Five trials involving 482 patients were included in the review. Magnesium sulfate reduced the risk of poor outcome and reduced the occurrence of delayed cerebral ischemia. In the treatment groups, relative risk for poor outcome was 0.73 (CI 0.57-0.93) and 0.66 (CI 0.47-0.92) for delayed cerebral ischemia. Case fatality assessment at three to six months did not show statistically significant data (RR 0.88; CI 0.61-1.29).

CONCLUSION

Magnesium sulfate appears to be an effective treatment option in the management of SAH. Further clinical trials are needed before magnesium sulfate can become a routine treatment for SAH.

Incorporation of assessments of risk of bias of primary studies in systematic reviews of randomised trials: a cross-sectional study

OBJECTIVE

We examined how assessments of risk of bias of primary studies are carried out and incorporated into the statistical analysis and overall findings of a systematic review.

DESIGN

A cross-sectional review.

SAMPLE

We assessed 200 systematic reviews of randomised trials published between January and March 2012; Cochrane (n=100), non-Cochrane (Database of Reviews of Effects) (n=100).

MAIN OUTCOMES

Our primary outcome was a descriptive analysis of how assessments of risk of bias are carried out, the methods used, and the extent to which such assessments were incorporated into the statistical analysis and overall review findings.

RESULTS

While Cochrane reviews routinely reported the method of risk of bias assessment and presented their results either in text or table format, 20% of non-Cochrane reviews failed to report the method used and 39% did not present the assessment results. Where it was possible to evaluate the individual results of the risk of bias assessment (n=154), 75% (n=116/154) of reviews had ≥1 trial at high risk of bias; the median proportion of trials per review at high risk of bias was 50% (IQR 31% to 89%). Despite this, only 56% (n=65/116) incorporated the risk of bias assessment into the interpretation of the results in the abstract and 41% (n=47/116) (49%; n=40/81 Cochrane and 20%; n=7/35 non-Cochrane) incorporated the risk of bias assessment into the interpretation of the conclusions. Of the 83% (n=166/200) systematic reviews which included a meta-analysis, only 11% (n=19/166) incorporated the risk of bias assessment into the statistical analysis.

CONCLUSIONS

Cochrane reviews were more likely than non-Cochrane reviews to report how risk of bias assessments of primary studies were carried out; however, both frequently failed to take such assessments into account in the statistical analysis and conclusions of the systematic review.

Usefulness of Cochrane Skin Group reviews for clinical practice

BACKGROUND AND OBJECTIVE

Systematic reviews are one of the most important sources of information for evidence-based medicine. However, there is a general impression that these reviews rarely report results that provide sufficient evidence to change clinical practice. The aim of this study was to determine the percentage of Cochrane Skin Group reviews reporting results with the potential to guide clinical decision-making.

MATERIAL AND METHODS

We performed a bibliometric analysis of all the systematic reviews published by the Cochrane Skin Group up to 16 August, 2012. We retrieved 55 reviews, which were analyzed and graded independently by 2 investigators into 3 categories: 0 (insufficient evidence to support or reject the use of an intervention), 1 (insufficient evidence to support or reject the use of an intervention but sufficient evidence to support recommendations or suggestions), and 2 (sufficient evidence to support or reject the use of an intervention).

RESULTS

Our analysis showed that 25.5% (14/55) of the studies did not provide sufficient evidence to support or reject the use of the interventions studied, 45.5% (25/25) provided sufficient but not strong evidence to support recommendations or suggestions, and 29.1% (16/55) provided strong evidence to support or reject the use of 1 or more of the interventions studied.

CONCLUSIONS

Most of the systematic reviews published by the Cochrane Skin Group provide useful information to improve clinical practice. Clinicians should read these reviews and reconsider their current practice.

Randomized controlled trials in endourology: a quality assessment

PURPOSE

To analyze the quality of studies reporting randomized clinical trials (RCTs) in the field of endourology.

MATERIALS AND METHODS

RCTs published in the Journal of Endourology from 1993 until 2011 were identified. The Jadad scale, van Tulder scale, and Cochrane Collaboration Risk of Bias Tool (CCRBT) were used to assess the quality of the studies. The review period was divided into early (1993-1999), mid (2000-2005), and late (2006-2011) terms. Studies were categorized by country of origin, subject matter, single- vs multicenter setting, Institutional Review Board (IRB) approval and funding support, and blinding vs nonblinding.

RESULTS

In total, 3339 articles had been published during the defined review period, of which 165 articles were reporting a RCT. There was a significant increase in the number of RCTs published over time, with 18 (2.81%), 43 (4.88%), and 104 (5.72%) studies identified in the early, mid, and late term, respectively (P=0.009). Nevertheless, there was no difference in terms of quality of reporting, as assessed with the Jadad scale, van Tulder scale, or CCRBT, between the three study terms. On the other hand, significant differences were found in both the number of high qualitative RCTs that used blinding methodology and those that had IRB review, when comparing the early, mid, and late terms.

CONCLUSION

There has been a growing number of Journal of Endourology publications reporting on RTC over the last two decades. The quality of reporting for these studies remains suboptimal, however. Researchers should focus on a more appropriate description of key features of any given RCT, such as randomization and allocation methods, as well as disclosure of IRB review and financial support.

Heterogeneity in meta-analyses of treatment of acute postoperative pain: a review

BACKGROUND

Heterogeneity and its causes must be assessed using meta-analyses (meta-analysis). Especially in meta-analysis dealing with treatment of acute postoperative pain, the type of surgery is a source of heterogeneity. We aimed to assess whether the type of surgery is considered a source of heterogeneity in meta-analysis and how it is taken into account in meta-analysis evaluating the efficacy of treatment of acute postoperative pain. We further compared meta-analysis that pooled trials of surgeries with highly heterogeneous postoperative pain levels, the heterogeneous group, with meta-analysis that pooled trials involving surgeries with homogeneous pain levels, the homogenous group.

METHODS

The meta-analysis reports available in Issue 3, 2011 of the electronic database of the Cochrane library and pooling results of randomized or quasi-randomized controlled trials that assessed the efficacy of treatment of acute postoperative pain alone were considered. A survey of experts established a rating of the postoperative pain levels for the type of surgery. For each meta-analysis, the different pain level ratings associated with the trials included in the meta-analysis were considered and the standard deviation (sd) of these ratings calculated. From the distribution of sd values, we defined the heterogeneous and homogeneous groups.

RESULTS

Sixty-one meta-analyses were included; all assessed heterogeneity. Twenty-six meta-analyses considered the type of surgery as a subgroup (50% vs 38% in the homogeneous group vs heterogeneous group). Forty-four reports discussed the type of surgery as a source of clinical heterogeneity (85% vs 62% for the homogeneous vs heterogeneous group). Twenty-nine meta-analyses compared 'postoperative pain from dental surgery' to 'other type of surgery'.

CONCLUSIONS

Meta-analyses evaluating treatment of postoperative pain should explore clinical heterogeneity associated with the type of surgery for better implications for practice.

Search strategies in systematic reviews in periodontology and implant dentistry

OBJECTIVE

To perform an overview of literature search strategies in systematic reviews (SRs) published in periodontology and implant dentistry.

MATERIALS AND METHODS

Two electronic databases (PubMed and Cochrane Database of SRs) were searched, independently and in duplicate, for SRs with meta-analyses on interventions, with the last search performed on 11 November 2012. Manual searches of the reference lists of included SRs and 10 specialty dental journals were conducted. Methodological issues of the search strategies of included SRs were assessed with Cochrane collaboration guidelines and AMSTAR recommendations. The search strategies employed in Cochrane and paper-based SRs were compared.

RESULTS

A total of 146 SRs with meta-analyses were included, including 19 Cochrane and 127 paper-based SRs. Some issues, such as "the use of keywords," were reported in most of the SRs (86%). Other issues, such as "search of grey literature" and "language restriction," were not fully reported (34% and 50% respectively). The quality of search strategy reporting in Cochrane SRs was better than that of paper-based SRs for seven of the eight criteria assessed.

CONCLUSION

There is room for improving the quality of reporting of search strategies in SRs in periodontology and implant dentistry, particularly in SRs published in paper-based journals.

Assessment of the quality of reporting in abstracts of systematic reviews with meta-analyses in periodontology and implant dentistry

BACKGROUND

Proper scientific reporting is necessary to ensure the correct interpretation of study results by readers. The main objective of this study was to assess the quality of reporting in abstracts of systematic reviews (SRs) with meta-analyses in periodontology and implant dentistry. Differences in reporting of abstracts in Cochrane and paper-based reviews were also assessed.

METHODS

The PubMed electronic database and the Cochrane database for SRs were searched on November 11, 2012, independently and in duplicate, for SRs with meta-analyses related to interventions in periodontology and implant dentistry. Assessment of the quality of reporting was performed independently and in duplicate, taking into account items related to the effect direction, numerical estimates of effect size, measures of precision, probability and consistency.

RESULTS

We initially screened 433 papers and included 146 (127 paper-based and 19 Cochrane reviews, respectively). The direction of evidence was reported in two-thirds of the abstracts while strength of evidence and measure of precision (i.e., confidence interval) were reported in less than half the selected abstracts. Measures of consistency such as I(2) statistics were reported in only 5% of the selected sample of abstracts. Cochrane abstracts reported the limitations of evidence and precision better than paper-based ones. Two items ("meta-analysis" in title and abstract, respectively), were nevertheless better reported in paper-based abstracts.

CONCLUSION

Abstracts of SRs with meta-analyses in periodontology and implant dentistry currently have no uniform standard of reporting, which may hinder readers' understanding of study outcomes.

Acute traumatic brain injury: is current management evidence based? An empirical analysis of systematic reviews

Traumatic brain injury (TBI) is a major health and socioeconomic problem worldwide with a high rate of death and long-term disability. Previous studies have summarized evidence from large-scale randomized trials, finding no intervention showing convincing efficacy for acute TBI management. The present empirical study set out to assess another crucial component of evidence base-systematic review, which contributes a lot to evidence-based health care, in terms of clinical issues, methodological aspects, and implication for practice and research. A total of 44 systematic reviews pertaining to therapeutic interventions for acute TBI were identified through electronic database searching, clinical guideline retrieval, and expert consultation, of which 21 were published in Cochrane Library and 23 in peer-reviewed journals. Their methodological quality was generally satisfactory, with the median Overview Quality Assessment Questionnaire score of 5.5 (interquartile range 2-7). Cochrane reviews are of better quality than regular journal reviews. Twenty-nine high-quality reviews provided no conclusive evidence for the investigated 22 interventions except for an adverse effect of corticosteroids. Less than one-third of the component trials were reported with adequate allocation concealment. Additionally other methodological flaws in design-for example, ignoring heterogeneity among the TBI population-also contributed to the failure of past clinical research. Based on the above findings, evidence from both systematic reviews and clinical trials does not fully support current management of acute TBI. Translating from laboratory success to clinical effect remains an unique challenge. Accordingly it may be the time to rethink the way in future practice and clinical research in TBI.

Citation bias favoring statistically significant studies was present in medical research

OBJECTIVE

Statistically significant studies may be cited more than negative studies on the same topic. We aimed to assess here whether such citation bias is present across the medical literature.

STUDY DESIGN AND SETTING

We conducted a cohort study of the association between statistical significance and citations. We selected all therapeutic intervention studies included in meta-analyses published between January and March 2010 in the Cochrane database, and retrieved citation counts of all individual studies using ISI Web of Knowledge. The association between the statistical significance of each study and the number of citations it received between 2008 and 2010 was assessed in mixed Poisson models.

RESULTS

We identified 89 research questions addressed in 458 eligible articles. Significant studies were cited twice as often as nonsignificant studies (multiplicative effect of significance: 2.14, 95% confidence interval: 1.38-3.33). This association was partly because of the higher impact factor of journals where significant studies are published (adjusted multiplicative effect of significance: 1.14, 95% confidence interval: 0.87-1.51).

CONCLUSION

A citation bias favoring significant results occurs in medical research. As a consequence, treatments may seem more effective to the readers of medical literature than they really are.

Conclusiveness of the Cochrane Reviews in Pediatric-Gastroenterology: a systematic analysis

AIM

To assess the conclusiveness of the Cochrane Reviews (CRs) in the field of Pediatric-Gastroenterology. We tested the hypotheses that (a) the majority of CRs are inconclusive, (b) the majority of CRs recognize the need for further studies, (c) the ability to reach a conclusion is dependent on both the number of studies and the number of patients. We also studied whether the conclusiveness of CRs changed over time.

METHODS

We selected all CRs in the field of Pediatric-Gastroenterology available in the Cochrane library. Each CR was analyzed for the number of randomized clinical trials (RCTs) found, the number of RCTs included for analysis, the number of patients enrolled, the stated need for further studies, and the conclusiveness of the CR.

RESULTS

Sixty-eight out of 85 CRs (80%) were conclusive. The percentage of articles included in conclusive studies was significantly higher than that in inconclusive ones (P<0.00001). Thus, the average number of RCTs retained in the analyses was significantly higher in conclusive CRs. The total number of patients in the RCTs retained for analysis was significantly higher in conclusive CRs (P<0.0001). The majority of inconclusive CRs (82.4%) recognized the need for further studies compared with 44.1% in conclusive ones (P=0.002). The percentage of conclusive CRs was not affected by the year of publication.

CONCLUSION

In the field of Pediatric-Gastroenterology, CRs appear to be a potent clinical tool that allows, in 80% of the cases, to reach a valid (from a clinical standpoint) conclusion, while emphasizing the weaknesses of available, published evidence and delineating future avenues of research.

Methodological quality and descriptive characteristics of prosthodontic-related systematic reviews

Ideally, healthcare systematic reviews (SRs) should be beneficial to practicing professionals in making evidence-based clinical decisions. However, the conclusions drawn from SRs are directly related to the quality of the SR and of the included studies. The aim was to investigate the methodological quality and key descriptive characteristics of SRs published in prosthodontics. Methodological quality was analysed using the Assessment of Multiple Reviews (AMSTAR) tool. Several electronic resources (MEDLINE, EMBASE, Web of Science and American Dental Association's Evidence-based Dentistry website) were searched. In total 106 SRs were located. Key descriptive characteristics and methodological quality features were gathered and assessed, and descriptive and inferential statistical testing performed. Most SRs in this sample originated from the European continent followed by North America. Two to five authors conducted most SRs; the majority was affiliated with academic institutions and had prior experience publishing SRs. The majority of SRs were published in specialty dentistry journals, with implant or implant-related topics, the primary topics of interest for most. According to AMSTAR, most quality aspects were adequately fulfilled by less than half of the reviews. Publication bias and grey literature searches were the most poorly adhered components. Overall, the methodological quality of the prosthodontic-related systematic was deemed limited. Future recommendations would include authors to have prior training in conducting SRs and for journals to include a universal checklist that should be adhered to address all key characteristics of an unbiased SR process.

Exercise therapy for bone and muscle health: an overview of systematic reviews

BACKGROUND

Musculoskeletal conditions (MSCs) are widely prevalent in present-day society, with resultant high healthcare costs and substantial negative effects on patient health and quality of life. The main aim of this overview was to synthesize evidence from systematic reviews on the effects of exercise therapy (ET) on pain and physical function for patients with MSCs. In addition, the evidence for the effect of ET on disease pathogenesis, and whether particular components of exercise programs are associated with the size of the treatment effects, was also explored.

METHODS

We included four common conditions: fibromyalgia (FM), low back pain (LBP), neck pain (NP), and shoulder pain (SP), and four specific musculoskeletal diseases: osteoarthritis (OA), rheumatoid arthritis (RA), ankylosing spondylitis (AS), and osteoporosis (OP). We first included Cochrane reviews with the most recent update being January 2007 or later, and then searched for non-Cochrane reviews published after this date. Pain and physical functioning were selected as primary outcomes.

RESULTS

We identified 9 reviews, comprising a total of 224 trials and 24,059 patients. In addition, one review addressing the effect of exercise on pathogenesis was included. Overall, we found solid evidence supporting ET in the management of MSCs, but there were substantial differences in the level of research evidence between the included diagnostic groups. The standardized mean differences for knee OA, LBP, FM, and SP varied between 0.30 and 0.65 and were significantly in favor of exercise for both pain and function. For NP, hip OA, RA, and AS, the effect estimates were generally smaller and not always significant. There was little or no evidence that ET can influence disease pathogenesis. The only exception was for osteoporosis, where there was evidence that ET increases bone mineral density in postmenopausal women, but no significant effects were found for clinically relevant outcomes (fractures). For LBP and knee OA, there was evidence suggesting that the treatment effect increases with the number of exercise sessions.

CONCLUSIONS

There is empirical evidence that ET has beneficial clinical effects for most MSCs. Except for osteoporosis, there seems to be a gap in the understanding of the ways in which ET influences disease mechanisms.

A comparison of statistical methods for identifying out-of-date systematic reviews

BACKGROUND

Systematic reviews (SRs) can provide accurate and reliable evidence, typically about the effectiveness of health interventions. Evidence is dynamic, and if SRs are out-of-date this information may not be useful; it may even be harmful. This study aimed to compare five statistical methods to identify out-of-date SRs.

METHODS

A retrospective cohort of SRs registered in the Cochrane Pregnancy and Childbirth Group (CPCG), published between 2008 and 2010, were considered for inclusion. For each eligible CPCG review, data were extracted and "3-years previous" meta-analyses were assessed for the need to update, given the data from the most recent 3 years. Each of the five statistical methods was used, with random effects analyses throughout the study.

RESULTS

Eighty reviews were included in this study; most were in the area of induction of labour. The numbers of reviews identified as being out-of-date using the Ottawa, recursive cumulative meta-analysis (CMA), and Barrowman methods were 34, 7, and 7 respectively. No reviews were identified as being out-of-date using the simulation-based power method, or the CMA for sufficiency and stability method. The overall agreement among the three discriminating statistical methods was slight (Kappa = 0.14; 95% CI 0.05 to 0.23). The recursive cumulative meta-analysis, Ottawa, and Barrowman methods were practical according to the study criteria.

CONCLUSION

Our study shows that three practical statistical methods could be applied to examine the need to update SRs.

A pilot study using machine learning and domain knowledge to facilitate comparative effectiveness review updating

BACKGROUND

Comparative effectiveness and systematic reviews require frequent and time-consuming updating.

RESULTS

of earlier screening should be useful in reducing the effort needed to screen relevant articles.

METHODS

We collected 16,707 PubMed citation classification decisions from 2 comparative effectiveness reviews: interventions to prevent fractures in low bone density (LBD) and off-label uses of atypical antipsychotic drugs (AAP). We used previously written search strategies to guide extraction of a limited number of explanatory variables pertaining to the intervention, outcome, and

STUDY DESIGN

We empirically derived statistical models (based on a sparse generalized linear model with convex penalties [GLMnet] and a gradient boosting machine [GBM]) that predicted article relevance. We evaluated model sensitivity, positive predictive value (PPV), and screening workload reductions using 11,003 PubMed citations retrieved for the LBD and AAP updates. Results. GLMnet-based models performed slightly better than GBM-based models. When attempting to maximize sensitivity for all relevant articles, GLMnet-based models achieved high sensitivities (0.99 and 1.0 for AAP and LBD, respectively) while reducing projected screening by 55.4% and 63.2%. The GLMnet-based model yielded sensitivities of 0.921 and 0.905 and PPVs of 0.185 and 0.102 when predicting articles relevant to the AAP and LBD efficacy/effectiveness analyses, respectively (using a threshold of P ≥ 0.02). GLMnet performed better when identifying adverse effect relevant articles for the AAP review (sensitivity = 0.981) than for the LBD review (0.685). The system currently requires MEDLINE-indexed articles.

CONCLUSIONS

We evaluated statistical classifiers that used previous classification decisions and explanatory variables derived from MEDLINE indexing terms to predict inclusion decisions. This pilot system reduced workload associated with screening 2 simulated comparative effectiveness review updates by more than 50% with minimal loss of relevant articles.

Quality and transparency of overviews of systematic reviews

OBJECTIVE

To evaluate the reporting and methodological quality of overviews of systematic reviews.

METHOD

We developed an 18-item assessment tool for overviews of systematic reviews. We then performed a systematic search for such overviews using the terms ('overview' AND ('meta analys*' OR 'systematic review*')) OR 'umbrella review' in the title. We only included those overviews that were limited to systematic reviews or meta-analyses. Their methodological and reporting quality were assessed by two independent reviewers using the checklist, and differences were resolved by a third reviewer. Data analyses was conducted by SPSS version 15.0 for Windows.

RESULTS

We identified 41 overviews of systematic reviews whose mean total reporting score was 10.78 (SD 3.84) and methodological score 3.05 (SD 2.09). Some important items were not adequately reported: only 69% reported defined eligibility criteria, 76% reported search strategy, 49% reported the process of review selection, 44% reported the data collection process, 5% reported evaluating the reporting quality, 46% reported evaluating methodological quality, and 20% reported assessing the evidence level for each outcome.

CONCLUSION

The reporting and methodological quality of overviews of systematic reviews was very poor, and there is still much room for improvement. A checklist for overviews of systematic reviews should be developed and used.

Effects of drop-out on efficacy estimates in five Cochrane reviews of popular antipsychotics for schizophrenia

OBJECTIVE

Our aim was to find out how Cochrane reviews of five popular or frequently prescribed second-generation antipsychotics in the UK (olanzapine, risperidone, quetiapine, amisulpride and aripiprazole) approached the problem of high drop-out in placebo-controlled trials.

METHOD

We examined the following: (i) whether reviews included data from studies with a level of drop-out exceeding their stated exclusion criterion; (ii) the level of missing data each efficacy outcome in each review relied upon; and (iii) impact of excluding studies with high drop-out.

RESULTS

All reviews included data they stated they would exclude because of unacceptable levels of attrition, four (risperidone, olanzapine, amisulpride, aripiprazole) without clear acknowledgement or justification. Several reviews also excluded data from a number of relatively low-attrition studies because of missing standard deviations.

CONCLUSION

Cochrane reviews of five popular antipsychotics for schizophrenia misrepresented the available evidence on their efficacy. The impact of including high-attrition studies was difficult to quantify because of the exclusion of relevant low-attrition studies. Further analysis of the efficacy of these drugs in studies with acceptable rates of attrition is required. To reduce the problem of high attrition, trialists should gather follow-up data from people who leave the double-blind process early.

Sensitivity and predictive value of 15 PubMed search strategies to answer clinical questions rated against full systematic reviews

Background

Clinicians perform searches in PubMed daily, but retrieving relevant studies is challenging due to the rapid expansion of medical knowledge. Little is known about the performance of search strategies when they are applied to answer specific clinical questions.

Objective

To compare the performance of 15 PubMed search strategies in retrieving relevant clinical trials on therapeutic interventions.

Methods

We used Cochrane systematic reviews to identify relevant trials for 30 clinical questions. Search terms were extracted from the abstract using a predefined procedure based on the population, interventions, comparison, outcomes (PICO) framework and combined into queries. We tested 15 search strategies that varied in their query (PIC or PICO), use of PubMed’s Clinical Queries therapeutic filters (broad or narrow), search limits, and PubMed links to related articles. We assessed sensitivity (recall) and positive predictive value (precision) of each strategy on the first 2 PubMed pages (40 articles) and on the complete search output.

Results

The performance of the search strategies varied widely according to the clinical question. Unfiltered searches and those using the broad filter of Clinical Queries produced large outputs and retrieved few relevant articles within the first 2 pages, resulting in a median sensitivity of only 10%–25%. In contrast, all searches using the narrow filter performed significantly better, with a median sensitivity of about 50% (all P < .001 compared with unfiltered queries) and positive predictive values of 20%–30% (P < .001 compared with unfiltered queries). This benefit was consistent for most clinical questions. Searches based on related articles retrieved about a third of the relevant studies.

Conclusions

The Clinical Queries narrow filter, along with well-formulated queries based on the PICO framework, provided the greatest aid in retrieving relevant clinical trials within the 2 first PubMed pages. These results can help clinicians apply effective strategies to answer their questions at the point of care.

How close is evidence to truth in evidence-based treatment of mental disorders?

Given the importance of the term 'evidence' in evidence-based medicine (EBM), the meaning of this term is evaluated, going back to the philosophical tradition and current meaning of the terms 'evidence' and 'truth'. Based on this, current problems in the definition of evidence and in the grading of evidence in EBM are described, taking examples from the field of psychiatry and especially pharmacopsychiatry. These problems underline that the use of the term evidence in EBM is inconsistent and inconclusive. This should be fairly stated in all EBM-related publications, especially in EBM-based guidelines, to avoid severe misunderstandings in and outside the field of psychiatry. Although EBM might have increased empirically driven rational decision-making in psychiatry/medicine, the current limitations should be carefully considered.

Systematic reviews synthesized evidence without consistent quality assessment of primary studies examining epidemiology of chronic diseases

OBJECTIVE

To evaluate how systematic reviews assess the quality of primary studies of incidence, prevalence, or risk factors for chronic diseases.

STUDY DESIGN AND SETTING

We searched several databases, identified 145 systematic reviews, and evaluated methods of quality assessment and quantitative synthesis of evidence by external or internal validity or overall quality of primary studies.

RESULTS

Of 145 reviews, 54 (37%) reported a planned quality assessment of primary studies with checklists or scales and 26 (18%) reported evaluation of some selected quality criteria. Thirty-nine percent of reviews judged appropriateness of sampling and proper controls for confounding factors in primary studies. Twelve percent synthesized evidence by overall quality, 17% by design, 42% by criteria of internal validity, and 24% by external validity of primary studies. Masking of quality assessment was conducted on 2.1% of reviews and 4% tested interobserver agreement for quality assessment.

CONCLUSION

Evaluation of internal and external validity of primary studies is uncommon in systematic reviews of studies of incidence, prevalence, or risk factors for chronic diseases. Inconsistent quality assessment practices reflect the absence of uniformly accepted standards and tools to examine the quality of observational nontherapeutic studies.

The quality of research synthesis in surgery: the case of laparoscopic surgery for colorectal cancer

BACKGROUND

Several systematic reviews and meta-analyses populate the literature on the effectiveness of laparoscopic surgery for colorectal cancer. The utility of this body of work is unclear. The objective of this study was to synthesize all such systematic reviews in terms of clinical effectiveness, to appraise their quality, and to determine whether areas of duplication exist across reviews.

METHODS

Systematic reviews comparing laparoscopic and open surgery for colorectal cancer were identified using a comprehensive search protocol (1991 to 2008). The primary outcome was overall survival. The methodological quality of reviews was appraised using the Assessment of Multiple Systematic Reviews (AMSTAR) instrument. Abstraction and quality appraisal was carried out by two independent reviewers. Reviews were synthesized, and outcomes were compared qualitatively. A citation analysis was carried out using simple matrices to assess the comprehensiveness of each review.

RESULTS

In total, 27 reviews were included; 13 reviews included only randomized controlled trials. Rectal cancer was addressed exclusively by four reviews. There was significant overlap between review purposes, populations and, outcomes. The mean AMSTAR score (out of 11) was 5.8 (95% CI: 4.6 to 7.0). Overall survival was evaluated by ten reviews, none of which found a significant difference. Three reviews provided a selective meta-analysis of time-to-event data. Previously published systematic reviews were poorly and highly selectively referenced (mean citation ratio 0.16, 95% CI: 0.093 to 0.22). Previously published trials were not comprehensively identified and cited (mean citation ratio 0.56, 95% CI: 0.46 to 0.65).

CONCLUSIONS

Numerous overlapping systematic reviews of laparoscopic and open surgery for colorectal cancer exist in the literature. Despite variable methods and quality, survival outcomes are congruent across reviews. A duplication of research efforts appears to exist in the literature. Further systematic reviews or meta-analyses are unlikely to be justified without specifying a significantly different research objective. This works lends support to the registration and updating of systematic reviews.

Reviewing methodologically disparate data: a practical guide for the patient safety research field

This article addresses key questions frequently asked by researchers conducting systematic reviews in patient safety. This discipline is relatively young, and asks complex questions about complex aspects of health care delivery and experience, therefore its studies are typically methodologically heterogeneous, non-randomized and complex; but content rich and highly relevant to practice. Systematic reviews are increasingly necessary to drive forward practice and research in this area, but the data do not always lend themselves to 'standard' review methodologies. This accessible 'how-to' article demonstrates that data diversity need not preclude high-quality systematic reviews. It draws together information from published guidelines and experience within our multidisciplinary patient safety research group to provide entry-level advice for the clinician-researcher new to systematic reviewing, to non-biomedical research data or to both. It offers entry-level advice, illustrated with detailed practical examples, on defining a research question, creating a comprehensive search strategy, selecting articles for inclusion, assessing study quality, extracting data, synthesizing data and evaluating the impact of your review. The article concludes with a comment on the vital role of robust systematic reviews in the continuing advancement of the patient safety field.

Spinal cord stimulation for complex regional pain syndrome: A systematic review of the clinical and cost-effectiveness literature and assessment of prognostic factors

OBJECTIVE

To review the clinical and cost-effectiveness of spinal cord stimulation (SCS) in the management of patients with complex regional pain syndrome (CRPS) and identify the potential predictors of SCS outcome.

DESIGN

Systematic review of the literature and meta-regression.

METHODS

Electronic databases were searched for controlled and uncontrolled studies and economic evaluations relating to the use of SCS in patients with either CRPS type I or II.

RESULTS

One randomised controlled trial, 25 case series and one cost-effectiveness study were included. In the randomised controlled trial in type I CRPS patients, SCS therapy lead to a reduction in pain intensity at 24 months of follow-up (mean change in VAS score -2.0), whereas pain was unchanged in the control group (mean change in VAS score 0.0) (p<0.001). In the case series studies, 67% (95% CI 51%, 84%) of type I and type II CRPS patients implanted with SCS reported pain relief of at least 50% over a median follow-up period of 33 months. No statistically significant predictors of pain relief with SCS were observed in multivariate meta-regression analysis across studies. An economic analysis based on the randomised controlled trial showed a lifetime cost saving of approximately 58,470 (60,800 US dollars) with SCS plus physical therapy compared with physical therapy alone. The mean cost per quality-adjusted life-year at 12-month follow-up was 22,580 (23,480 US dollars).

CONCLUSIONS

SCS appears to be an effective therapy in the management of patients with CRPS type I (Level A evidence) and type II (Level D evidence). Moreover, there is evidence to demonstrate that SCS is a cost-effective treatment for CRPS type I.

Cochrane systematic reviews of Chinese herbal medicines: an overview

OBJECTIVES

our study had two objectives: a) to systematically identify all existing systematic reviews of Chinese herbal medicines (CHM) published in Cochrane Library; b) to assess the methodological quality of included reviews.

METHODOLOGY/PRINCIPAL FINDINGS

We performed a systematic search of the Cochrane Database of Systematic Reviews (CDSR, Issue 5, 2010) to identify all reviews of CHM. A total of fifty-eight reviews were eligible for our study. Twenty-one of the included reviews had at least one Traditional Chinese Medicine (TCM) practitioner as its co-author. 7 reviews didn't include any primary study, the remaining reviews (n = 51) included a median of 9 studies and 936 participants. 50% of reviews were last assessed as up-to-date prior to 2008. The questions addressed by 39 reviews were broad in scope, in which 9 reviews combined studies with different herbal medicines. For OQAQ, the mean of overall quality score (item 10) was 5.05 (95% CI; 4.58-5.52). All reviews assessed the methodological quality of primary studies, 16% of included primary studies used adequate sequence generation and 7% used adequate allocation concealment. Of the 51 nonempty reviews, 23 reviews were reported as being inconclusive, while 27 concluded that there might be benefit of CHM, which was limited by the poor quality or inadequate quantity of included studies. 58 reviews reported searching a median of seven electronic databases, while 10 reviews did not search any Chinese database.

CONCLUSIONS

Now CDSR has included large numbers of CHM reviews, our study identified some areas which could be improved, such as almost half of included reviews did not have the participation of TCM practitioners and were not up-to-date according to Cochrane criteria, some reviews pooled the results of different herbal medicines and ignored the searching of Chinese databases.

The effect of platelet concentrates on graft maturation and graft-bone interface healing in anterior cruciate ligament reconstruction in human patients: a systematic review of controlled trials

PURPOSE

To systematically review the current evidence for the effects of platelet concentrates on (1) graft maturation and (2) graft-bone interface healing in anterior cruciate ligament (ACL) reconstruction in human, controlled trials and for ensuing differences in clinical outcomes.

METHODS

A systematic search of PubMed, CINAHL (Cumulative Index to Nursing and Allied Health Literature), Embase, Cochrane Central Register of Controlled Trials, and Cochrane Database of Systematic Reviews was performed for controlled trials of human ACL reconstruction with and without platelet concentrates. Data validity was assessed, and data were collected on graft maturation, graft-bone interface healing, and clinical outcome.

RESULTS

Eight studies met the inclusion criteria. Seven studies reported on graft maturation with significantly better outcomes in the platelet groups in 4, and there were large differences in means in an additional 2 studies. Five studies reported on tunnel healing, and 4 found no difference between groups. Three studies assessed clinical outcome but found no differences, regardless of whether they had shown a beneficial effect (1 of 3) or no effect (2 of 3) of platelets on graft and tunnel healing.

CONCLUSIONS

The current best evidence suggests that the addition of platelet concentrates to ACL reconstruction may have a beneficial effect on graft maturation and could improve it by 20% to 30% on average, but with substantial variability. The most likely mode of action is that treatment with platelets accelerates graft repopulation and remodeling, and this interpretation is supported by the existing data and is biologically plausible. However, the current evidence also shows only a very limited influence of platelet concentrates on graft-bone interface healing and no significant difference in clinical outcomes.

LEVEL OF EVIDENCE

Level III, systematic review of Level I, II, and III studies.

Using the needs of WHO to prioritise Cochrane reviews: The case of antipsychotic drugs

Background

This study aimed to investigate existing trialling activity relating to three antipsychotic drugs from the WHO List of Essential Medicines (chlorpromazine, fluphenazine decanoate, haloperidol), link existing trials to existing Cochrane reviews, identify gaps in reviewing activity on accessible treatments for people with schizophrenia.

Methods

We used the Cochrane Schizophrenia Group's register searching for all studies comparing the three antipsychotic drugs with each other and with all other pharmacological interventions listed on the Essential Medicines List (with the addition of 'placebo or no drug'). For each we also considered studies that focussed on administration, dose, withdrawal and use of that drug in specific circumstances administration. Data were then extracted on a number of studies, number of participants within those studies, and as to whether a maintained review already exists. Finally, every effort was made to consider as to whether there were possibilities for missing comparisons that no one had ever investigated.

Results

There has been considerable research activity involving the three 'essential' antipsychotics and also comparing those three drugs to others on the 'essential' list. We found 490 studies with 77957 participants for haloperidol, 316 studies with 29179 participants for chlorpromazine and 33 studies with 4503 participants for fluphenazine decanoate. Reviewing activity has also been considerable in this area but there are notable omissions which would necessitate new reviews to comprehensively cover the area.

Conclusions

We have used the 'sample frame' of the WHO Essential drug list as a starting point. WHO prioritises for us those drugs that have universal accessibility but they may not be the compounds that are first choice if others are available. It is encouraging to see how many maintained reviews already exist to service those undertaking WHO guidelines. The needs of those guiding care can be taken as a means of prioritising research. For largest global impact WHO Essential Medicine list provides clear direction. By using this technique workload can be anticipated, prioritising can take place for new reviews and updates.

The evidence base for interventions delivered to children in primary care: an overview of cochrane systematic reviews

BACKGROUND

As a first step in developing a framework to evaluate and improve the quality of care of children in primary care there is a need to identify the evidence base underpinning interventions relevant to child health. Our objective was to identify all Cochrane systematic reviews relevant to the management of childhood conditions in primary care and to assess the extent to which Cochrane reviews reflect the burden of childhood illness presenting in primary care.

METHODOLOGY/PRINCIPAL FINDINGS

We used the Cochrane Child Health Field register of child-relevant systematic reviews to complete an overview of Cochrane reviews related to the management of children in primary care. We compared the proportion of systematic reviews with the proportion of consultations in Australia, US, Dutch and UK general practice in children. We identified 396 relevant systematic reviews; 358 included primary studies on children while 251 undertook a meta-analysis. Most reviews (n = 218, 55%) focused on chronic conditions and over half (n = 216, 57%) evaluated drug interventions. Since 2000, the percentage of pediatric primary care relevant reviews only increased by 2% (7% to 9%) compared to 18% (10% to 28%) in all child relevant reviews. Almost a quarter of reviews (n = 78, 23%) were published on asthma treatments which only account for 3-5% of consultations. Conversely, 15-23% of consultations are due to skin conditions yet they represent only 7% (n = 23) of reviews.

CONCLUSIONS/SIGNIFICANCE

Although Cochrane systematic reviews focus on clinical trials and do not provide a comprehensive picture of the evidence base underpinning the management of children in primary care, the mismatch between the focus of the published research and the focus of clinical activity is striking. Clinical trials are an important component of the evidence base and the lack of trial evidence to demonstrate intervention effectiveness in substantial areas of primary care for children should be addressed.

Writing narrative literature reviews for peer-reviewed journals: secrets of the trade

OBJECTIVE

To describe and discuss the process used to write a narrative review of the literature for publication in a peer-reviewed journal. Publication of narrative overviews of the literature should be standardized to increase their objectivity.

BACKGROUND

In the past decade numerous changes in research methodology pertaining to reviews of the literature have occurred. These changes necessitate authors of review articles to be familiar with current standards in the publication process.

METHODS

Narrative overview of the literature synthesizing the findings of literature retrieved from searches of computerized databases, hand searches, and authoritative texts.

DISCUSSION

An overview of the use of three types of reviews of the literature is presented. Step by step instructions for how to conduct and write a narrative overview utilizing a 'best-evidence synthesis' approach are discussed, starting with appropriate preparatory work and ending with how to create proper illustrations. Several resources for creating reviews of the literature are presented and a narrative overview critical appraisal worksheet is included. A bibliography of other useful reading is presented in an appendix.

CONCLUSION

Narrative overviews can be a valuable contribution to the literature if prepared properly. New and experienced authors wishing to write a narrative overview should find this article useful in constructing such a paper and carrying out the research process. It is hoped that this article will stimulate scholarly dialog amongst colleagues about this research design and other complex literature review methods.

Evaluating systematic reviews in urology: A practical guide

Systematic reviews of good quality randomized controlled trials that have little heterogeneity (variability) are considered to provide the best source of evidence for the efficacy of interventions in healthcare. With the recent national provision for access to The Cochrane Library to all residents in India, urologists and other clinicians now have access to this reliable source of regularly updated systematic reviews. This article uses six systematic reviews relevant to urologists from The Cochrane Library produced by different collaborative review groups in The Cochrane Collaboration to illustrate the methods used to minimize bias, improve transparency and provide reliable estimates of treatment effects. Issues in evaluating results, especially when subsequent trials produce discrepant results, are discussed.