An encouraging assessment of methods to inform priorities for updating systematic reviews

Epidemiology and reporting characteristics of non-Cochrane updates of systematic reviews: A cross-sectional study

BACKGROUND

It is important that systematic reviews (SRs) are up-to-date, otherwise they cannot be relied upon to guide decision-making in practice and policy. Our aim was to investigate epidemiological, descriptive and reporting characteristics of a cross-section of recently published updates of SRs.

METHODS

A SR update was defined as a new edition of a SR, either published by the same or new authors. We searched PubMed for SR updates published from January 01, 2016 to January 22, 2018 and included a random sample of n = 100 non-Cochrane updates of SRs on interventions reported in English.

RESULTS

Most SR updates had a corresponding author from the United Kingdom, United States, or Canada (in total 48/100) and dealt with nonpharmacological interventions (63/100). The SR updates were published a median of 5 years (interquartile range [IQR] 3-7) after the previous SR and included a median of 19 (IQR 9-28) studies. 31/100 SR updates reported that the conclusion had changed since the previous version. Only 51/100 SR updates used the term "update" in the title and none reported having based the decision to update the previous SR on an existing method/decision tool. The number of newly included studies and participants and the number of studies and participants included in/from the previous SR were often not reported.

CONCLUSIONS

The included non-Cochrane updates were frequently missing important information that would be expected to be present in a SR update. Thus, structured and detailed reporting guidance specific to SR updates is needed. It should focus particularly on appropriate labeling and justification of updates, and how to incorporate information regarding the previous SR.

The risk of conclusion change in systematic review updates can be estimated by learning from a database of published examples

OBJECTIVES

To determine which systematic review characteristics are needed to estimate the risk of conclusion change in systematic review updates.

STUDY DESIGN AND SETTING

We applied classification trees (a machine learning method) to model the risk of conclusion change in systematic review updates, using pairs of systematic reviews and their updates as samples. The classifiers were constructed using a set of features extracted from systematic reviews and the relevant trials added in published updates. Model performance was measured by recall, precision, and area under the receiver operating characteristic curve (AUC).

RESULTS

We identified 63 pairs of systematic reviews and updates, of which 20 (32%) exhibited a change in conclusion in their updates. A classifier using information about new trials exhibited the highest performance (AUC: 0.71; recall: 0.75; precision: 0.43) compared to a classifier that used fewer features (AUC: 0.65; recall: 0.75; precision: 0.39).

CONCLUSION

When estimating the risk of conclusion change in systematic review updates, information about the sizes of trials that will be added in an update are most useful. Future tools aimed at signaling conclusion change risks would benefit from complementary tools that automate screening of relevant trials.

Study-based registers reduce waste in systematic reviewing: discussion and case report

BACKGROUND

Maintained study-based registers (SBRs) have, at their core, study records linked to, potentially, multiple other records such as references, data sets, standard texts and full-text reports. Such registers can minimise and refine searching, de-duplicating, screening and acquisition of full texts. SBRs can facilitate new review titles/updates and, within seconds, inform the team about the potential workload of each task.

METHODS

We discuss the advantages/disadvantages of SBRs and report a case of how such a register was used to develop a successful grant application and deliver results-reducing considerable redundancy of effort.

RESULTS

SBRs saved time in question-setting and scoping and made rapid production of nine Cochrane systematic reviews possible.

CONCLUSION

Whilst helping prioritise and conduct systematic reviews, SBRs improve quality. Those funding information specialists for literature reviewing could reasonably stipulate the resulting SBR to be delivered for dissemination and use beyond the life of the project.

Living network meta-analysis was feasible when considering the pace of evidence generation

OBJECTIVES

The aim of the study was to assess the feasibility of living network meta-analysis (NMA) taking into account the pace of evidence generation across different medical areas.

STUDY DESIGN AND SETTING

We performed a systematic review to identify published NMAs. For each NMA, we calculated the cumulative number of new trials. To assess the feasibility of living NMA, we considered different update frequencies (4, 6, and 12 months), then evaluated the number of new trials to be included at each update in the NMA and the workload percentage for an update relative to the initial NMA.

RESULTS

We identified 77 NMAs covering 17 different medical areas; 60 (78%) had fewer than four new trials included per year, on average, and 5 (7%) had more than seven trials. With an update frequency of 4, 6, and 12 months, the median number of new trials to be included in the NMA was 0 (interquartile range, 0-1), 1 (0-2), and 2 (1-4), respectively, with mean of 4%, 5%, and 11% workload per update, respectively.

CONCLUSION

The workload associated with updating a living NMA represents about one-tenth of the initial workload; therefore, living NMA is manageable.

Methodological systematic review identifies major limitations in prioritization processes for updating

OBJECTIVES

The aim of the study was to identify and describe strategies to prioritize the updating of systematic reviews (SRs), health technology assessments (HTAs), or clinical guidelines (CGs).

STUDY DESIGN AND SETTING

We conducted an SR of studies describing one or more methods to prioritize SRs, HTAs, or CGs for updating. We searched MEDLINE (PubMed, from 1966 to August 2016) and The Cochrane Methodology Register (The Cochrane Library, Issue 8 2016). We hand searched abstract books, reviewed reference lists, and contacted experts. Two reviewers independently screened the references and extracted data.

RESULTS

We included 14 studies. Six studies were classified as descriptive (6 of 14, 42.9%) and eight as implementation studies (8 of 14, 57.1%). Six studies reported an updating strategy (6 of 14, 42.9%), six a prioritization process (6 of 14, 42.9%), and two a prioritization criterion (2 of 14, 14.2%). Eight studies focused on SRs (8 of 14, 57.1%), six studies focused on CGs (6 of 14, 42.9%), and none were about HTAs. We identified 76 prioritization criteria that can be applied when prioritizing documents for updating. The most frequently cited criteria were as follows: available evidence (19 of 76, 25.0%), clinical relevance (10 of 76; 13.2%), and users' interest (10 of 76; 13.2%).

CONCLUSION

There is wide variability and suboptimal reporting of the methods used to develop and implement processes to prioritize updating of SRs, HTAs, and CGs.

The INVEST project: investigating the use of evidence synthesis in the design and analysis of clinical trials

BACKGROUND

When designing and analysing clinical trials, using previous relevant information, perhaps in the form of evidence syntheses, can reduce research waste. We conducted the INVEST (INVestigating the use of Evidence Synthesis in the design and analysis of clinical Trials) survey to summarise the current use of evidence synthesis in trial design and analysis, to capture opinions of trialists and methodologists on such use, and to understand any barriers.

METHODS

Our sampling frame was all delegates attending the International Clinical Trials Methodology Conference in November 2015. Respondents were asked to indicate (1) their views on the use of evidence synthesis in trial design and analysis, (2) their own use during the past 10 years and (3) the three greatest barriers to use in practice.

RESULTS

Of approximately 638 attendees of the conference, 106 (17%) completed the survey, half of whom were statisticians. Support was generally high for using a description of previous evidence, a systematic review or a meta-analysis in trial design. Generally, respondents did not seem to be using evidence syntheses as often as they felt they should. For example, only 50% (42/84 relevant respondents) had used a meta-analysis to inform whether a trial is needed compared with 74% (62/84) indicating that this is desirable. Only 6% (5/81 relevant respondents) had used a value of information analysis to inform sample size calculations versus 22% (18/81) indicating support for this. Surprisingly large numbers of participants indicated support for, and previous use of, evidence syntheses in trial analysis. For example, 79% (79/100) of respondents indicated that external information about the treatment effect should be used to inform aspects of the analysis. The greatest perceived barrier to using evidence synthesis methods in trial design or analysis was time constraints, followed by a belief that the new trial was the first in the area.

CONCLUSIONS

Evidence syntheses can be resource-intensive, but their use in informing the design, conduct and analysis of clinical trials is widely considered desirable. We advocate additional research, training and investment in resources dedicated to ways in which evidence syntheses can be undertaken more efficiently, offering the potential for cost savings in the long term.

Prioritizing covariates in the planning of future studies in the meta-analytic framework

Science can be seen as a sequential process where each new study augments evidence to the existing knowledge. To have the best prospects to make an impact in this process, a new study should be designed optimally taking into account the previous studies and other prior information. We propose a formal approach for the covariate prioritization, that is the decision about the covariates to be measured in a new study. The decision criteria can be based on conditional power, change of the p-value, change in lower confidence limit, Kullback-Leibler divergence, Bayes factors, Bayesian false discovery rate or difference between prior and posterior expectation. The criteria can be also used for decisions on the sample size. As an illustration, we consider covariate prioritization based on genome-wide association studies for C-reactive protein levels and make suggestions on the genes to be studied further.

Considerations from the risk of bias perspective for updating Cochrane reviews

Authors of Cochrane reviews are expected to update their reviews every 2 years. The updating process helps to ensure that reviews are current and include recent evidence. However, the updating process is time-consuming for authors, particularly when Cochrane methods evolve and authors are required to revisit some of the originally included studies.The Cochrane Collaboration's 'Risk of bias' tool is a mandatory component of Cochrane reviews, providing an assessment of the potential biases of included studies. The tool has been modified most recently in 2011, and the expectation is that new versions will continue to be produced and utilised in all Cochrane reviews. In this commentary we discuss, in the context of updating scenarios that are likely to be encountered, the potential options systematic review authors may have recourse to when the Cochrane Collaboration's 'Risk of bias' tool has been modified between the original review and its update. We recommend that authors who are updating reviews should revise their original assessments of included studies using the most recent version of the risk of bias tool. Despite the increased workload, use of the most recent version of the tool facilitates consistency of methods and reporting both across and within reviews, and ensures currency to the methodological rigour.

An exploration of synthesis methods in public health evaluations of interventions concludes that the use of modern statistical methods would be beneficial

OBJECTIVES

To review the methods currently used to synthesize evidence in public health evaluations and demonstrate the availability of more sophisticated approaches.

STUDY DESIGN AND SETTING

A systematic review of National Institute for Health and Care Excellence (NICE) public health appraisals published between 2006 and 2012 was performed to assess the methods used for the synthesis of effectiveness evidence. The ability of new developments in evidence synthesis methodology to address the challenges and opportunities present in a public health context is demonstrated.

RESULTS

Nine (23%) of the 39 NICE appraisals included in the review performed pairwise meta-analyses as part of the effectiveness review with one of these also including a network meta-analysis. Of the remainder, 29 (74.4%) presented narrative summaries of the evidence only, and 1 (2.6%) appraisal did not present any review of effectiveness and/or cost-effectiveness evidence. Heterogeneity of outcomes, methods, and interventions were the main reasons given for not pooling the data. Exploration of quantitative synthesis methods shows that pairwise meta-analyses can be extended to incorporate individual participant data (when it is available), extend the number of interventions being compared using a network meta-analysis, and adjust for both subject- and summary-level covariates. All these can contribute to ensuring the analysis answers directly the policy-relevant questions.

CONCLUSION

More sophisticated methods in evidence synthesis should be considered to make evaluations in public health more useful for decision makers.

Overlapping meta-analyses on the same topic: survey of published studies

OBJECTIVE

To assess how common it is to have multiple overlapping meta-analyses of randomized trials published on the same topic.

DESIGN

Survey of published meta-analyses.

DATA SOURCES

PubMed.

STUDY SELECTION AND METHODS

Meta-analyses published in 2010 were identified, and 5% of them were randomly selected. We further selected those that included randomized trials and examined effectiveness of any medical intervention. For eligible meta-analyses, we searched for other meta-analyses on the same topic (covering the same comparisons, indications/settings, and outcomes or overlapping subsets of them) published until February 2013.

RESULTS

Of 73 eligible meta-analyses published in 2010, 49 (67%) had at least one other overlapping meta-analysis (median two meta-analyses per topic, interquartile range 1-4, maximum 13). In 17 topics at least one author was involved in at least two of the overlapping meta-analyses. No characteristics of the index meta-analyses were associated with the potential for overlapping meta-analyses. Among pairs of overlapping meta-analyses in 20 randomly selected topics, 13 of the more recent meta-analyses did not include any additional outcomes. In three of the four topics with eight or more published meta-analyses, many meta-analyses examined only a subset of the eligible interventions or indications/settings covered by the index meta-analysis. Conversely, for statins in the prevention of atrial fibrillation after cardiac surgery, 11 meta-analyses were published with similar eligibility criteria for interventions and setting: there was still variability on which studies were included, but the results were always similar or even identical across meta-analyses.

CONCLUSIONS

While some independent replication of meta-analyses by different teams is possibly useful, the overall picture suggests that there is a waste of efforts with many topics covered by multiple overlapping meta-analyses.

Planning future studies based on the conditional power of a meta-analysis

Systematic reviews often provide recommendations for further research. When meta-analyses are inconclusive, such recommendations typically argue for further studies to be conducted. However, the nature and amount of future research should depend on the nature and amount of the existing research. We propose a method based on conditional power to make these recommendations more specific. Assuming a random-effects meta-analysis model, we evaluate the influence of the number of additional studies, of their information sizes and of the heterogeneity anticipated among them on the ability of an updated meta-analysis to detect a prespecified effect size. The conditional powers of possible design alternatives can be summarized in a simple graph which can also be the basis for decision making. We use three examples from the Cochrane Database of Systematic Reviews to demonstrate our strategy. We demonstrate that if heterogeneity is anticipated, it might not be possible for a single study to reach the desirable power no matter how large it is.

A comparison of statistical methods for identifying out-of-date systematic reviews

BACKGROUND

Systematic reviews (SRs) can provide accurate and reliable evidence, typically about the effectiveness of health interventions. Evidence is dynamic, and if SRs are out-of-date this information may not be useful; it may even be harmful. This study aimed to compare five statistical methods to identify out-of-date SRs.

METHODS

A retrospective cohort of SRs registered in the Cochrane Pregnancy and Childbirth Group (CPCG), published between 2008 and 2010, were considered for inclusion. For each eligible CPCG review, data were extracted and "3-years previous" meta-analyses were assessed for the need to update, given the data from the most recent 3 years. Each of the five statistical methods was used, with random effects analyses throughout the study.

RESULTS

Eighty reviews were included in this study; most were in the area of induction of labour. The numbers of reviews identified as being out-of-date using the Ottawa, recursive cumulative meta-analysis (CMA), and Barrowman methods were 34, 7, and 7 respectively. No reviews were identified as being out-of-date using the simulation-based power method, or the CMA for sufficiency and stability method. The overall agreement among the three discriminating statistical methods was slight (Kappa = 0.14; 95% CI 0.05 to 0.23). The recursive cumulative meta-analysis, Ottawa, and Barrowman methods were practical according to the study criteria.

CONCLUSION

Our study shows that three practical statistical methods could be applied to examine the need to update SRs.

Graphical augmentations to the funnel plot assess the impact of additional evidence on a meta-analysis

OBJECTIVE

We aim to illustrate the potential impact of a new study on a meta-analysis, which gives an indication of the robustness of the meta-analysis.

STUDY DESIGN AND SETTING

A number of augmentations are proposed to one of the most widely used of graphical displays, the funnel plot. Namely, 1) statistical significance contours, which define regions of the funnel plot in which a new study would have to be located to change the statistical significance of the meta-analysis; and 2) heterogeneity contours, which show how a new study would affect the extent of heterogeneity in a given meta-analysis. Several other features are also described, and the use of multiple features simultaneously is considered.

RESULTS

The statistical significance contours suggest that one additional study, no matter how large, may have a very limited impact on the statistical significance of a meta-analysis. The heterogeneity contours illustrate that one outlying study can increase the level of heterogeneity dramatically.

CONCLUSION

The additional features of the funnel plot have applications including 1) informing sample size calculations for the design of future studies eligible for inclusion in the meta-analysis; and 2) informing the updating prioritization of a portfolio of meta-analyses such as those prepared by the Cochrane Collaboration.

Simulation methods to estimate design power: an overview for applied research

BACKGROUND

Estimating the required sample size and statistical power for a study is an integral part of study design. For standard designs, power equations provide an efficient solution to the problem, but they are unavailable for many complex study designs that arise in practice. For such complex study designs, computer simulation is a useful alternative for estimating study power. Although this approach is well known among statisticians, in our experience many epidemiologists and social scientists are unfamiliar with the technique. This article aims to address this knowledge gap.

METHODS

We review an approach to estimate study power for individual- or cluster-randomized designs using computer simulation. This flexible approach arises naturally from the model used to derive conventional power equations, but extends those methods to accommodate arbitrarily complex designs. The method is universally applicable to a broad range of designs and outcomes, and we present the material in a way that is approachable for quantitative, applied researchers. We illustrate the method using two examples (one simple, one complex) based on sanitation and nutritional interventions to improve child growth.

RESULTS

We first show how simulation reproduces conventional power estimates for simple randomized designs over a broad range of sample scenarios to familiarize the reader with the approach. We then demonstrate how to extend the simulation approach to more complex designs. Finally, we discuss extensions to the examples in the article, and provide computer code to efficiently run the example simulations in both R and Stata.

CONCLUSIONS

Simulation methods offer a flexible option to estimate statistical power for standard and non-traditional study designs and parameters of interest. The approach we have described is universally applicable for evaluating study designs used in epidemiologic and social science research.

Systematic reviews and meta-analysis

BACKGROUND

Systematic reviews and meta-analyses are important research tools in modern medicine. They serve to condense and clarify large amounts of data into resources that can educate clinicians, enhance patient care, help formulate clinical guidelines, and guide future research endeavors.

METHODS

The existing literature, including recently updated guidelines, on systematic reviews and meta-analysis was reviewed and summarized.

RESULTS

A brief background on the origins of systematic reviews is presented, and the advantages and disadvantages of this type of study are discussed. A step-by-step guide to conducting a proper systematic review is outlined, with many illustrative examples. The recently updated reporting guidelines for this type of study are included.

CONCLUSIONS

Using clinical examples and published guidelines, a framework is presented to help the reader properly conduct a systematic review. These guidelines also help the reader conduct a critical appraisal of systematic reviews published in the scientific literature. Even more importantly, principles regarding application of systematic review results to individual patients are addressed.

The prevalence and effect of publication bias in orthopaedic meta-analyses

BACKGROUND

Meta-analyses of randomized, controlled trials are considered the highest level-of-evidence, thus strongest source of information. However, questions concerning the validity of meta-analyses in orthopaedic surgery emerged recently. Among the most common sources for errors is publication bias. This describes the fact that studies with small or non-significant outcomes are less likely to be published, thus less likely to be identified and included in systematic reviews and meta-analyses. In this study we asked three questions: (1) Whether publication bias is assessed in orthopaedic meta-analyses, (2) What the actual prevalence of publication bias is, and (3) what effect publication bias has on the outcomes of orthopaedic meta-analyses.

METHODS

Using the estimate of 35 ± 20% for the prevalence of publication bias in meta-analyses obtained from earlier research we calculated a required samples size of 22 (plus 20% to account for attrition) and randomly selected 26 orthopaedic meta-analyses. To answer our first question we calculated the percentage of papers that report on formal assessment of publication bias. For our second question we obtained all primary studies (n = 321) from the included meta-analyses and used Egger's regression to search for evidence for publication bias. Third, we used the trim-and-fill method to assess the impact of publication bias, if present. This method estimates publication bias in a meta-analysis and adds hypothetical studies to reduce this bias; it thus produces an estimate of adjusted, unbiased outcomes that can be compared with the unadjusted, publication-biased outcomes to assess the effect of publication bias.

RESULTS

We found that only 35% (95% CI 20-57) of all orthopaedic meta-analyses published between 1992 and 2008 in English and German assessed publication bias. Most studies used funnel plots, which are rather insensitive. The prevalence of publication bias, based on a sensitivity analysis, ranged between 12 and 19%. Adjustment for publication bias did not produce significantly different results, but the magnitude of the pooled estimates in the affected meta-analyses changed by 29% (95% CI 0-63) on average.

CONCLUSION

We found a rather low prevalence of publication bias in orthopaedic meta-analyses, but recommend assessing for it and its effects, which might be substantial.