Characteristics and methods of incorporating randomized and nonrandomized evidence in network meta-analyses: a scoping review

crossnma: An R package to synthesize cross-design evidence and cross-format data using network meta-analysis and network meta-regression

BACKGROUND

Although aggregate data (AD) from randomised clinical trials (RCTs) are used in the majority of network meta-analyses (NMAs), other study designs (e.g., cohort studies and other non-randomised studies, NRS) can be informative about relative treatment effects. The individual participant data (IPD) of the study, when available, are preferred to AD for adjusting for important participant characteristics and to better handle heterogeneity and inconsistency in the network.

RESULTS

We developed the R package crossnma to perform cross-format (IPD and AD) and cross-design (RCT and NRS) NMA and network meta-regression (NMR). The models are implemented as Bayesian three-level hierarchical models using Just Another Gibbs Sampler (JAGS) software within the R environment. The R package crossnma includes functions to automatically create the JAGS model, reformat the data (based on user input), assess convergence and summarize the results. We demonstrate the workflow within crossnma by using a network of six trials comparing four treatments.

CONCLUSIONS

The R package crossnma enables the user to perform NMA and NMR with different data types in a Bayesian framework and facilitates the inclusion of all types of evidence recognising differences in risk of bias.

Twenty years of network meta-analysis: Continuing controversies and recent developments

Network meta-analysis (NMA) is an extension of pairwise meta-analysis (PMA) which combines evidence from trials on multiple treatments in connected networks. NMA delivers internally consistent estimates of relative treatment efficacy, needed for rational decision making. Over its first 20 years NMA's use has grown exponentially, with applications in both health technology assessment (HTA), primarily re-imbursement decisions and clinical guideline development, and clinical research publications. This has been a period of transition in meta-analysis, first from its roots in educational and social psychology, where large heterogeneous datasets could be explored to find effect modifiers, to smaller pairwise meta-analyses in clinical medicine on average with less than six studies. This has been followed by narrowly-focused estimation of the effects of specific treatments at specific doses in specific populations in sparse networks, where direct comparisons are unavailable or informed by only one or two studies. NMA is a powerful and well-established technique but, in spite of the exponential increase in applications, doubts about the reliability and validity of NMA persist. Here we outline the continuing controversies, and review some recent developments. We suggest that heterogeneity should be minimized, as it poses a threat to the reliability of NMA which has not been fully appreciated, perhaps because it has not been seen as a problem in PMA. More research is needed on the extent of heterogeneity and inconsistency in datasets used for decision making, on formal methods for making recommendations based on NMA, and on the further development of multi-level network meta-regression.

Comparison of statistical methods for integrating real-world evidence in a rare events meta-analysis of randomized controlled trials

Rare events meta-analyses of randomized controlled trials (RCTs) are often underpowered because the outcomes are infrequent. Real-world evidence (RWE) from non-randomized studies may provide valuable complementary evidence about the effects of rare events, and there is growing interest in including such evidence in the decision-making process. Several methods for combining RCTs and RWE studies have been proposed, but the comparative performance of these methods is not well understood. We describe a simulation study that aims to evaluate an array of alternative Bayesian methods for including RWE in rare events meta-analysis of RCTs: the naïve data synthesis, the design-adjusted synthesis, the use of RWE as prior information, the three-level hierarchical models, and the bias-corrected meta-analysis model. The percentage bias, root-mean-square-error, mean 95% credible interval width, coverage probability, and power are used to measure performance. The various methods are illustrated using a systematic review to evaluate the risk of diabetic ketoacidosis among patients using sodium/glucose co-transporter 2 inhibitors as compared with active-comparators. Our simulations show that the bias-corrected meta-analysis model is comparable to or better than the other methods in terms of all evaluated performance measures and simulation scenarios. Our results also demonstrate that data solely from RCTs may not be sufficiently reliable for assessing the effects of rare events. In summary, the inclusion of RWE could increase the certainty and comprehensiveness of the body of evidence of rare events from RCTs, and the bias-corrected meta-analysis model may be preferable.

Synthesizing cross-design evidence and cross-format data using network meta-regression

In network meta-analysis (NMA), we synthesize all relevant evidence about health outcomes with competing treatments. The evidence may come from randomized clinical trials (RCT) or non-randomized studies (NRS) as individual participant data (IPD) or as aggregate data (AD). We present a suite of Bayesian NMA and network meta-regression (NMR) models allowing for cross-design and cross-format synthesis. The models integrate a three-level hierarchical model for synthesizing IPD and AD into four approaches. The four approaches account for differences in the design and risk of bias (RoB) in the RCT and NRS evidence. These four approaches variously ignoring differences in RoB, using NRS to construct penalized treatment effect priors and bias-adjustment models that control the contribution of information from high RoB studies in two different ways. We illustrate the methods in a network of three pharmacological interventions and placebo for patients with relapsing-remitting multiple sclerosis. The estimated relative treatment effects do not change much when we accounted for differences in design and RoB. Conducting network meta-regression showed that intervention efficacy decreases with increasing participant age. We also re-analysed a network of 431 RCT comparing 21 antidepressants, and we did not observe material changes in intervention efficacy when adjusting for studies' high RoB. We re-analysed both case studies accounting for different study RoB. In summary, the described suite of NMA/NMR models enables the inclusion of all relevant evidence while incorporating information on the within-study bias in both observational and experimental data and enabling estimation of individualized treatment effects through the inclusion of participant characteristics.

Methods for the Inclusion of Real-World Evidence in a Rare Events Meta-Analysis of Randomized Controlled Trials

BACKGROUND

Many rare events meta-analyses of randomized controlled trials (RCTs) have lower statistical power, and real-world evidence (RWE) is becoming widely recognized as a valuable source of evidence. The purpose of this study is to investigate methods for including RWE in a rare events meta-analysis of RCTs and the impact on the level of uncertainty around the estimates.

METHODS

Four methods for the inclusion of RWE in evidence synthesis were investigated by applying them to two previously published rare events meta-analyses: the naïve data synthesis (NDS), the design-adjusted synthesis (DAS), the use of RWE as prior information (RPI), and the three-level hierarchical models (THMs). We gauged the effect of the inclusion of RWE by varying the degree of confidence placed in RWE.

RESULTS

This study showed that the inclusion of RWE in a rare events meta-analysis of RCTs could increase the precision of the estimates, but this depended on the method of inclusion and the level of confidence placed in RWE. NDS cannot consider the bias of RWE, and its results may be misleading. DAS resulted in stable estimates for the two examples, regardless of whether we placed high- or low-level confidence in RWE. The results of the RPI approach were sensitive to the confidence level placed in RWE. The THM was effective in allowing for accommodating differences between study types, while it had a conservative result compared with other methods.

CONCLUSION

The inclusion of RWE in a rare events meta-analysis of RCTs could increase the level of certainty of the estimates and enhance the decision-making process. DAS might be appropriate for inclusion of RWE in a rare event meta-analysis of RCTs, but further evaluation in different scenarios of empirical or simulation studies is still warranted.

Lack of Efficacy of Bone Void Filling Materials in Medial Opening-Wedge High Tibial Osteotomy: A Systematic Review and Network Meta-Analysis

PURPOSE

To systematically review the clinical and radiologic outcomes of isolated medial opening-wedge high tibial osteotomies with different bone void filling materials and to compare the outcomes by network meta-analysis.

METHODS

This systematic review and network meta-analysis included searches of Medline, Embase, Cochrane Library, Web of Science, and Scopus from inception to July 30, 2022, for clinical comparative studies comparing 2 or more bone void filling materials in patients undergoing medial opening-wedge high tibial osteotomies. We performed Bayesian random-effect network meta-analyses to summarize the evidence and applied the Grading of Recommendations Assessment, Development, and Evaluation frameworks to rate the certainty of evidence, calculate the absolute effects, and present the findings. Cochrane Risk of Bias Tool 2.0 and modified Newcastle-Ottawa Scale were used to assess the risk of bias.

RESULTS

In total, 2,755 citations were identified by our search, of which 25 eligible trials, including 10 randomized controlled trials and 15 nonrandomized comparative trials (NCTs) enrolled 1,420 participants and 6 different interventions (autografts, allografts, synthetic grafts, mixed grafts, xenografts, and without grafts). There were some concerns on the risk of bias assessment among randomized controlled trials, and the median Newcastle-Ottawa Scale score was 6 for NCTs. All fillers showed no significantly superior treatment effects when compared with unfilled group in final Knee Society Scoring, Western Ontario and McMasters Universities score, time to bone union (TBU), and loss of correction (LOC). Exceptionally, moderate-certainty evidence suggested that autograft would produce superior incidence of complete bone union (CBU) than the unfilled at postoperative 1 year (odds ratio [OR] 13.0, 95% confidence interval [CI] 1.60-95.6), whereas low- to very low-certainty evidence suggested allografts (OR 0.2, 95% CI 0.06-0.52) and synthetic grafts (OR 0.29, 95% CI 0.10-0.68) would result in inferior CBU. Low-certainty evidence suggested allografts would result in larger LOC angle than unfilled group (mean difference 1.1, 95% CI 0.1-2.3). As for TBU, low-certainty evidence suggested mixed grafts would take longer time to reach clinical bone union (mean difference -14.04, 95% CI -21.0 to -6.9).

CONCLUSIONS

There is a lack of efficacy for different bone void filling materials to result better outcomes in Knee Society Scoring, Western Ontario and McMasters Universities score, TBU, and LOC than without graft. Although applying the autografts would produce a superior possibility of radiologic CBU than other fillers, because of the inclusion of NCTs, the overall certainty of the evidence synthesis is low.

LEVEL OF EVIDENCE

Level Ⅲ, meta-analysis of Level I randomized controlled trials and Level Ⅱ∼Ⅲ non-randomized comparative trails.

Stepwise-Hierarchical Pooled Analysis for Synergistic Interpretation of Meta-analyses Involving Randomized and Observational Studies: Methodology Development

Background

The necessity of including observational studies in meta-analyses has been discussed in the literature, but a synergistic analysis method for combining randomized and observational studies has not been reported. Observational studies differ in validity depending on the degree of the confounders’ influence. Combining interpretations may be challenging, especially if the statistical directions are similar but the magnitude of the pooled results are different between randomized and observational studies (the ”gray zone”).

Objective

To overcome these hindrances, in this study, we aim to introduce a logical method for clinical interpretation of randomized and observational studies.

Methods

We designed a stepwise-hierarchical pooled analysis method to analyze both distribution trends and individual pooled results by dividing the included studies into at least three stages (eg, all studies, balanced studies, and randomized studies).

Results

According to the model, the validity of a hypothesis is mostly based on the pooled results of randomized studies (the highest stage). Ascending patterns in which effect size and statistical significance increase gradually with stage strengthen the validity of the hypothesis; in this case, the effect size of the observational studies is lower than that of the true effect (eg, because of the uncontrolled effect of negative confounders). Descending patterns in which decreasing effect size and statistical significance gradually weaken the validity of the hypothesis suggest that the effect size and statistical significance of the observational studies is larger than the true effect (eg, because of researchers’ bias).

Conclusions

We recommend using the stepwise-hierarchical pooled analysis approach for meta-analyses involving randomized and observational studies.