Model averaging for robust extrapolation in evidence synthesis

Problematic meta-analyses: Bayesian and frequentist perspectives on combining randomized controlled trials and non-randomized studies

PURPOSE

In the literature, the propriety of the meta-analytic treatment-effect produced by combining randomized controlled trials (RCT) and non-randomized studies (NRS) is questioned, given the inherent confounding in NRS that may bias the meta-analysis. The current study compared an implicitly principled pooled Bayesian meta-analytic treatment-effect with that of frequentist pooling of RCT and NRS to determine how well each approach handled the NRS bias.

MATERIALS & METHODS

Binary outcome Critical-Care meta-analyses, reflecting the importance of such outcomes in Critical-Care practice, combining RCT and NRS were identified electronically. Bayesian pooled treatment-effect and 95% credible-intervals (BCrI), posterior model probabilities indicating model plausibility and Bayes-factors (BF) were estimated using an informative heavy-tailed heterogeneity prior (half-Cauchy). Preference for pooling of RCT and NRS was indicated for Bayes-factors > 3 or < 0.333 for the converse. All pooled frequentist treatment-effects and 95% confidence intervals (FCI) were re-estimated using the popular DerSimonian-Laird (DSL) random effects model.

RESULTS

Fifty meta-analyses were identified (2009-2021), reporting pooled estimates in 44; 29 were pharmaceutical-therapeutic and 21 were non-pharmaceutical therapeutic. Re-computed pooled DSL FCI excluded the null (OR or RR = 1) in 86% (43/50). In 18 meta-analyses there was an agreement between FCI and BCrI in excluding the null. In 23 meta-analyses where FCI excluded the null, BCrI embraced the null. BF supported a pooled model in 27 meta-analyses and separate models in 4. The highest density of the posterior model probabilities for 0.333 < Bayes factor < 1 was 0.8.

CONCLUSIONS

In the current meta-analytic cohort, an integrated and multifaceted Bayesian approach gave support to including NRS in a pooled-estimate model. Conversely, caution should attend the reporting of naïve frequentist pooled, RCT and NRS, meta-analytic treatment effects.

Sharing information across patient subgroups to draw conclusions from sparse treatment networks

Network meta-analysis (NMA) usually provides estimates of the relative effects with the highest possible precision. However, sparse networks with few available studies and limited direct evidence can arise, threatening the robustness and reliability of NMA estimates. In these cases, the limited amount of available information can hamper the formal evaluation of the underlying NMA assumptions of transitivity and consistency. In addition, NMA estimates from sparse networks are expected to be imprecise and possibly biased as they rely on large-sample approximations that are invalid in the absence of sufficient data. We propose a Bayesian framework that allows sharing of information between two networks that pertain to different population subgroups. Specifically, we use the results from a subgroup with a lot of direct evidence (a dense network) to construct informative priors for the relative effects in the target subgroup (a sparse network). This is a two-stage approach where at the first stage, we extrapolate the results of the dense network to those expected from the sparse network. This takes place by using a modified hierarchical NMA model where we add a location parameter that shifts the distribution of the relative effects to make them applicable to the target population. At the second stage, these extrapolated results are used as prior information for the sparse network. We illustrate our approach through a motivating example of psychiatric patients. Our approach results in more precise and robust estimates of the relative effects and can adequately inform clinical practice in presence of sparse networks.

Estimation of treatment effects in early-phase randomized clinical trials involving external control data

There are good reasons to perform a randomized controlled trial (RCT) even in early phases of clinical development. However, the low sample sizes in those settings lead to high variability of the treatment effect estimate. The variability could be reduced by adding external control data if available. For the common setting of suitable subject-level control group data only available from one external (clinical trial or real-world) data source, we evaluate different analysis options for estimating the treatment effect via hazard ratios. The impact of the external control data is usually guided by the level of similarity with the current RCT data. Such level of similarity can be determined via outcome and/or baseline covariate data comparisons. We provide an overview over existing methods, propose a novel option for a combined assessment of outcome and baseline data, and compare a selected set of approaches in a simulation study under varying assumptions regarding observable and unobservable confounder distributions using a time-to-event model. Our various simulation scenarios also reflect the differences between external clinical trial and real-world data. Data combinations via simple outcome-based borrowing or simple propensity score weighting with baseline covariate data are not recommended. Analysis options which conflate outcome and baseline covariate data perform best in our simulation study.

Summarizing empirical information on between-study heterogeneity for Bayesian random-effects meta-analysis

In Bayesian meta-analysis, the specification of prior probabilities for the between-study heterogeneity is commonly required, and is of particular benefit in situations where only few studies are included. Among the considerations in the set-up of such prior distributions, the consultation of available empirical data on a set of relevant past analyses sometimes plays a role. How exactly to summarize historical data sensibly is not immediately obvious; in particular, the investigation of an empirical collection of heterogeneity estimates will not target the actual problem and will usually only be of limited use. The commonly used normal-normal hierarchical model for random-effects meta-analysis is extended to infer a heterogeneity prior. Using an example data set, we demonstrate how to fit a distribution to empirically observed heterogeneity data from a set of meta-analyses. Considerations also include the choice of a parametric distribution family. Here, we focus on simple and readily applicable approaches to then translate these into (prior) probability distributions.

Comparison of Bayesian methods for incorporating adult clinical trial data to improve certainty of treatment effect estimates in children

There are challenges associated with recruiting children to take part in randomised clinical trials and as a result, compared to adults, in many disease areas we are less certain about which treatments are most safe and effective. This can lead to weaker recommendations about which treatments to prescribe in practice. However, it may be possible to 'borrow strength' from adult evidence to improve our understanding of which treatments work best in children, and many different statistical methods are available to conduct these analyses. In this paper we discuss four Bayesian methods for extrapolating adult clinical trial evidence to children. Using an exemplar dataset, we compare the effect of their modelling assumptions on the estimated treatment effect and associated heterogeneity. These modelling assumptions range from adult evidence being completely generalisable to being completely unrelated to the children's evidence. We finally discuss the appropriateness of these modelling assumptions in the context of estimating treatment effect in children.

Using the bayesmeta R package for Bayesian random-effects meta-regression

BACKGROUND

Random-effects meta-analysis within a hierarchical normal modeling framework is commonly implemented in a wide range of evidence synthesis applications. More general problems may even be tackled when considering meta-regression approaches that in addition allow for the inclusion of study-level covariables.

METHODS

We describe the Bayesian meta-regression implementation provided in the bayesmetaR package including the choice of priors, and we illustrate its practical use.

RESULTS

A wide range of example applications are given, such as binary and continuous covariables, subgroup analysis, indirect comparisons, and model selection. Example R code is provided.

CONCLUSIONS

The bayesmeta package provides a flexible implementation. Due to the avoidance of MCMC methods, computations are fast and reproducible, facilitating quick sensitivity checks or large-scale simulation studies.

BRIDGING RANDOMIZED CONTROLLED TRIALS AND SINGLE-ARM TRIALS USING COMMENSURATE PRIORS IN ARM-BASED NETWORK META-ANALYSIS

Network meta-analysis (NMA) is a powerful tool to compare multiple treatments directly and indirectly by combining and contrasting multiple independent clinical trials. Because many NMAs collect only a few eligible randomized controlled trials (RCTs), there is an urgent need to synthesize different sources of information, e.g., from both RCTs and single-arm trials. However, single-arm trials and RCTs may have different populations and quality, so that assuming they are exchangeable may be inappropriate. This article presents a novel method using a commensurate prior on variance (CPV) to borrow variance (rather than mean) information from single-arm trials in an arm-based (AB) Bayesian NMA. We illustrate the advantages of this CPV method by reanalyzing an NMA of immune checkpoint inhibitors in cancer patients. Comprehensive simulations investigate the impact on statistical inference of including single-arm trials. The simulation results show that the CPV method provides efficient and robust estimation even when the two sources of information are moderately inconsistent.

Bayesian model-averaged meta-analysis in medicine

We outline a Bayesian model-averaged (BMA) meta-analysis for standardized mean differences in order to quantify evidence for both treatment effectiveness δ and across-study heterogeneity τ . We construct four competing models by orthogonally combining two present-absent assumptions, one for the treatment effect and one for across-study heterogeneity. To inform the choice of prior distributions for the model parameters, we used 50% of the Cochrane Database of Systematic Reviews to specify rival prior distributions for δ and τ . The relative predictive performance of the competing models and rival prior distributions was assessed using the remaining 50% of the Cochrane Database. On average, ℋ 1 r -the model that assumes the presence of a treatment effect as well as across-study heterogeneity-outpredicted the other models, but not by a large margin. Within ℋ 1 r , predictive adequacy was relatively constant across the rival prior distributions. We propose specific empirical prior distributions, both for the field in general and for each of 46 specific medical subdisciplines. An example from oral health demonstrates how the proposed prior distributions can be used to conduct a BMA meta-analysis in the open-source software R and JASP. The preregistered analysis plan is available at https://osf.io/zs3df/.

Classifying information-sharing methods

BACKGROUND

Sparse relative effectiveness evidence is a frequent problem in Health Technology Assessment (HTA). Where evidence directly pertaining to the decision problem is sparse, it may be feasible to expand the evidence-base to include studies that relate to the decision problem only indirectly: for instance, when there is no evidence on a comparator, evidence on other treatments of the same molecular class could be used; similarly, a decision on children may borrow-strength from evidence on adults. Usually, in HTA, such indirect evidence is either included by ignoring any differences ('lumping') or not included at all ('splitting'). However, a range of more sophisticated methods exists, primarily in the biostatistics literature. The objective of this study is to identify and classify the breadth of the available information-sharing methods.

METHODS

Forwards and backwards citation-mining techniques were used on a set of seminal papers on the topic of information-sharing. Papers were included if they specified (network) meta-analytic methods for combining information from distinct populations, interventions, outcomes or study-designs.

RESULTS

Overall, 89 papers were included. A plethora of evidence synthesis methods have been used for information-sharing. Most papers (n=79) described methods that shared information on relative treatment effects. Amongst these, there was a strong emphasis on methods for information-sharing across multiple outcomes (n=42) and treatments (n=25), with fewer papers focusing on study-designs (n=23) or populations (n=8). We categorise and discuss the methods under four 'core' relationships of information-sharing: functional, exchangeability-based, prior-based and multivariate relationships, and explain the assumptions made within each of these core approaches.

CONCLUSIONS

This study highlights the range of information-sharing methods available. These methods often impose more moderate assumptions than lumping or splitting. Hence, the degree of information-sharing that they impose could potentially be considered more appropriate. Our identification of four 'core' methods of information-sharing allows for an improved understanding of the assumptions underpinning the different methods. Further research is required to understand how the methods differ in terms of the strength of sharing they impose and the implications of this for health care decisions.

A novel equivalence probability weighted power prior for using historical control data in an adaptive clinical trial design: A comparison to standard methods

A standard two-arm randomised controlled trial usually compares an intervention to a control treatment with equal numbers of patients randomised to each treatment arm and only data from within the current trial are used to assess the treatment effect. Historical data are used when designing new trials and have recently been considered for use in the analysis when the required number of patients under a standard trial design cannot be achieved. Incorporating historical control data could lead to more efficient trials, reducing the number of controls required in the current study when the historical and current control data agree. However, when the data are inconsistent, there is potential for biased treatment effect estimates, inflated type I error and reduced power. We introduce two novel approaches for binary data which discount historical data based on the agreement with the current trial controls, an equivalence approach and an approach based on tail area probabilities. An adaptive design is used where the allocation ratio is adapted at the interim analysis, randomising fewer patients to control when there is agreement. The historical data are down-weighted in the analysis using the power prior approach with a fixed power. We compare operating characteristics of the proposed design to historical data methods in the literature: the modified power prior; commensurate prior; and robust mixture prior. The equivalence probability weight approach is intuitive and the operating characteristics can be calculated exactly. Furthermore, the equivalence bounds can be chosen to control the maximum possible inflation in type I error.

Assessing efficacy in important subgroups in confirmatory trials: An example using Bayesian dynamic borrowing

Assessment of efficacy in important subgroups - such as those defined by sex, age, race and region - in confirmatory trials is typically performed using separate analysis of the specific subgroup. This ignores relevant information from the complementary subgroup. Bayesian dynamic borrowing uses an informative prior based on analysis of the complementary subgroup and a weak prior distribution centred on a mean of zero to construct a robust mixture prior. This combination of priors allows for dynamic borrowing of prior information; the analysis learns how much of the complementary subgroup prior information to borrow based on the consistency between the subgroup of interest and the complementary subgroup. A tipping point analysis can be carried out to identify how much prior weight needs to be placed on the complementary subgroup component of the robust mixture prior to establish efficacy in the subgroup of interest. An attractive feature of the tipping point analysis is that it enables the evidence from the source subgroup, the evidence from the target subgroup, and the combined evidence to be displayed alongside each other. This method is illustrated with an example trial in severe asthma where efficacy in the adolescent subgroup was assessed using a mixture prior combining an informative prior from the adult data in the same trial with a non-informative prior.

On weakly informative prior distributions for the heterogeneity parameter in Bayesian random-effects meta-analysis

The normal-normal hierarchical model (NNHM) constitutes a simple and widely used framework for meta-analysis. In the common case of only few studies contributing to the meta-analysis, standard approaches to inference tend to perform poorly, and Bayesian meta-analysis has been suggested as a potential solution. The Bayesian approach, however, requires the sensible specification of prior distributions. While noninformative priors are commonly used for the overall mean effect, the use of weakly informative priors has been suggested for the heterogeneity parameter, in particular in the setting of (very) few studies. To date, however, a consensus on how to generally specify a weakly informative heterogeneity prior is lacking. Here we investigate the problem more closely and provide some guidance on prior specification.

ECMO, ARDS and meta-analyses: Bayes to the rescue?

PURPOSE

A recent meta-analysis by Munshi et al. (Lancet Respiratory Medicine, 2019) claimed mortality treatment efficacy for extra corporeal membrane oxygenation (ECMO) in the acute respitratory syndrome (ARDS) despite very low meta-analytic study numbers (n = 2 (RCTs), risk-ratio (RR) 0·73 (95%CI: 0·58-0·92); n = 5 (2 RCT, 3 observational), RR 0·69 (95%CI: 0·50-0·95)). We explore this efficacy claim by a comprehensive re-analysis of the data.

METHODS

Data were sourced from the two- and five-study meta-analyses, conducted using the Der-Simonian & Laird (DSL) method. A variety of frequentist (DSL, restricted maximum likelihood (REML), Paul-Mandel (PM), with/without Hartung-Knapp-Sidik-Jonkman variance correction), a beta-binomial model (BBN)) and Bayesian models (2 finite-mixture and several Markov-Chain-Monte-Carlo) were used to estimate treatment effects. Fragility-indices, the minimum patients changing mortality outcome needed to induce a conclusion change were also applied.

RESULTS

For the 2-study and 5-study meta-analysis only the uncorrected frequentist estimators (DSL, REML, PM) demonstrated significant RR. Except for the BBN model, which was significant for the 2-study meta-analysis, intervals for all other models included the null. Both meta-analyses demonstrated fragility.

CONCLUSIONS

Having canvassed the conduct of both meta-analyses presented by Munshi et al. and proffered alternative methods, we find no certainty regarding the efficacy of ECMO in ARDS.

Revisit of test-then-pool methods and some practical considerations

Test-then-pool is a simple statistical method that borrows historical information to improve efficiency of the drug development process. The original test-then-pool method examines the difference between the historical and current information and then pools the information if there is no significant difference. One drawback of this method is that a nonsignificant difference may not always imply consistency between the historical and current information. As a result, the original test-then-pool method is more likely to incorrectly borrow information from the historical control when the current trial has a small sample size. Statistically, it is more natural to use an equivalence test for examining the consistency. This manuscript develops an equivalence-based test-then-pool method for a continuous endpoint, explains the relationship between the two test-then-pool methods, explores the choice of an equivalence margin through the overlap probability, and proposes an adjustment to the nominal testing level for controlling type I error under the true consistency scenario. Furthermore, the analytical forms of the type I error and power for the two test-then-pool methods are derived, and practical considerations for using them are presented.

A multicenter, randomized, placebo-controlled, double-blind phase 3 trial with open-arm comparison indicates safety and efficacy of nephroprotective therapy with ramipril in children with Alport's syndrome

Children with Alport syndrome develop renal failure early in life. Since the safety and efficacy of preemptive nephroprotective therapy are uncertain we conducted a randomized, placebo-controlled, double-blind trial in 14 German sites of pediatric patients with ramipril for three to six years plus six months follow-up to determine these parameters. Pretreated children and those whose parents refused randomization became an open-arm control, which were compared to prospective real-world data from untreated children. The co-primary endpoints were safety (adverse drug reactions) and efficacy (time to progression). Out of 66 oligosymptomatic children, 22 were randomized and 44 joined the open-arm comparison. Ramipril therapy showed no safety issues (total of 216.4 patient-years on ramipril; adverse event rate-ratio 1.00; 95% confidence interval 0.66-1.53). Although not significant, our results cautiously showed that ramipril therapy was effective: in the randomized arm, Ramipril decreased the risk of disease progression by almost half (hazard ratio 0.51 (0.12-2.20)), diminished the slope of albuminuria progression and the decline in glomerular filtration. In adjusted analysis, indications of efficacy were supported by prospective data from participants treated open label compared with untreated children, in whom ramipril again seemed to reduce progression by almost half (0.53 (0.22-1.29)). Incorporating these results into the randomized data by Bayesian evidence synthesis resulted in a more precise estimate of the hazard-ratio of 0.52 (0.19-1.39). Thus, our study shows the safety of early initiation of therapy and supports the hope to slow renal failure by many years, emphasizing the value of preemptive therapy. Hence, screening programs for glomerular hematuria in children and young adults could benefit from inclusion of genetic testing for Alport-related gene-variants.

Clinical Sequelae From Overfeeding in Enterally Fed Critically Ill Adults: Where Is the Evidence?

Enteral energy delivery above requirements (overfeeding) is believed to cause adverse effects during critical illness, but the literature supporting this is limited. We aimed to quantify the reported frequency and clinical sequelae of energy overfeeding with enterally delivered nutrition in critically ill adult patients. A systematic search of MEDLINE, EMBASE, and CINAHL from conception to November 28, 2018, identified clinical studies of nutrition interventions in enterally fed critically ill adults that reported overfeeding in 1 or more study arms. Overfeeding was defined as energy delivery > 2000 kcal/d, > 25 kcal/kg/d, or ≥ 110% of energy prescription. Data were extracted on methodology, demographics, prescribed and delivered nutrition, clinical variables, and predefined outcomes. Cochrane "Risk of Bias" tool was used to assess the quality of randomized controlled trials (RCTs). Eighteen studies were included, of which 10 were randomized (n = 4386 patients) and 8 were nonrandomized (n = 223). Only 4 studies reported a separation in energy delivery between treatment groups whereby 1 arm met the definition of overfeeding, which reported no between-group differences in mortality, infectious complications, or ventilatory support. Overfeeding was associated with increased insulin administration (median 3 [interquartile range: 0-41.8] vs 0 [0-30.6] units/d) and upper-gastrointestinal intolerance in 1 large RCT and with duration of antimicrobial therapy in a small RCT. There are limited high-quality data to determine the impact of energy overfeeding of critically ill patients by the enteral route; however, based on available evidence, overfeeding does not appear to affect mortality or other important clinical outcomes.