Classifying information-sharing methods

Problematic meta-analyses: Bayesian and frequentist perspectives on combining randomized controlled trials and non-randomized studies

PURPOSE

In the literature, the propriety of the meta-analytic treatment-effect produced by combining randomized controlled trials (RCT) and non-randomized studies (NRS) is questioned, given the inherent confounding in NRS that may bias the meta-analysis. The current study compared an implicitly principled pooled Bayesian meta-analytic treatment-effect with that of frequentist pooling of RCT and NRS to determine how well each approach handled the NRS bias.

MATERIALS & METHODS

Binary outcome Critical-Care meta-analyses, reflecting the importance of such outcomes in Critical-Care practice, combining RCT and NRS were identified electronically. Bayesian pooled treatment-effect and 95% credible-intervals (BCrI), posterior model probabilities indicating model plausibility and Bayes-factors (BF) were estimated using an informative heavy-tailed heterogeneity prior (half-Cauchy). Preference for pooling of RCT and NRS was indicated for Bayes-factors > 3 or < 0.333 for the converse. All pooled frequentist treatment-effects and 95% confidence intervals (FCI) were re-estimated using the popular DerSimonian-Laird (DSL) random effects model.

RESULTS

Fifty meta-analyses were identified (2009-2021), reporting pooled estimates in 44; 29 were pharmaceutical-therapeutic and 21 were non-pharmaceutical therapeutic. Re-computed pooled DSL FCI excluded the null (OR or RR = 1) in 86% (43/50). In 18 meta-analyses there was an agreement between FCI and BCrI in excluding the null. In 23 meta-analyses where FCI excluded the null, BCrI embraced the null. BF supported a pooled model in 27 meta-analyses and separate models in 4. The highest density of the posterior model probabilities for 0.333 < Bayes factor < 1 was 0.8.

CONCLUSIONS

In the current meta-analytic cohort, an integrated and multifaceted Bayesian approach gave support to including NRS in a pooled-estimate model. Conversely, caution should attend the reporting of naïve frequentist pooled, RCT and NRS, meta-analytic treatment effects.

Twenty years of network meta-analysis: Continuing controversies and recent developments

Network meta-analysis (NMA) is an extension of pairwise meta-analysis (PMA) which combines evidence from trials on multiple treatments in connected networks. NMA delivers internally consistent estimates of relative treatment efficacy, needed for rational decision making. Over its first 20 years NMA's use has grown exponentially, with applications in both health technology assessment (HTA), primarily re-imbursement decisions and clinical guideline development, and clinical research publications. This has been a period of transition in meta-analysis, first from its roots in educational and social psychology, where large heterogeneous datasets could be explored to find effect modifiers, to smaller pairwise meta-analyses in clinical medicine on average with less than six studies. This has been followed by narrowly-focused estimation of the effects of specific treatments at specific doses in specific populations in sparse networks, where direct comparisons are unavailable or informed by only one or two studies. NMA is a powerful and well-established technique but, in spite of the exponential increase in applications, doubts about the reliability and validity of NMA persist. Here we outline the continuing controversies, and review some recent developments. We suggest that heterogeneity should be minimized, as it poses a threat to the reliability of NMA which has not been fully appreciated, perhaps because it has not been seen as a problem in PMA. More research is needed on the extent of heterogeneity and inconsistency in datasets used for decision making, on formal methods for making recommendations based on NMA, and on the further development of multi-level network meta-regression.

Incorporating Prior Beliefs Into Meta-Analyses of Health-State Utility Values Using the Bayesian Power Prior

OBJECTIVES

Health-state utility values (HSUVs) directly affect estimates of Quality-Adjusted Life-Years and thus the cost-utility estimates. In practice a single preferred value (SPV) is often selected for HSUVs, despite meta-analysis being an option when multiple (credible) HSUVs are available. Nevertheless, the SPV approach is often reasonable because meta-analysis implicitly considers all HSUVs as equally relevant. This article presents a method for the incorporation of weights to HSUV synthesis, allowing more relevant studies to have greater influence.

METHODS

Using 4 case studies in lung cancer, hemodialysis, compensated liver cirrhosis, and diabetic retinopathy blindness, a Bayesian Power Prior (BPP) approach is used to incorporate beliefs on study applicability, reflecting the authors' perceived suitability for UK decision making. Older studies, non-UK value sets, and vignette studies are thus downweighted (but not disregarded). BPP HSUV estimates were compared with a SPV, random effects meta-analysis, and fixed effects meta-analysis. Sensitivity analyses were conducted iteratively updating the case studies, using alternative weighting methods, and simulated data.

RESULTS

Across all case studies, SPVs did not accord with meta-analyzed values, and fixed effects meta-analysis produced unrealistically narrow CIs. Point estimates from random effects meta-analysis and BPP models were similar in the final models, although BPP reflected additional uncertainty as wider credible intervals, particularly when fewer studies were available. Differences in point estimates were seen in iterative updating, weighting approaches, and simulated data.

CONCLUSIONS

The concept of the BPP can be adapted for synthesizing HSUVs, incorporating expert opinion on relevance. Because of the downweighting of studies, the BPP reflected structural uncertainty as wider credible intervals, with all forms of synthesis showing meaningful differences compared with SPVs. These differences would have implications for both cost-utility point estimates and probabilistic analyses.

A Guide to Selecting Flexible Survival Models to Inform Economic Evaluations of Cancer Immunotherapies

OBJECTIVES

Parametric models are routinely used to estimate the benefit of cancer drugs beyond trial follow-up. The advent of immune checkpoint inhibitors has challenged this paradigm, and emerging evidence suggests that more flexible survival models, which can better capture the shapes of complex hazard functions, might be needed for these interventions. Nevertheless, there is a need for an algorithm to help analysts decide whether flexible models are required and, if so, which should be chosen for testing. This position article has been produced to bridge this gap.

METHODS

A virtual advisory board comprising 7 international experts with in-depth knowledge of survival analysis and health technology assessment was held in summer 2021. The experts discussed 24 questions across 6 topics: the current survival model selection procedure, data maturity, heterogeneity of treatment effect, cure and mortality, external evidence, and additions to existing guidelines. Their responses culminated in an algorithm to inform selection of flexible survival models.

RESULTS

The algorithm consists of 8 steps and 4 questions. Key elements include the systematic identification of relevant external data, using clinical expert input at multiple points in the selection process, considering the future and the observed hazard functions, assessing the potential for long-term survivorship, and presenting results from all plausible models.

CONCLUSIONS

This algorithm provides a systematic, evidence-based approach to justify the selection of survival extrapolation models for cancer immunotherapies. If followed, it should reduce the risk of selecting inappropriate models, partially addressing a key area of uncertainty in the economic evaluation of these agents.

Structured expert elicitation to inform long-term survival extrapolations using alternative parametric distributions: a case study of CAR T therapy for relapsed/ refractory multiple myeloma

Background

Our aim was to extend traditional parametric models used to extrapolate survival in cost-effectiveness analyses (CEAs) by integrating individual-level patient data (IPD) from a clinical trial with estimates from experts regarding long-term survival. This was illustrated using a case study evaluating survival of patients with triple-class exposed relapsed/refractory multiple myeloma treated with the chimeric antigen receptor (CAR) T cell therapy idecabtagene vicleucel (ide-cel, bb2121) in KarMMa (a phase 2, single-arm trial).

Methods

The distribution of patients expected to be alive at 3, 5, and 10 years given the observed survival from KarMMa (13.3 months of follow-up) was elicited from 6 experts using the SHeffield ELicitation Framework. Quantities of interest were elicited from each expert individually, which informed the consensus elicitation including all experts. Estimates for each time point were assumed to follow a truncated normal distribution. These distributions were incorporated into survival models, which constrained the expected survival based on standard survival distributions informed by IPD from KarMMa.

Results

Models for ide-cel that combined KarMMa data with expert opinion were more consistent in terms of survival as well as mean survival at 10 years (survival point estimates under different parametric models were 29–33% at 3 years, 5–17% at 5 years, and 0–6% at 10 years) versus models with KarMMa data alone (11–39% at 3 years, 0–25% at 5 years, and 0–11% at 10 years).

Conclusion

This case study demonstrates a transparent approach to integrate IPD from trials with expert opinion using traditional parametric distributions to ensure long-term survival extrapolations are clinically plausible.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-022-01745-z.

Challenges around quantifying uncertainty in a holistic approach to hard-to-heal wound management: Health economic perspective

Treatment of hard-to-heal wounds involves a holistic approach for choosing between available treatment options. However, evidence for informing these choices is sparse, introducing uncertainty into decisions about the optimum treatment pathways that reflect the vast heterogeneity in this patient population. This paper discusses the existing clinical and health economic literature in order to provide insight into sources of uncertainty in the evaluation of the holistic approach to management of the hard-to-heal wounds, and how this uncertainty can be appropriately reflected in research. We identified three key sources of uncertainty in the evaluation of chronic wound treatments, namely heterogeneity in aetiology and patient populations, heterogeneity in treatment pathways, and challenges around capturing all relevant outcomes. Reflecting these complexities requires sophisticated modelling of treatment sequencing and long-term outcomes. The paper discusses how the scope specification, scenario analyses, and sensitivity analyses can be used to fully characterise analytical uncertainty.