Parametric G-computation for compatible indirect treatment comparisons with limited individual patient data

Advancing unanchored simulated treatment comparisons: A novel implementation and simulation study

Population-adjusted indirect comparisons, developed in the 2010s, enable comparisons between two treatments in different studies by balancing patient characteristics in the case where individual patient-level data (IPD) are available for only one study. Health technology assessment (HTA) bodies increasingly rely on these methods to inform funding decisions, typically using unanchored indirect comparisons (i.e., without a common comparator), due to the need to evaluate comparative efficacy and safety for single-arm trials. Unanchored matching-adjusted indirect comparison (MAIC) and unanchored simulated treatment comparison (STC) are currently the only two approaches available for population-adjusted indirect comparisons based on single-arm trials. However, there is a notable underutilisation of unanchored STC in HTA, largely due to a lack of understanding of its implementation. We therefore develop a novel way to implement unanchored STC by incorporating standardisation/marginalisation and the NORmal To Anything (NORTA) algorithm for sampling covariates. This methodology aims to derive a suitable marginal treatment effect without aggregation bias for HTA evaluations. We use a non-parametric bootstrap and propose separately calculating the standard error for the IPD study and the comparator study to ensure the appropriate quantification of the uncertainty associated with the estimated treatment effect. The performance of our proposed unanchored STC approach is evaluated through a comprehensive simulation study focused on binary outcomes. Our findings demonstrate that the proposed approach is asymptotically unbiased. We argue that unanchored STC should be considered when conducting unanchored indirect comparisons with single-arm studies, presenting a robust approach for HTA decision-making.

A Targeted Review of Worldwide Indirect Treatment Comparison Guidelines and Best Practices

OBJECTIVES

Controls and governance over the methodology and reporting of indirect treatment comparisons (ITCs) have been introduced to minimize bias and ensure scientific credibility and transparency in healthcare decision making. The objective of this study was to highlight ITC techniques that are key to conducting objective and analytically sound analyses and to ascertain circumstantial suitability of ITCs as a source of comparative evidence for healthcare interventions.

METHODS

Ovid MEDLINE was searched from January 2010 through August 2023 to identify publicly available ITC-related documents (ie, guidelines and best practices) in the English language. This was supplemented with hand searches of websites of various international organizations, regulatory agencies, and reimbursement agencies of Europe, North America, and Asia-Pacific. The jurisdiction-specific ITC methodology and reporting recommendations were reviewed.

RESULTS

Sixty-eight guidelines from 10 authorities worldwide were included for synthesis. Many of the included guidelines were updated within the last 5 years and commonly cited the absence of direct comparative studies as primary justification for using ITCs. Most jurisdictions favored population-adjusted or anchored ITC techniques opposed to naive comparisons. Recommendations on the reporting and presentation of these ITCs varied across authorities; however, there was some overlap among the key elements.

CONCLUSIONS

Given the challenges of conducting head-to-head randomized controlled trials, comparative data from ITCs offer valuable insights into clinical-effectiveness. As such, multiple ITC guidelines have emerged worldwide. According to the most recent versions of the guidelines, the suitability and subsequent acceptability of the ITC technique used depends on the data sources, available evidence, and magnitude of benefit/uncertainty.

Unanchored Population-Adjusted Indirect Comparison Methods for Time-to-Event Outcomes Using Inverse Odds Weighting, Regression Adjustment, and Doubly Robust Methods With Either Individual Patient or Aggregate Data

OBJECTIVES

Several methods for unanchored population-adjusted indirect comparisons (PAICs) are available. Exploring alternative adjustment methods, depending on the available individual patient data (IPD) and the aggregate data (AD) in the external study, may help minimize bias in unanchored indirect comparisons. However, methods for time-to-event outcomes are not well understood. This study provides an overview and comparison of methods using a case study to increase familiarity. A recent method is applied to marginalize conditional hazard ratios, which allows for the comparisons of methods, and a doubly robust method is proposed.

METHODS

The following PAIC methods were compared through a case study in third-line small cell lung cancer, comparing nivolumab with standard of care based on a single-arm phase II trial (CheckMate 032) and real-world study (Flatiron) in terms of overall survival: IPD-IPD analyses using inverse odds weighting, regression adjustment, and a doubly robust method; IPD-AD analyses using matching-adjusted indirect comparison, simulated treatment comparison, and a doubly robust method.

RESULTS

Nivolumab extended survival versus standard of care with hazard ratios ranging from 0.63 (95% CI 0.44-0.90) in naive comparisons (identical estimates for IPD-IPD and IPD-AD analyses) to 0.69 (95% CI 0.44-0.98) in the IPD-IPD analyses using regression adjustment. Regression-based and doubly robust estimates yielded slightly wider confidence intervals versus the propensity score-based analyses.

CONCLUSIONS

The proposed doubly robust approach for time-to-event outcomes may help to minimize bias due to model misspecification. However, all methods for unanchored PAIC rely on the strong assumption that all prognostic covariates have been included.

Model-based standardization using multiple imputation

BACKGROUND

When studying the association between treatment and a clinical outcome, a parametric multivariable model of the conditional outcome expectation is often used to adjust for covariates. The treatment coefficient of the outcome model targets a conditional treatment effect. Model-based standardization is typically applied to average the model predictions over the target covariate distribution, and generate a covariate-adjusted estimate of the marginal treatment effect.

METHODS

The standard approach to model-based standardization involves maximum-likelihood estimation and use of the non-parametric bootstrap. We introduce a novel, general-purpose, model-based standardization method based on multiple imputation that is easily applicable when the outcome model is a generalized linear model. We term our proposed approach multiple imputation marginalization (MIM). MIM consists of two main stages: the generation of synthetic datasets and their analysis. MIM accommodates a Bayesian statistical framework, which naturally allows for the principled propagation of uncertainty, integrates the analysis into a probabilistic framework, and allows for the incorporation of prior evidence.

RESULTS

We conduct a simulation study to benchmark the finite-sample performance of MIM in conjunction with a parametric outcome model. The simulations provide proof-of-principle in scenarios with binary outcomes, continuous-valued covariates, a logistic outcome model and the marginal log odds ratio as the target effect measure. When parametric modeling assumptions hold, MIM yields unbiased estimation in the target covariate distribution, valid coverage rates, and similar precision and efficiency than the standard approach to model-based standardization.

CONCLUSION

We demonstrate that multiple imputation can be used to marginalize over a target covariate distribution, providing appropriate inference with a correctly specified parametric outcome model and offering statistical performance comparable to that of the standard approach to model-based standardization.

Twenty years of network meta-analysis: Continuing controversies and recent developments

Network meta-analysis (NMA) is an extension of pairwise meta-analysis (PMA) which combines evidence from trials on multiple treatments in connected networks. NMA delivers internally consistent estimates of relative treatment efficacy, needed for rational decision making. Over its first 20 years NMA's use has grown exponentially, with applications in both health technology assessment (HTA), primarily re-imbursement decisions and clinical guideline development, and clinical research publications. This has been a period of transition in meta-analysis, first from its roots in educational and social psychology, where large heterogeneous datasets could be explored to find effect modifiers, to smaller pairwise meta-analyses in clinical medicine on average with less than six studies. This has been followed by narrowly-focused estimation of the effects of specific treatments at specific doses in specific populations in sparse networks, where direct comparisons are unavailable or informed by only one or two studies. NMA is a powerful and well-established technique but, in spite of the exponential increase in applications, doubts about the reliability and validity of NMA persist. Here we outline the continuing controversies, and review some recent developments. We suggest that heterogeneity should be minimized, as it poses a threat to the reliability of NMA which has not been fully appreciated, perhaps because it has not been seen as a problem in PMA. More research is needed on the extent of heterogeneity and inconsistency in datasets used for decision making, on formal methods for making recommendations based on NMA, and on the further development of multi-level network meta-regression.

Indirect treatment comparison of brexucabtagene autoleucel (ZUMA-2) versus standard of care (SCHOLAR-2) in relapsed/refractory mantle cell lymphoma

The SCHOLAR-2 retrospective study highlighted poor overall survival (OS) with standard of care (SOC) regimens among patients with relapsed/refractory (R/R) mantle cell lymphoma (MCL) who failed a covalent Bruton tyrosine kinase inhibitor (BTKi). In the ZUMA-2 single-arm trial, brexucabtagene autoleucel (brexu-cel; autologous anti-CD19 CAR T-cell therapy) demonstrated high rates of durable responses in patients with R/R MCL who had previous BTKi exposure. Here, we compared OS in ZUMA-2 and SCHOLAR-2 using three different methods which adjusted for imbalances in prognostic factors between populations: inverse probability weighting (IPW), regression adjustment (RA), and doubly robust (DR). Brexu-cel was associated with improved OS compared to SOC across all unadjusted and adjusted comparisons. Hazard ratios (95% confidence intervals) were 0.38 (0.23, 0.61) for IPW, 0.45 (0.28, 0.74) for RA, and 0.37 (0.23, 0.59) for DR. These results suggest a substantial survival benefit with brexu-cel versus SOC in patients with R/R MCL after BTKi exposure.

A Comparison of Relative-Efficacy Estimate(S) Derived From Both Matching-Adjusted Indirect Comparisons and Standard Anchored Indirect Treatment Comparisons: A Review of Matching-Adjusted Indirect Comparisons

OBJECTIVES

We present an empirical comparison of relative-efficacy estimate(s) from matching-adjusted indirect comparisons (MAICs) with estimates from corresponding standard anchored indirect treatment comparisons.

METHODS

A total of 80 comparisons were identified from 17 publications through a systematic rapid review. A standardized metric that used reported relative treatment efficacy estimates and their associated uncertainty was used to compare the methods across different treatment indications and outcome measures.

RESULTS

On aggregate, MAICs presented for connected networks tended to report a more favorable relative-efficacy estimate for the treatment for which individual-level patient data were available relative to the reported indirect treatment comparison estimate.

CONCLUSIONS

Although we recognize the importance of MAIC and other population adjustment methods in certain situations, we recommend that results from these analyses are interpreted with caution. Researchers and analysts should carefully consider if MAICs are appropriate where presented and whether MAICs would have added value where omitted.

Population adjusted-indirect comparisons in health technology assessment: A methodological systematic review

In health technology assessment (HTA), population-adjusted indirect comparisons (PAICs) are increasingly considered to adjust for the difference in the target population between studies. We aim to assess the conduct and reporting of PAICs in recent HTA practice, by performing, a methodological systematic review of studies implementing PAICs from PubMed, EMBASE Classic, Embase/Ovid Medline All, and Cochrane databases from January 1, 2010 to Feb 13, 2023. Four independent researchers screened the titles, abstracts, and full-texts of the identified records, then extracted data on methodological and reporting characteristics of 106 eligible articles. Most PAIC analyses (96.9%, n = 157) were conducted by (or received funding from) pharmaceutical companies. Prior to adjustment, 44.5% of analyses (n = 72) (partially) aligned the eligibility criteria of different studies to enhance the similarity of their target populations. In 37.0% of analyses (n = 60), the clinical and methodological heterogeneity across studies were extensively assessed. In 9.3% of analyses (n = 15), the quality (or bias) of individual studies was evaluated. Among 18 analyses using methods that required an outcome model specification, results of the model fitting procedure were adequately reported in three analyses (16.7%). These findings suggest that the conduct and reporting of PAICs are remarkably heterogeneous and suboptimal in current practice. More recommendations and guidelines on PAICs are thus warranted to enhance the quality of these analyses in the future.

Belimumab versus anifrolumab in adults with systemic lupus erythematosus: an indirect comparison of clinical response at 52 weeks

OBJECTIVE

To generate comparative efficacy evidence of belimumab versus anifrolumab in SLE that can inform treatment practices.

METHODS

The SLE Responder Index (SRI)-4 response at 52 weeks of belimumab versus anifrolumab was evaluated with an indirect treatment comparison. The evidence base consisted of randomised trials that were compiled through a systemic literature review.A feasibility assessment was performed to comprehensively compare the eligible trials and to determine the most appropriate indirect treatment comparison analysis method. A multilevel network meta-regression (ML-NMR) was implemented that adjusted for differences across trials in four baseline characteristics: SLE Disease Activity Index-2K, anti-double-stranded DNA antibody positive, low complement (C)3 and low C4. Additional analyses were conducted to explore if the results were robust to different sets of baseline characteristics included for adjustment, alternative adjustment methods and changes to the trials included in the evidence base.

RESULTS

The ML-NMR included eight trials: five belimumab trials (BLISS-52, BLISS-76, NEA, BLISS-SC, EMBRACE) and three anifrolumab trials (MUSE, TULIP-1, TULIP-2). Belimumab and anifrolumab were comparable in terms of SRI-4 response (OR (95% credible interval), 1.04 (0.74-1.45)), with the direction of the point estimate slightly favouring belimumab. Belimumab had a 0.58 probability of being the more effective treatment. The results were highly consistent across all analysis scenarios.

CONCLUSIONS

Our results suggest that the SRI-4 response of belimumab and anifrolumab are similar at 52 weeks in the general SLE population, but the level of uncertainty around the point estimate means we cannot rule out the possibility of a clinically meaningful benefit for either treatment. It remains to be seen if specific groups of patients could derive a greater benefit from anifrolumab or from belimumab, and there is certainly an unmet need to identify robust predictors towards more personalised selection of available biological agents in SLE.

A cautionary note on the use of G-computation in population adjustment

In a recent issue of the Journal; Remiro-Azócar et al. introduce a new method to adjust for population difference between two trials; when the individual patient data (IPD) are only accessible for one study. The proposed method generates the covariate data for the trial without IPD; then using a G-computation approach to transport information about the treatment effect from the other study with IPD to this trial. The authors advocate the use of G-computation over matching-adjusted indirect comparison because (i) the former allows for "useful extrapolation" when there is poor case-mix overlap between populations; and (ii) nonparametric; data-adaptive methods can be used to reduce the risk of (outcome) model misspecification. In this commentary; we provide a different perspective from these arguments. Despite certain disagreements; we believe that the proposed data generation approaches can open new and interesting research directions for population adjustment methodology in the future.

Two-stage matching-adjusted indirect comparison

BACKGROUND

Anchored covariate-adjusted indirect comparisons inform reimbursement decisions where there are no head-to-head trials between the treatments of interest, there is a common comparator arm shared by the studies, and there are patient-level data limitations. Matching-adjusted indirect comparison (MAIC), based on propensity score weighting, is the most widely used covariate-adjusted indirect comparison method in health technology assessment. MAIC has poor precision and is inefficient when the effective sample size after weighting is small.

METHODS

A modular extension to MAIC, termed two-stage matching-adjusted indirect comparison (2SMAIC), is proposed. This uses two parametric models. One estimates the treatment assignment mechanism in the study with individual patient data (IPD), the other estimates the trial assignment mechanism. The first model produces inverse probability weights that are combined with the odds weights produced by the second model. The resulting weights seek to balance covariates between treatment arms and across studies. A simulation study provides proof-of-principle in an indirect comparison performed across two randomized trials. Nevertheless, 2SMAIC can be applied in situations where the IPD trial is observational, by including potential confounders in the treatment assignment model. The simulation study also explores the use of weight truncation in combination with MAIC for the first time.

RESULTS

Despite enforcing randomization and knowing the true treatment assignment mechanism in the IPD trial, 2SMAIC yields improved precision and efficiency with respect to MAIC in all scenarios, while maintaining similarly low levels of bias. The two-stage approach is effective when sample sizes in the IPD trial are low, as it controls for chance imbalances in prognostic baseline covariates between study arms. It is not as effective when overlap between the trials' target populations is poor and the extremity of the weights is high. In these scenarios, truncation leads to substantial precision and efficiency gains but induces considerable bias. The combination of a two-stage approach with truncation produces the highest precision and efficiency improvements.

CONCLUSIONS

Two-stage approaches to MAIC can increase precision and efficiency with respect to the standard approach by adjusting for empirical imbalances in prognostic covariates in the IPD trial. Further modules could be incorporated for additional variance reduction or to account for missingness and non-compliance in the IPD trial.