Assessing the benefit-risk of new treatments using generalised pairwise comparisons: the case of erlotinib in pancreatic cancer

Study design for restricted mean time analysis of recurrent events and death

The restricted mean time in favor (RMT-IF) of treatment has just been added to the analytic toolbox for composite endpoints of recurrent events and death. To help practitioners design new trials based on this method, we develop tools to calculate the sample size and power. Specifically, we formulate the outcomes as a multistate Markov process with a sequence of transient states for recurrent events and an absorbing state for death. The transition intensities, in this case the instantaneous risks of another nonfatal event or death, are assumed to be time-homogeneous but nonetheless allowed to depend on the number of past events. Using the properties of Coxian distributions, we derive the RMT-IF effect size under the alternative hypothesis as a function of the treatment-to-control intensity ratios along with the baseline intensities, the latter of which can be easily estimated from historical data. We also reduce the variance of the nonparametric RMT-IF estimator to calculable terms under a standard set-up for censoring. Simulation studies show that the resulting formulas provide accurate approximation to the sample size and power in realistic settings. For illustration, a past cardiovascular trial with recurrent-hospitalization and mortality outcomes is analyzed to generate the parameters needed to design a future trial. The procedures are incorporated into the rmt package along with the original methodology on the Comprehensive R Archive Network (CRAN).

Trial Design for Cancer Immunotherapy: A Methodological Toolkit

Immunotherapy with checkpoint inhibitors (CPIs) and cell-based products has revolutionized the treatment of various solid tumors and hematologic malignancies. These agents have shown unprecedented response rates and long-term benefits in various settings. These clinical advances have also pointed to the need for new or adapted approaches to trial design and assessment of efficacy and safety, both in the early and late phases of drug development. Some of the conventional statistical methods and endpoints used in other areas of oncology appear to be less appropriate in immuno-oncology. Conversely, other methods and endpoints have emerged as alternatives. In this article, we discuss issues related to trial design in the early and late phases of drug development in immuno-oncology, with a focus on CPIs. For early trials, we review the most salient issues related to dose escalation, use and limitations of tumor response and progression criteria for immunotherapy, the role of duration of response as an endpoint in and of itself, and the need to conduct randomized trials as early as possible in the development of new therapies. For late phases, we discuss the choice of primary endpoints for randomized trials, review the current status of surrogate endpoints, and discuss specific statistical issues related to immunotherapy, including non-proportional hazards in the assessment of time-to-event endpoints, alternatives to the Cox model in these settings, and the method of generalized pairwise comparisons, which can provide a patient-centric assessment of clinical benefit and be used to design randomized trials.

Composite endpoints, including patient reported outcomes, in rare diseases

BACKGROUND

When assessing the efficacy of a treatment in any clinical trial, it is recommended by the International Conference on Harmonisation to select a single meaningful endpoint. However, a single endpoint is often not sufficient to reflect the full clinical benefit of a treatment in multifaceted diseases, which is often the case in rare diseases. Therefore, the use of a combination of several clinically meaningful outcomes is preferred. Many methodologies that allow for combining outcomes in a so-called composite endpoint are however limited in a number of ways, not in the least in the number and type of outcomes that can be combined and in the poor small-sample properties. Moreover, patient reported outcomes, such as quality of life, often cannot be integrated in a composite analysis, in spite of their intrinsic value.

RESULTS

Recently, a class of non-parametric generalized pairwise comparisons tests have been proposed, which members do allow for any number and type of outcomes, including patient reported outcomes. The class enjoys good small-sample properties. Moreover, this very flexible class of methods allows for prioritizing the outcomes by clinical severity, allows for matched designs and for adding a threshold of clinical relevance. Our aim is to introduce the generalized pairwise comparison ideas and concepts for rare disease clinical trial analysis, and demonstrate their benefit in a post-hoc analysis of a small-sample trial in epidermolysis bullosa. More precisely, we will include a patient relevant outcome (Quality of life), in a composite endpoint. This publication is part of the European Joint Programme on Rare Diseases (EJP RD) series on innovative methodologies for rare diseases clinical trials, which is based on the webinars presented within the educational activity of EJP RD. This publication covers the webinar topic on composite endpoints in rare diseases and includes participants' response to a questionnaire on this topic.

CONCLUSIONS

Generalized pairwise comparisons is a promising statistical methodology for evaluating any type of composite endpoints in rare disease trials and may allow a better evaluation of therapy efficacy including patients reported outcomes in addition to outcomes related to the diseases signs and symptoms.

On restricted mean time in favor of treatment

The restricted mean time in favor (RMT-IF) of treatment is a nonparametric effect size for complex life history data. It is defined as the net average time the treated spend in a more favorable state than the untreated over a prespecified time window. It generalizes the familiar restricted mean survival time (RMST) from the two-state life-death model to account for intermediate stages in disease progression. The overall estimand can be additively decomposed into stage-wise effects, with the standard RMST as a component. Alternate expressions of the overall and stage-wise estimands as integrals of the marginal survival functions for a sequence of landmark transitioning events allow them to be easily estimated by plug-in Kaplan-Meier estimators. The dynamic profile of the estimated treatment effects as a function of follow-up time can be visualized using a multilayer, cone-shaped "bouquet plot." Simulation studies under realistic settings show that the RMT-IF meaningfully and accurately quantifies the treatment effect and outperforms traditional tests on time to the first event in statistical efficiency thanks to its fuller utilization of patient data. The new methods are illustrated on a colon cancer trial with relapse and death as outcomes and a cardiovascular trial with recurrent hospitalizations and death as outcomes. The R-package rmt implements the proposed methodology and is publicly available from the Comprehensive R Archive Network (CRAN).

Impact of correlations between prioritized outcomes on the net benefit and its estimate by generalized pairwise comparisons

Benefit-risk balance is gaining interest in clinical trials. For the comprehensive assessment of benefits and risks, generalized pairwise comparisons are increasingly used to estimate the net benefit based on multiple prioritized outcomes. Although previous research has demonstrated that the correlations between the outcomes impact the net benefit and its estimate, the direction and magnitude of this impact remain unclear. In this study, we investigated the impact of correlations between two binary or Gaussian variables on the true net benefit values via theoretical and numerical analyses. We also explored the impact of correlations between survival and categorical variables on the net benefit estimates based on four existing methods (Gehan, Péron, Gehan with correction, and Péron with correction) in the presence of right censoring via simulation and application to actual oncology clinical trial data. Our theoretical and numerical analyses revealed that the true net benefit values were impacted by the correlations in various directions depending on the outcome distributions. With binary endpoints, this direction was governed by a simple rule with a threshold of 50% for a favorable outcome. Our simulation showed that the net benefit estimates based on Gehan's or Péron's scoring rule could be substantially biased in the presence of right censoring, and that the direction and magnitude of this bias were associated with the outcome correlations. The recently proposed correction method greatly reduced this bias, even in the presence of strong outcome correlations. The impact of correlations should be carefully considered when interpreting the net benefit and its estimate.

Individualized Net Benefit estimation and meta-analysis using generalized pairwise comparisons in N-of-1 trials

BACKGROUND

The Net Benefit (Δ) is a measure of the benefit-risk balance in clinical trials, based on generalized pairwise comparisons (GPC) using several prioritized outcomes and thresholds of clinical relevance. We extended Δ to N-of-1 trials, with a focus on patient-level and population-level Δ.

METHODS

We developed a Δ estimator at the individual level as an extension of the stratum-specific Δ, and at the population-level as an extension of the stratified Δ. We performed a simulation study mimicking PROFIL, a series of 38 N-of-1 trials testing sildenafil in Raynaud's phenomenon, to assess the power for such an analysis with realistic data. We then reanalyzed PROFIL using GPC. This reanalysis was finally interpreted in the context of the main analysis of PROFIL which used Bayesian individual probabilities of efficacy.

RESULTS

Simulations under the null showed good size of the test for both individual and population levels. The test lacked power when being simulated from the true PROFIL data, even when increasing the number of repetitions up to 140 days per patient. PROFIL individual-level estimated Δ were well correlated with the probabilities of efficacy from the Bayesian analysis while showing similarly wide confidence intervals. Population-level estimated Δ was not significantly different from zero, consistently with the previous Bayesian analysis.

CONCLUSION

GPC can be used to estimate individual Δ which can then be aggregated in a meta-analytic way in N-of-1 trials. GPC ability to easily incorporate patient preferences allow for more personalized treatment evaluation, while needing much less computing time than Bayesian modeling.

Operational characteristics of generalized pairwise comparisons for hierarchically ordered endpoints

The method of generalized pairwise comparisons (GPC) is a multivariate extension of the well-known non-parametric Wilcoxon-Mann-Whitney test. It allows comparing two groups of observations based on multiple hierarchically ordered endpoints, regardless of the number or type of the latter. The summary measure, "net benefit," quantifies the difference between the probabilities that a random observation from one group is doing better than an observation from the opposite group. The method takes into account the correlations between the endpoints. We have performed a simulation study for the case of two hierarchical endpoints to evaluate the impact of their correlation on the type-I error probability and power of the test based on GPC. The simulations show that the power of the GPC test for the primary endpoint is modified if the secondary endpoint is included in the hierarchical GPC analysis. The change in power depends on the correlation between the endpoints. Interestingly, a decrease in power can occur, regardless of whether there is any marginal treatment effect on the secondary endpoint. It appears that the overall power of the hierarchical GPC procedure depends, in a complex manner, on the entire variance-covariance structure of the set of outcomes. Any additional factors (such as thresholds of clinical relevance, drop out, or censoring scheme) will also affect the power and will have to be taken into account when designing a trial based on the hierarchical GPC procedure.

The Net Benefit of a treatment should take the correlation between benefits and harms into account

OBJECTIVE

The assessment of benefits and harms from experimental treatments often ignores the association between outcomes. In a randomized trial, generalized pairwise comparisons (GPC) can be used to assess a Net Benefit that takes this association into account.

STUDY DESIGN AND SETTINGS

We use GPC to analyze a fictitious trial of treatment versus control, with a binary efficacy outcome (response) and a binary toxicity outcome, as well as data from two actual randomized trials in oncology. In all cases, we compute the Net Benefit for scenarios with different orders of priority between response and toxicity, and a range of odds ratios (ORs) for the association between these outcomes.

RESULTS

The GPC Net Benefit was quite different from the benefit/harm computed using marginal treatment effects on response and toxicity. In the fictitious trial using response as first priority, treatment had an unfavorable Net Benefit if OR < 1, but favorable if OR > 1. With OR = 1, the Net Benefit was 0. Results changed drastically using toxicity as first priority.

CONCLUSION

Even in a simple situation, marginal treatment effects can be misleading. In contrast, GPC assesses the Net Benefit as a function of the treatment effects on each outcome, the association between outcomes, and individual patient priorities.

Unbiasedness and efficiency of non-parametric and UMVUE estimators of the probabilistic index and related statistics

In reliability theory, diagnostic accuracy, and clinical trials, the quantity P(X>Y)+1/2P(X=Y), also known as the Probabilistic Index (PI), is a common treatment effect measure when comparing two groups of observations. The quantity P(X>Y)-P(Y>X), a linear transformation of PI known as the net benefit, has also been advocated as an intuitively appealing treatment effect measure. Parametric estimation of PI has received a lot of attention in the past 40 years, with the formulation of the Uniformly Minimum-Variance Unbiased Estimator (UMVUE) for many distributions. However, the non-parametric Mann-Whitney estimator of the PI is also known to be UMVUE in some situations. To understand this seeming contradiction, in this paper a systematic comparison is performed between the non-parametric estimator for the PI and parametric UMVUE estimators in various settings. We show that the Mann-Whitney estimator is always an unbiased estimator of the PI with univariate, completely observed data, while the parametric UMVUE is not when the distribution is misspecified. Additionally, the Mann-Whitney estimator is the UMVUE when observations belong to an unrestricted family. When observations come from a more restrictive family of distributions, the loss in efficiency for the non-parametric estimator is limited in realistic clinical scenarios. In conclusion, the Mann-Whitney estimator is simple to use and is a reliable estimator for the PI and net benefit in realistic clinical scenarios.

Net benefit in the presence of correlated prioritized outcomes using generalized pairwise comparisons: A simulation study

BACKGROUND

The prioritized net benefit (Δ) is a measure of the benefit-risk balance in clinical trials, based on generalized pairwise comparisons (GPC) using several prioritized outcomes. Its estimation requires the classification as Wins or Losses of all possible pairs of patients, one from the experimental treatment (E) group and one from the control treatment (C) group. In this simulation study, we assessed the impact of the correlation between prioritized outcomes on Δ, its estimate, bias, size, and power.

METHODS

The theoretical Δ value was derived for the specific case of two correlated binary outcomes when a normal copula is used. Focusing on one efficacy and one toxicity outcome, two situations frequently met in practice were simulated: binary efficacy outcome with binary toxicity outcome, or time to event efficacy outcome with categorical toxicity outcome. Several scenarios of efficacy and toxicity were generated, with various levels of correlation.

RESULTS

When E was more effective than C, positive correlations were mainly associated with a decrease in the proportion of Losses, while negative correlations were associated with a decrease in the proportion of Wins on the toxicity outcome. This resulted in an increase of Δ ^ with the intensity of the positive correlation without adding any bias. Results were similar whatever the type of outcomes generated but led to power alteration.

CONCLUSION

Correlations between outcomes analyzed with GPC led to substantial but predictable modifications of Δ and its estimate. Correlations should be taken into consideration when performing sample size estimations in clinical trials.

Methodologies for pragmatic and efficient assessment of benefits and harms: Application to the SOCRATES trial

BACKGROUND/AIMS

Standard approaches to trial design and analyses can be inefficient and non-pragmatic. Failure to consider a range of outcomes impedes evidence-based interpretation and reduces power. Traditional approaches synthesizing information obtained from separate analysis of each outcome fail to incorporate associations between outcomes and recognize the cumulative nature of outcomes in individual patients, suffer from competing risk complexities during interpretation, and since efficacy and safety analyses are often conducted on different populations, generalizability is unclear. Pragmatic and efficient approaches to trial design and analyses are needed.

METHODS

Approaches providing a pragmatic assessment of benefits and harms of interventions, summarizing outcomes experienced by patients, and providing sample size efficiencies are described. Ordinal outcomes recognize finer gradations of patient responses. Desirability of outcome ranking is an ordinal outcome combining benefits and harms within patients. Analysis of desirability of outcome ranking can be based on rank-based methodologies including the desirability of outcome ranking probability, the win ratio, and the proportion in favor of treatment. Partial credit analyses, involving grading the levels of the desirability of outcome ranking outcome similar to an academic test, provides an alternative approach. The methodologies are demonstrated using the acute stroke or transient ischemic attack treated with aspirin or ticagrelor and patient outcomes study (SOCRATES; NCT01994720), a randomized clinical trial.

RESULTS

Two 5-level ordinal outcomes were developed for SOCRATES. The first was based on a modified Rankin scale. The odds ratio is 0.86 (95% confidence interval = 0.75, 0.99; p = 0.04) indicating that the odds of worse stroke categorization for a trial participant assigned to ticagrelor is 0.86 times that of a trial participant assigned to aspirin. The 5-level desirability of outcome ranking outcome incorporated and prioritized survival; the number of strokes, myocardial infarction, and major bleeding events; and whether a stroke event was disabling. The desirability of outcome ranking probability and win ratio are 0.504 (95% confidence interval = 0.499, 0.508; p = 0.10) and 1.11 (95% confidence interval = 0.98, 1.26; p = 0.10), respectively, implying that the probability of a more desirable result with ticagrelor is 50.4% and that a more desirable result occurs 1.11 times more frequently on ticagrelor versus aspirin.

CONCLUSION

Ordinal outcomes can improve efficiency through required pre-specification, careful construction, and analyses. Greater pragmatism can be obtained by composing outcomes within patients. Desirability of outcome ranking provides a global assessment of the benefits and harms that more closely reflect the experience of patients. The desirability of outcome ranking probability, the proportion in favor of treatment, the win ratio, and partial credit can more optimally inform patient treatment, enhance the understanding of the totality of intervention effects on patients, and potentially provide efficiencies over standard analyses. The methods provide the infrastructure for incorporating patient values and estimating personalized effects.

The Benefit-Risk Balance of Nab-Paclitaxel in Metastatic Pancreatic Adenocarcinoma

OBJECTIVES

The benefit-risk balance of 5-fluorouracil, oxaliplatin, irinotecan, and leucovorin versus gemcitabine assessed using generalized pairwise comparison was strongly positive. We sought to assess the benefit-risk balance of nab-paclitaxel plus gemcitabine using the data of the MPACT trial, as it is an alternative to 5-fluorouracil, oxaliplatin, irinotecan, and leucovorin.

METHODS

This statistical method allows for the simultaneous analysis of several prioritized outcomes. The first priority outcome was survival time (overall survival). The second priority outcome was toxicity. The overall treatment effect was quantified using the overall net benefit. Multiple sensitivity analyses were performed to assess the consistency of the results according to possible patients' preferences.

RESULTS

In this trial, 861 patients received nab-paclitaxel plus gemcitabine or gemcitabine alone. The overall net benefit favored strongly and significantly the combination group. When only large survival differences were considered clinically relevant, the net benefit was not in favor of the combination group.

CONCLUSIONS

The overall net benefit is a clinically intuitive way of comparing patients with respect to all important efficacy and toxicity outcomes. The nab-paclitaxel plus gemcitabine combination has a positive benefit-risk balance, but it might not be suitable for patients who would consider losing several months of survival to avoid a significant toxic event.

An assessment of the benefit-risk balance of FOLFIRINOX in metastatic pancreatic adenocarcinoma

Background

We sought to assess the benefit-risk balance of FOLFIRINOX versus gemcitabine in patients with metastatic pancreatic adenocarcinoma.

Methods

We used generalized pairwise comparisons. This statistical method permits the simultaneous analysis of several prioritized outcome measures. The first priority outcome was survival time (OS). Differences in OS that exceeded two months were considered clinically relevant. The second priority outcome was toxicity. The overall treatment effect was quantified using the net chance of a better outcome, which can be interpreted as the net probability for a random patient treated in the FOLFIRINOX group to have a better overall outcome than a random patient in the gemcitabine group.

Results

In this trial 342 patients received either FOLFIRINOX or gemcitabine. The net chance of a better outcome favored strongly and significantly the FOLFIRINOX group (24.7; P<.001), suggesting a favorable benefit-risk balance of FOLFIRINOX versus gemcitabine. The positive benefit-risk balance of FOLFIRINOX was observed throughout all sensitivity analyses.

Conclusions

Generalized pairwise comparisons are useful to perform a quantitative assessment of the benefit-risk balance of new treatments. It provides a clinically intuitive way of comparing patients with respect to all important efficacy and toxicity outcomes. Overall the benefit-risk balance of FOLFIRINOX was strongly positive.

An extension of generalized pairwise comparisons for prioritized outcomes in the presence of censoring

Generalized pairwise comparisons have been proposed to permit a comprehensive assessment of several prioritized outcomes between two groups of observations. This procedure estimates Δ, the net chance of a better outcome with treatment than with control by comparing the patients outcomes among all possible pairs taking one patient from the treatment group and one patient from the control group. For time to event outcomes, the standard procedure of generalized pairwise comparisons is analogous to the Gehan’s modification of the Mann-Whitney test which is biased in presence of censored observation and less powerful than Efron’s modification of this test. We adapt Efron’s modification to generalized pairwise comparisons. We show how a pairwise contribution to Δ can be calculated from the estimates of the survival function in the presence of right-censored data. We performed a simulation study to assess the bias, the type I error and the power of the new procedure. The estimate of Δ with the new procedure is only slightly biased even in presence of heavy censoring. We also show how this bias can be corrected when only one time-to-event outcome is analyzed. The new procedure has higher power in most cases compared to the standard procedure.