Weighted win loss approach for analyzing prioritized outcomes

Application of the Win Ratio Method in the ENGAGE AF-TIMI 48 Trial Comparing Edoxaban With Warfarin in Patients With Atrial Fibrillation

BACKGROUND

Cardiovascular trials often use a composite end point and a time-to-first event model. We sought to compare edoxaban versus warfarin using the win ratio, which offers data complementary to time-to-first event analysis, emphasizing the most severe clinical events.

METHODS

ENGAGE AF-TIMI 48 (Effective Anticoagulation With Factor Xa Next Generation in Atrial Fibrillation-Thrombolysis in Myocardial Infarction 48) was a double-blind, randomized trial in which patients with atrial fibrillation were assigned 1:1:1 to a higher dose edoxaban regimen (60/30 mg daily), a lower dose edoxaban regimen (30/15 mg daily), or warfarin. In an exploratory analysis, we analyzed the trial outcomes using an unmatched win ratio approach. The win ratio for each edoxaban regimen was the total number of edoxaban wins divided by the number of warfarin wins for the following ranked clinical outcomes: 1: death; 2: hemorrhagic stroke; 3: ischemic stroke/systemic embolic event/epidural or subdural bleeding; 4: noncerebral International Society on Thrombosis and Haemostasis major bleeding; and 5: cardiovascular hospitalization.

RESULTS

21 105 patients were randomized to higher dose edoxaban regimen (N=7035), lower dose edoxaban regimen (N=7034), or warfarin (N=7046), yielding >49 million pairs for each treatment comparison. The median age was 72 years, 38% were women, and 59% had prior vitamin K antagonist use. The win ratio was 1.11 (95% CI, 1.05-1.18) for higher dose edoxaban regimen versus warfarin and 1.11 (95% CI, 1.05-1.18) for lower dose edoxaban regimen versus warfarin. The favorable impacts of edoxaban on death (34% of wins) and cardiovascular hospitalization (41% of wins) were the major contributors to the win ratio. Results consistently favored edoxaban in subgroups based on creatine clearance and dose reduction at baseline, with heightened benefit among those without prior vitamin K antagonist use.

CONCLUSIONS

In a win ratio analysis of the ENGAGE AF-TIMI 48 trial, both dose regimens of edoxaban were superior to warfarin for the net clinical outcome incorporating ischemic and bleeding events. As the win ratio emphasizes the most severe clinical events, this analysis supports the superiority of edoxaban over warfarin in patients with atrial fibrillation.

REGISTRATION

URL: https://www.clinicaltrials.gov; Unique identifier: NCT00781391.

A win ratio approach for comparing crossing survival curves in clinical trials

Many clinical trials include time-to-event or survival data as an outcome. To compare two survival distributions, the log-rank test is often used to produce a P-value for a statistical test of the null hypothesis that the two survival curves are identical. However, such a P-value does not provide the magnitude of the difference between the curves regarding the treatment effect. As a result, the P-value is often accompanied by an estimate of the hazard ratio from the proportional hazards model or Cox model as a measurement of treatment difference. However, one of the most important assumptions for Cox model is that the hazard functions for the two treatment groups are proportional. When the hazard curves cross, the Cox model could lead to misleading results and the log-rank test could also perform poorly. To address the problem of crossing curves in survival analysis, we propose the use of the win ratio method put forward by Pocock et al. as an estimand for analysing such data. The subjects in the test and control treatment groups are formed into all possible pairs. For each pair, the test treatment subject is labelled a winner or a loser if it is known who had the event of interest such as death. The win ratio is the total number of winners divided by the total number of losers and its standard error can be estimated using Bebu and Lachin method. Using real trial datasets and Monte Carlo simulations, this study investigates the power and type I error and compares the win ratio method with the log-rank test and Cox model under various scenarios of crossing survival curves with different censoring rates and distribution parameters. The results show that the win ratio method has similar power as the log-rank test and Cox model to detect the treatment difference when the assumption of proportional hazards holds true, and that the win ratio method outperforms log-rank test and Cox model in terms of power to detect the treatment difference when the survival curves cross.

Win statistics (win ratio, win odds, and net benefit) can complement one another to show the strength of the treatment effect on time-to-event outcomes

Conventional analyses of a composite of multiple time-to-event outcomes use the time to the first event. However, the first event may not be the most important outcome. To address this limitation, generalized pairwise comparisons and win statistics (win ratio, win odds, and net benefit) have become popular and have been applied to clinical trial practice. However, win ratio, win odds, and net benefit have typically been used separately. In this article, we examine the use of these three win statistics jointly for time-to-event outcomes. First, we explain the relation of point estimates and variances among the three win statistics, and the relation between the net benefit and the Mann-Whitney U statistic. Then we explain that the three win statistics are based on the same win proportions, and they test the same null hypothesis of equal win probabilities in two groups. We show theoretically that the Z-values of the corresponding statistical tests are approximately equal; therefore, the three win statistics provide very similar p-values and statistical powers. Finally, using simulation studies and data from a clinical trial, we demonstrate that, when there is no (or little) censoring, the three win statistics can complement one another to show the strength of the treatment effect. However, when the amount of censoring is not small, and without adjustment for censoring, the win odds and the net benefit may have an advantage for interpreting the treatment effect; with adjustment (e.g., IPCW adjustment) for censoring, the three win statistics can complement one another to show the strength of the treatment effect. For calculations we use the R package WINS, available on the CRAN (Comprehensive R Archive Network).

Stratified proportional win-fractions regression analysis

The recently proposed proportional win-fractions (PW) model extends the two-sample win ratio analysis of prioritized composite endpoints to regression. Its proportionality assumption ensures that the covariate-specific win ratios are invariant to the follow-up time. However, this assumption is strong and may not be satisfied by every covariate in the model. We develop a stratified PW model that adjusts for certain prognostic factors without setting them as covariates, thus bypassing the proportionality requirement. We formulate the stratified model based on pairwise comparisons within each stratum, with a common win ratio across strata modeled as a multiplicative function of the covariates. Correspondingly, we construct an estimating function for the regression coefficients in the form of an incomplete U $$ U $$ -statistic consisting of within-stratum pairs. Two types of asymptotic variance estimators are developed depending on the number of strata relative to the sample size. This in particular allows valid inference even when the strata are extremely small, such as with matched pairs. Simulation studies in realistic settings show that the stratified model outperforms the unstratified version in robustness and efficiency. Finally, real data from a major cardiovascular trial are analyzed to illustrate the potential benefits of stratification. The proposed methods are implemented in the R package WR, publicly available on the Comprehensive R Archive Network (CRAN).

Robust statistical inference for matched win statistics

As alternatives to the time-to-first-event analysis of composite endpoints, the win statistics, that is, the net benefit, the win ratio, and the win odds have been proposed to assess treatment effects, using a hierarchy of prioritized component outcomes based on clinical relevance or severity. Whether we are using paired organs of a human body or pair-matching patients by risk profiles or propensity scores, we can leverage the level of granularity of matched win statistics to assess the treatment effect. However, inference for the matched win statistics (net benefit, win ratio, and win odds)-quantities related to proportions-is either not available or unsatisfactory, especially in samples of small to moderate size or when the proportion of wins (or losses) is near 0 or 1. In this paper, we present methods to address these limitations. First, we introduce a different statistic to test for the null hypothesis of no treatment effect and provided a sample size formula. Then, we use the method of variance estimates recovery to derive reliable, boundary-respecting confidence intervals for the matched net benefit, win ratio, and win odds. Finally, a simulation study demonstrates the performance of the proposed methods. We illustrate the proposed methods with two data examples.

On recurrent-event win ratio

The win ratio approach proposed by Pocock et al. (2012) has become a popular tool for analyzing composite endpoints of death and non-fatal events like hospitalization. Its standard version, however, draws on the non-fatal event only through the first occurrence. For statistical efficiency and clinical interpretability, we construct and compare different win ratio variants that make fuller use of recurrent events. We pay special attention to a variant called last-event-assisted win ratio, which compares two patients on the cumulative frequency of the non-fatal event, with ties broken by the time of its latest episode. It is shown that last-event-assisted win ratio uses more data than the standard win ratio does but reduces to the latter when the non-fatal event occurs at most once. We further prove that last-event-assisted win ratio rejects the null hypothesis with large probability if the treatment stochastically delays all events. Simulations under realistic settings show that the last-event-assisted win ratio test consistently enjoys higher power than the standard win ratio and other competitors. Analysis of a real cardiovascular trial provides further evidence for the practical advantages of the last-event-assisted win ratio. Finally, we discuss future work to develop meaningful effect size estimands based on the extended rules of comparison. The R-code for the proposed methods is included in the package WR openly available on the Comprehensive R Archive Network.

Event-specific win ratios for inference with terminal and non-terminal events

For semi-competing risks data involving a non-terminal event and a terminal event we derive the asymptotic distributions of the event-specific win ratios under proportional hazards (PH) assumptions for the relevant cause-specific hazard functions of the non-terminal and terminal event, respectively. The win ratios converge to the respective hazard ratios under the PH assumptions and therefore are censoring-free, whether or not the censoring distributions in the two treatment arms are the same. With the asymptotic bivariate normal distributions of the win ratios, confidence intervals and testing procedures are obtained. Through extensive simulation studies and data analysis, we identified proper transformations of the win ratios that yield good control of the type one error rate for various testing procedures while maintaining competitive power. The confidence intervals also have good coverage probabilities. Furthermore, a test for the PH assumptions and a test of equal hazard ratios are developed. The new procedures are illustrated in the clinical trial Aldosterone Antagonist Therapy for Adults With Heart Failure and Preserved Systolic Function, which evaluated the effects of spironolactone in patients with heart failure and a preserved left ventricular ejection fraction.

The asymptotic distribution of the Net Benefit estimator in presence of right-censoring

The benefit-risk balance is a critical information when evaluating a new treatment. The Net Benefit has been proposed as a metric for the benefit-risk assessment, and applied in oncology to simultaneously consider gains in survival and possible side effects of chemotherapies. With complete data, one can construct a U-statistic estimator for the Net Benefit and obtain its asymptotic distribution using standard results of the U-statistic theory. However, real data is often subject to right-censoring, e.g. patient drop-out in clinical trials. It is then possible to estimate the Net Benefit using a modified U-statistic, which involves the survival time. The latter can be seen as a nuisance parameter affecting the asymptotic distribution of the Net Benefit estimator. We present here how existing asymptotic results on U-statistics can be applied to estimate the distribution of the net benefit estimator, and assess their validity in finite samples. The methodology generalizes to other statistics obtained using generalized pairwise comparisons, such as the win ratio. It is implemented in the R package BuyseTest (version 2.3.0 and later) available on Comprehensive R Archive Network.

Statistical models for composite endpoints of death and non-fatal events: a review

The proper analysis of composite endpoints consisting of both death and non-fatal events is an intriguing and sometimes contentious topic. The current practice of analyzing time to the first event often draws criticisms for ignoring the unequal importance between component events and for leaving recurrent-event data unused. Novel methods that address these limitations have recently been proposed. To compare the novel versus traditional approaches, we review three typical models for composite endpoints based on time to the first event, composite event process, and pairwise hierarchical comparisons. The pros and cons of these models are discussed with reference to the relevant regulatory guidelines, such as the recently released ICH-E9(R1) Addendum "Estimands and Sensitivity Analysis in Clinical Trials". We also discuss the impact of censoring when the model assumptions are violated and explore sensitivity analysis strategies. Simulation studies are conducted to assess the performance of the reviewed methods under different settings. As demonstration, we use publicly available R-packages to analyze real data from a major cardiovascular trial.

Win odds: An adaptation of the win ratio to include ties

The win ratio, a recently proposed measure for comparing the benefit of two treatment groups, allows ties in the data but ignores ties in the inference. In this article, we highlight some difficulties that this can lead to, and we propose to focus on the win odds instead, a modification of the win ratio which takes ties into account. We construct hypothesis tests and confidence intervals for the win odds, and we investigate their properties through simulations and in a case study. We conclude that the win odds should be preferred over the win ratio.

Adjusting win statistics for dependent censoring

For composite outcomes whose components can be prioritized on clinical importance, the win ratio, the net benefit and the win odds apply that order in comparing patients pairwise to produce wins and subsequently win proportions. Because these three statistics are derived using the same win proportions and they test the same hypothesis of equal win probabilities in the two treatment groups, we refer to them as win statistics. These methods, particularly the win ratio and the net benefit, have received increasing attention in methodological research and in design and analysis of clinical trials. For time-to-event outcomes, however, censoring may introduce bias. Previous work has shown that inverse-probability-of-censoring weighting (IPCW) can correct the win ratio for bias from independent censoring. The present article uses the IPCW approach to adjust win statistics for dependent censoring that can be predicted by baseline covariates and/or time-dependent covariates (producing the CovIPCW-adjusted win statistics). Theoretically and with examples and simulations, we show that the CovIPCW-adjusted win statistics are unbiased estimators of treatment effect in the presence of dependent censoring.

Event-specific win ratios and testing with terminal and non-terminal events

BACKGROUND/AIMS

In clinical trials, the primary outcome is often a composite endpoint defined as time to the first occurrence of either death or certain non-fatal events. Thus, a portion of available data would be omitted. In the win ratio approach, priorities are given to the clinically more important events, and more data are used. However, its power may be low if the treatment effect is predominantly on the non-terminal event.

METHODS

We propose event-specific win ratios obtained separately on the terminal and non-terminal events. They can then be used to form global tests such as a linear combination test, the maximum test, or a χ2 test.

RESULTS

In simulations, these tests often improve the power of the original win ratio test. Furthermore, when the terminal and non-terminal events experience differential treatment effects, the new tests are often more powerful than the log-rank test for the composite outcome. Whether the treatment effect is primarily on the terminal events or not, the new tests based on the event-specific win ratios can be useful when different types of events are present. The new tests can reject the null hypothesis of no difference in the event distributions in the two treatment arms with the terminal event showing detrimental effect and the non-terminal event showing beneficial effect. The maximum test and the χ2 test do not have test-estimation coherency, but the maximum test has the coherency that the global null is rejected if and only if the null for one of the event types is rejected. When applied to data from the trial Aldosterone Antagonist Therapy for Adults With Heart Failure and Preserved Systolic Function (TOPCAT), the new tests all reject the null hypothesis of no treatment effect while both the log-rank test used in TOPCAT and the original win ratio approach show non-significant p-values.

CONCLUSION

Whether the treatment effect is primarily on the terminal events or the non-terminal events, the maximum test based on the event-specific win ratios can be a useful alternative for testing treatment effect in clinical trials with time-to-event outcomes when different types of events are present.

A class of proportional win-fractions regression models for composite outcomes

The win ratio is gaining traction as a simple and intuitive approach to analysis of prioritized composite endpoints in clinical trials. To extend it from two-sample comparison to regression, we propose a novel class of semiparametric models that includes as special cases both the two-sample win ratio and the traditional Cox proportional hazards model on time to the first event. Under the assumption that the covariate-specific win and loss fractions are proportional over time, the regression coefficient is unrelated to the censoring distribution and can be interpreted as the log win ratio resulting from one-unit increase in the covariate. U-statistic estimating functions, in the form of an arbitrary covariate-specific weight process integrated by a pairwise residual process, are constructed to obtain consistent estimators for the regression parameter. The asymptotic properties of the estimators are derived using uniform weak convergence theory for U-processes. Visual inspection of a "score" process provides useful clues as to the plausibility of the proportionality assumption. Extensive numerical studies using both simulated and real data from a major cardiovascular trial show that the regression methods provide valid inference on covariate effects and outperform the two-sample win ratio in both efficiency and robustness. The proposed methodology is implemented in the R-package WR, publicly available from the Comprehensive R Archive Network (CRAN).

Adjusted win ratio with stratification: Calculation methods and interpretation

The win ratio is a general method of comparing locations of distributions of two independent, ordinal random variables, and it can be estimated without distributional assumptions. In this paper we provide a unified theory of win ratio estimation in the presence of stratification and adjustment by a numeric variable. Building step by step on the estimate of the crude win ratio we compare corresponding tests with well known non-parametric tests of group difference (Wilcoxon rank-sum test, Fligner-Policello test, van Elteren test, test based on the regression on ranks, and the rank analysis of covariance test). We show that the win ratio gives an interpretable treatment effect measure with corresponding test to detect treatment effect difference under minimal assumptions.

The inverse-probability-of-censoring weighting (IPCW) adjusted win ratio statistic: an unbiased estimator in the presence of independent censoring

The win ratio method has received much attention in methodological research, ad hoc analyses, and designs of prospective studies. As the primary analysis, it supported the approval of tafamidis for the treatment of cardiomyopathy to reduce cardiovascular mortality and cardiovascular-related hospitalization. However, its dependence on censoring is a potential shortcoming. In this article, we propose the inverse-probability-of-censoring weighting (IPCW) adjusted win ratio statistic (i.e., the IPCW-adjusted win ratio statistic) to overcome censoring issues. We consider independent censoring, common censoring across endpoints, and right censoring. We develop an asymptotic variance estimator for the logarithm of the IPCW-adjusted win ratio statistic and evaluate it via simulation. Our simulation studies show that, as the amount of censoring increases, the unadjusted win proportions may decrease greatly. Consequently, the bias of the unadjusted win ratio estimate may increase greatly, producing either an overestimate or an underestimate. We demonstrate theoretically and through simulation that the IPCW-adjusted win ratio statistic gives an unbiased estimate of treatment effect.

Choosing primary endpoints for clinical trials of health care interventions

The purpose of late phase clinical trials is to generate evidence of sufficient validity and generalisability to be translated into practice and policy to improve health outcomes. It is therefore crucial that the chosen endpoints are meaningful to the clinicians, patients and policymakers that are the end-users of evidence generated by these trials. The choice of endpoints may be improved by understanding their characteristics and properties. This narrative review describes the evolution, range and relative strengths and weaknesses of endpoints used in late phase trials. It is intended to serve as a reference to assist those designing trials when choosing primary endpoint(s), and for the end-users charged with interpreting these trials to inform practice and policy.

Use of composite outcomes to assess risk-benefit in clinical trials

Before a novel treatment can be deemed a clinical success, an assessment of its risk-benefit profile must be made. One of the inherent challenges for this assessment comes from the multiplicity that arises from comparing treatment groups across multiple outcomes. Composite outcomes that summarize a patient's clinical status, or severity, across a prioritized list of safety and efficacy outcomes have become increasing popular. In this article, we review these approaches and illustrate through examples some of the challenges and complexities of a composite derived from prioritized outcomes, such as the win ratio. These challenges include the difficult tension between the analytical validity that comes from choosing a pre-specified outcome and an evaluation that is responsive to unexpected safety events that arise during the course of a trial. Other challenges include a sensitivity of the resulting test statistic to the underlying censoring distribution and other nuisance parameters. Approaches that resolve some of the difficulties of the analytical challenges associated with prioritized outcomes are then discussed. Ultimately, a composite outcome of net clinical benefit is another decision tool, but one to be used alongside more traditional analyses of efficacy and safety, and with the broader perspective that investigators, the data safety monitoring board, and regulators bring to an evaluation of risk-benefit.