The win ratio approach to analyzing composite outcomes: An application to the EVOLVE trial

Comparing Multivisceral Resection with Tumor-only Resection of Liposarcoma Using the Win Ratio

BACKGROUND

Multivisceral resection of retroperitoneal liposarcoma (LPS) is associated with increased morbidity and may not confer a survival benefit compared with tumor-only (TO) resection. We compared both approaches using a novel statistical method called the "win ratio" (WR).

METHODS

Patients who underwent resection of LPS from 2004 to 2015 were identified from the National Cancer Database. Multivisceral resection was defined as removal of the primary site in addition to other organs. The WR was calculated based on a hierarchy of postoperative outcomes: 30-day and 90-day mortality, long-term survival, and severe complication.

RESULTS

Among 958 patients (multivisceral 634, TO 324) who underwent resection, the median age was 63 years (interquartile range [IQR] 54-71) with a median follow-up of 51 months (IQR 30-86). There was no difference in the WR among patients who underwent TO versus multivisceral resection in the matched cohort (WR 0.82, 95% confidence interval [CI] 0.61-1.10). In patients aged 72-90 years, those who underwent multivisceral resection had 36% lower odds of winning compared with patients undergoing TO resection (WR 0.64, 95% CI 0.40-0.98). A subgroup analysis of patients classified as not having adjacent tumor involvement at the time of surgery revealed that those patients who underwent multivisceral resection had 33% lower odds of winning compared to TO resection (WR 0.67, 95% CI 0.45-0.99).

CONCLUSIONS

Based on win-ratio assessments of a hierarchical composite endpoint, multivisceral resection in patients without adjacent tumor involvement may not confer improved outcomes. This method supports the rationale for less invasive resection of LPS in select patients, especially older patients.

Dissecting the restricted mean time in favor of treatment

The restricted mean time in favor (RMT-IF) summarizes the treatment effect on a hierarchical composite endpoint with mortality at the top. Its crude decomposition into "stage-wise effects," i.e., the net average time gained by the treatment prior to each component event, does not reveal the patient state in which the extra time is spent. To obtain this information, we break each stage-wise effect into subcomponents according to the specific state to which the reference condition is improved. After re-expressing the subcomponents as functionals of the marginal survival functions of outcome events, we estimate them conveniently by plugging in the Kaplan -- Meier estimators. Their robust variance matrices allow us to construct joint tests on the decomposed units, which are particularly powerful against component-wise differential treatment effects. By reanalyzing a cancer trial and a cardiovascular trial, we acquire new insights into the quality and composition of the extra survival times, as well as the extra time with fewer hospitalizations, gained by the treatment in question. The proposed methods are implemented in the rmt package freely available on the Comprehensive R Archive Network (CRAN).

Reflections on estimands for patient-reported outcomes in cancer clinical trials

It is common and important to include the patient's perspective of the impact of treatment on health-related quality of life (HRQoL) outcomes. In this commentary, we focus on applying the new addendum to ICH E9 guideline E9 (R1) relating to the estimand framework to Patient Reported Outcomes (PROs) collected in cancer clinical trials, from a statistician's viewpoint. Currently, common practice for statistical analysis of PRO endpoints of published cancer clinical trials demonstrates ambiguity, leaving critical questions unspecified, hindering conclusions about the effect of treatment on PRO endpoints as well as comparability between clinical trials. To avoid this scenario, we advocate the systematic use of the estimand framework which requires the prospective definition of clear PRO research questions. Among the five attributes of the estimands framework, the definition of the endpoint (what is the right PRO measure and timeframe to target and why?), the intercurrent event identification and management (what happens with PRO data post-disease progression, what is the impact of death?) and the population-level summary (what is an acceptable statistical summary for PRO data?) require the most attention for PRO estimands. We identify good practice and highlight discussion points including the challenges of statistical analysis in the presence of missing and/or unobservable data and in relation to death. Through this discussion we highlight that there is no "statistical magic", but that the estimand framework will help you find out what you really want to know when quantifying the benefit of treatments from the patients' perspective.

Generalized Pairwise Comparisons to Assess Treatment Effects: JACC Review Topic of the Week

A time-to-first-event composite endpoint analysis has well-known shortcomings in evaluating a treatment effect in cardiovascular clinical trials. It does not fully describe the clinical benefit of therapy because the severity of the events, events repeated over time, and clinically relevant nonsurvival outcomes cannot be considered. The generalized pairwise comparisons (GPC) method adds flexibility in defining the primary endpoint by including any number and type of outcomes that best capture the clinical benefit of a therapy as compared with standard of care. Clinically important outcomes, including bleeding severity, number of interventions, and quality of life, can easily be integrated in a single analysis. The treatment effect in GPC can be expressed by the net treatment benefit, the success odds, or the win ratio. This review provides guidance on the use of GPC and the choice of treatment effect measures for the analysis and reporting of cardiovascular trials.

A win ratio approach for comparing crossing survival curves in clinical trials

Many clinical trials include time-to-event or survival data as an outcome. To compare two survival distributions, the log-rank test is often used to produce a P-value for a statistical test of the null hypothesis that the two survival curves are identical. However, such a P-value does not provide the magnitude of the difference between the curves regarding the treatment effect. As a result, the P-value is often accompanied by an estimate of the hazard ratio from the proportional hazards model or Cox model as a measurement of treatment difference. However, one of the most important assumptions for Cox model is that the hazard functions for the two treatment groups are proportional. When the hazard curves cross, the Cox model could lead to misleading results and the log-rank test could also perform poorly. To address the problem of crossing curves in survival analysis, we propose the use of the win ratio method put forward by Pocock et al. as an estimand for analysing such data. The subjects in the test and control treatment groups are formed into all possible pairs. For each pair, the test treatment subject is labelled a winner or a loser if it is known who had the event of interest such as death. The win ratio is the total number of winners divided by the total number of losers and its standard error can be estimated using Bebu and Lachin method. Using real trial datasets and Monte Carlo simulations, this study investigates the power and type I error and compares the win ratio method with the log-rank test and Cox model under various scenarios of crossing survival curves with different censoring rates and distribution parameters. The results show that the win ratio method has similar power as the log-rank test and Cox model to detect the treatment difference when the assumption of proportional hazards holds true, and that the win ratio method outperforms log-rank test and Cox model in terms of power to detect the treatment difference when the survival curves cross.

The win ratio: A novel approach to define and analyze postoperative composite outcomes to reflect patient and clinician priorities

BACKGROUND

The "win ratio" (WR) is a novel statistical technique that hierarchically weighs various postoperative outcomes (eg, mortality weighted more than complications) into a composite metric to define an overall benefit or "win." We sought to use the WR to assess the impact of social vulnerability on the likelihood of achieving a "win" after hepatopancreatic surgery.

METHODS

Individuals who underwent an elective hepatopancreatic procedure between 2013 and 2017 were identified using the Medicare database, which was merged with the Center for Disease Control and Prevention's Social Vulnerability Index. The win ratio was defined based on a hierarchy of postoperative outcomes: 90-day mortality, perioperative complications, 90-day readmissions, and length of stay. Patients matched based on procedure type, race, sex, age, and Charlson Comorbidity Index score were compared and assessed relative to win ratio.

RESULTS

Among 32,557 Medicare beneficiaries who underwent hepatectomy (n = 11,621, 35.7%) or pancreatectomy (n = 20,936, 64.3%), 16,846 (51.7%) patients were male with median age of 72 years (interquartile range 68-77) and median Charlson Comorbidity Index of 3 (interquartile range 2-8), and a small subset of patients were a racial/ethnic minority (n = 3,759, 11.6%). Adverse events associated with lack of a postoperative optimal outcome included 90-day mortality (n = 2,222, 6.8%), postoperative complication (n = 8,029, 24.7%), readmission (n = 6,349, 19.5%), and length of stay (median: 7 days, interquartile range 5-11). Overall, the patients from low Social Vulnerability Index areas were more likely to "win" with a textbook outcome (win ratio 1.07, 95% confidence interval 1.01-1.12) compared with patients from high social vulnerability counties; in contrast, there was no difference in the win ratio among patients living in average versus high Social Vulnerability Index (win ratio 1.04, 95% confidence interval 0.98-1.10). In assessing surgeon volume, patients who had a liver or pancreas procedure performed by a high-volume surgeon had a higher win ratio versus patients who were treated by a low-volume surgeon (win ratio 1.21, 95% confidence interval 1.16-1.25). In contrast, there was no difference in the win ratio (win ratio 1.01, 95% confidence interval 0.97-1.06) among patients relative to teaching hospital status.

CONCLUSION

Using a novel statistical approach, the win ratio ranked outcomes to create a composite measure to assess a postoperative "win." The WR demonstrated that social vulnerability was an important driver in explaining disparate postoperative outcomes.

Win ratio approach for analyzing composite time-to-event endpoint with opposite treatment effects in its components

There is an increasing interest in the use of win ratio with composite time-to-event due to its flexibility in combining component endpoints. Exploring this flexibility further, one interesting question is in assessing the impact when there is a difference in treatment effect in the component endpoints. For example, the active treatment may prolong the time to occurrence of the negative event such as death or ventilation; meanwhile, the treatment effect may also shorten the time to achieving positive events, such as recovery or improvement. Notably, this portrays a situation where the treatment effect on time to recovery is in a different direction of benefit compared to the time to ventilation or death. Under such circumstances, if a single endpoint is used, the benefit gained for other individual outcomes is not counted and is diminished. As consequence, the study may need a larger sample size to detect a significant effect of treatment. Such a scenario can be handled by win ratio in a novel way by ranking component events, which is different from the usual composite endpoint approach such as time-to-first event. To evaluate how the different directions of treatment effect on component endpoints will impact the win ratio analysis, we use a Clayton copula-based bivariate survival simulation to investigate the correlation of component time-to-event. Through simulation, we found that compared to the marginal model using single endpoints, the win ratio analysis on composite endpoint performs better, especially when the correlation between two events is weak. Then, we applied the methodology to an infectious disease progression simulated study motivated by COVID-19. The application demonstrates that the win ratio approach offers advantages in empirical power compared to the traditional Cox proportional hazard approach when there is a difference in treatment effect in the marginal events.

On recurrent-event win ratio

The win ratio approach proposed by Pocock et al. (2012) has become a popular tool for analyzing composite endpoints of death and non-fatal events like hospitalization. Its standard version, however, draws on the non-fatal event only through the first occurrence. For statistical efficiency and clinical interpretability, we construct and compare different win ratio variants that make fuller use of recurrent events. We pay special attention to a variant called last-event-assisted win ratio, which compares two patients on the cumulative frequency of the non-fatal event, with ties broken by the time of its latest episode. It is shown that last-event-assisted win ratio uses more data than the standard win ratio does but reduces to the latter when the non-fatal event occurs at most once. We further prove that last-event-assisted win ratio rejects the null hypothesis with large probability if the treatment stochastically delays all events. Simulations under realistic settings show that the last-event-assisted win ratio test consistently enjoys higher power than the standard win ratio and other competitors. Analysis of a real cardiovascular trial provides further evidence for the practical advantages of the last-event-assisted win ratio. Finally, we discuss future work to develop meaningful effect size estimands based on the extended rules of comparison. The R-code for the proposed methods is included in the package WR openly available on the Comprehensive R Archive Network.

Statistical models for composite endpoints of death and non-fatal events: a review

The proper analysis of composite endpoints consisting of both death and non-fatal events is an intriguing and sometimes contentious topic. The current practice of analyzing time to the first event often draws criticisms for ignoring the unequal importance between component events and for leaving recurrent-event data unused. Novel methods that address these limitations have recently been proposed. To compare the novel versus traditional approaches, we review three typical models for composite endpoints based on time to the first event, composite event process, and pairwise hierarchical comparisons. The pros and cons of these models are discussed with reference to the relevant regulatory guidelines, such as the recently released ICH-E9(R1) Addendum "Estimands and Sensitivity Analysis in Clinical Trials". We also discuss the impact of censoring when the model assumptions are violated and explore sensitivity analysis strategies. Simulation studies are conducted to assess the performance of the reviewed methods under different settings. As demonstration, we use publicly available R-packages to analyze real data from a major cardiovascular trial.

Win odds: An adaptation of the win ratio to include ties

The win ratio, a recently proposed measure for comparing the benefit of two treatment groups, allows ties in the data but ignores ties in the inference. In this article, we highlight some difficulties that this can lead to, and we propose to focus on the win odds instead, a modification of the win ratio which takes ties into account. We construct hypothesis tests and confidence intervals for the win odds, and we investigate their properties through simulations and in a case study. We conclude that the win odds should be preferred over the win ratio.

The win ratio approach for composite endpoints: practical guidance based on previous experience

The win ratio was introduced in 2012 as a new method for examining composite endpoints and has since been widely adopted in cardiovascular (CV) trials. Improving upon conventional methods for analysing composite endpoints, the win ratio accounts for relative priorities of the components and allows the components to be different types of outcomes. For example, the win ratio can combine the time to death with the number of occurrences of a non-fatal outcome such as CV-related hospitalizations (CVHs) in a single hierarchical composite endpoint. The win ratio can provide greater statistical power to detect and quantify a treatment difference by using all available information contained in the component outcomes. The win ratio can also incorporate quantitative outcomes such as exercise tests or quality-of-life scores. There is a need for more practical guidance on how best to design trials using the win ratio approach. This manuscript provides an overview of the principles behind the win ratio and provides insights into how to implement the win ratio in CV trial design and reporting, including how to determine trial size.

A win ratio approach to the re-analysis of Multiple Risk Factor Intervention Trial

Background:

Composite outcomes, which combine multiple types of clinical events into a single outcome, are common in clinical trials. The usual analysis considers the time to first occurrence of any event in the composite. The major criticisms of such an approach are (1) this implicitly treats the outcomes as if they were of equal importance, but they often vary in terms of clinical relevance and severity, (2) study participants often experience more than one type of event, and (3) often less severe events occur before more severe ones, but the usual analysis disregards any information beyond that first event.

Methods:

A novel approach, referred to as the win ratio, which addresses the aforementioned criticisms of composite outcomes, is illustrated with a re-analysis of data on fatal and non-fatal cardiovascular disease time-to-event outcomes reported for the Multiple Risk Factor Intervention Trial. In this trial, 12,866 participants were randomized to a special intervention group ( n = 6428) or a usual care ( n = 6438) group. Non-fatal outcomes were ranked by risk of cardiovascular disease death up to 20 years after trial. In one approach, participants in the special intervention and usual care groups were first matched on coronary heart disease risk at baseline and time of enrollment. Each matched pair was categorized as a winner or loser depending on which one experienced a cardiovascular disease death first. If neither died of cardiovascular disease causes, they were evaluated on the most severe non-fatal outcome. This process continued for all the non-fatal outcomes. A second win ratio statistic, obtained from Cox partial likelihood, was also estimated. This statistic provides a valid estimate of the win ratio using multiple events if the marginal and conditional survivor functions of each outcome satisfy proportional hazards. Loss ratio statistics (inverse of win ratios) are compared to hazard ratios from the usual first event analysis. A larger 11-event composite was also considered.

Results:

For the 7-event cardiovascular disease composite, the previously reported first event analysis based on 581 events in the special intervention group and 652 events in the usual care group yielded a hazard ratio (95% confidence interval) of 0.89 (0.79–0.99), compared to 0.86 (0.77–0.97) and 0.91 (0.81–1.02) for the severity ranked estimates. Results for the 11-event composite also confirmed the findings of the first event analysis.

Conclusion:

The win ratio analysis was able to leverage information collected past the first experienced event and rank events by severity. The results were similar to and confirmed previously reported traditional first event analysis. The win ratio statistic is a useful adjunct to the traditional first event analysis for trials with composite outcomes.

Weighted win loss approach for analyzing prioritized outcomes

To analyze prioritized outcomes, Buyse (2010) and Pocock et al. (2012) proposed the win loss approach. In this paper, we first study the relationship between the win loss approach and the traditional survival analysis on the time to the first event. We then propose the weighted win loss statistics to improve the efficiency of the unweighted methods. A closed-form variance estimator of the weighted win loss statistics is derived to facilitate hypothesis testing and study design. We also calculated the contribution index to better interpret the results of the weighted win loss approach. Simulation studies and real data analysis demonstrated the characteristics of the proposed statistics. Copyright © 2017 John Wiley & Sons, Ltd.