Review and comparison of ROC curve estimators for a time-dependent outcome with marker-dependent censoring: ROC estimators with marker-dependent censoring

Prediction Under Interventions: Evaluation of Counterfactual Performance Using Longitudinal Observational Data

Predictions under interventions are estimates of what a person's risk of an outcome would be if they were to follow a particular treatment strategy, given their individual characteristics. Such predictions can give important input to medical decision-making. However, evaluating the predictive performance of interventional predictions is challenging. Standard ways of evaluating predictive performance do not apply when using observational data, because prediction under interventions involves obtaining predictions of the outcome under conditions that are different from those that are observed for a subset of individuals in the validation dataset. This work describes methods for evaluating counterfactual performance of predictions under interventions for time-to-event outcomes. This means we aim to assess how well predictions would match the validation data if all individuals had followed the treatment strategy under which predictions are made. We focus on counterfactual performance evaluation using longitudinal observational data, and under treatment strategies that involve sustaining a particular treatment regime over time. We introduce an estimation approach using artificial censoring and inverse probability weighting that involves creating a validation dataset mimicking the treatment strategy under which predictions are made. We extend measures of calibration, discrimination (c-index and cumulative/dynamic AUCt) and overall prediction error (Brier score) to allow assessment of counterfactual performance. The methods are evaluated using a simulation study, including scenarios in which the methods should detect poor performance. Applying our methods in the context of liver transplantation shows that our procedure allows quantification of the performance of predictions supporting crucial decisions on organ allocation.

A novel nonparametric time-dependent precision-recall curve estimator for right-censored survival data

In order to assess prognostic risk for individuals in precision health research, risk prediction models are increasingly used, in which statistical models are used to estimate the risk of future outcomes based on clinical and nonclinical characteristics. The predictive accuracy of a risk score must be assessed before it can be used in routine clinical decision making, where the receiver operator characteristic curves, precision-recall curves, and their corresponding area under the curves are commonly used metrics to evaluate the discriminatory ability of a continuous risk score. Among these the precision-recall curves have been shown to be more informative when dealing with unbalanced biomarker distribution between classes, which is common in rare event, even though except one, all existing methods are proposed for classic uncensored data. This paper is therefore to propose a novel nonparametric estimation approach for the time-dependent precision-recall curve and its associated area under the curve for right-censored data. A simulation is conducted to show the better finite sample property of the proposed estimator over the existing method and a real-world data from primary biliary cirrhosis trial is used to demonstrate the practical applicability of the proposed estimator.

Time-dependent receiver operating characteristic curve estimator for correlated right-censored time-to-event data

In clinical trials, evaluating the accuracy of risk scores (markers) derived from prognostic models for prediction of survival outcomes is of major concern. The time-dependent receiver operating characteristic curve and the corresponding area under the receiver operating characteristic curve are appealing measures to evaluate the predictive accuracy. Several estimation methods have been proposed in the context of classical right-censored data which assumes the event time of individuals are independent. In many applications, however, this may not hold true if, for example, individuals belong to clusters or experience recurrent events. Estimates may be biased if this correlated nature is not taken into account. This paper is then aimed to fill this knowledge gap to introduce a time-dependent receiver operating characteristic curve and the corresponding area under the receiver operating characteristic curve estimation method for right-censored data that take the correlated nature into account. In the proposed method, the unknown status of censored subjects is imputed using conditional survival functions given the marker and frailty of the subjects. An extensive simulation study is conducted to evaluate and demonstrate the finite sample performance of the proposed method. Finally, the proposed method is illustrated using two real-world examples of lung cancer and kidney disease.

Predicting diagnostic conversion from major depressive disorder to bipolar disorder: an EHR based study from Colombia

Bipolar Disorder (BD) is a severe and chronic disorder characterized by recurrent episodes of depression, mania, and/or hypomania. Most BD patients initially present with depressive symptoms, resulting in a delayed diagnosis of BD and poor clinical outcomes. This study leverages electronic health record (EHR) data from the Clínica San Juan de Dios Manizales in Colombia to identify features predictive of the transition from Major Depressive Disorder (MDD) to BD. Analyzing EHR data from 13,607 patients diagnosed with MDD over 15 years, we identified 1,610 cases of conversion to BD. Using a multivariate Cox regression model, we identified severity of the initial MDD episode, the presence of psychosis and hospitalization at first episode, family history of mood or psychotic disorders, female gender to be predictive of the conversion to BD. Additionally, we observed associations with medication classes (prescriptions of mood stabilizers, antipsychotics, and antidepressants) and clinical features (delusions, suicide attempt, suicidal ideation, use of marijuana and alcohol use/abuse) derived from natural language processing (NLP) of clinical notes. Together, these risk factors predicted BD conversion within five years of the initial MDD diagnosis, with a recall of 72% and a precision of 38%. Our study confirms many previously identified risk factors identified through registry-based studies (such as female gender and psychotic depression at the index MDD episode), and identifies novel ones (specifically, suicidal ideation and suicide attempt extracted from clinical notes). These results simultaneously demonstrate the validity of using EHR data for predicting BD conversion as well as underscore its potential for the identification of novel risk factors and improving early diagnosis.

Time-Dependent ROC Curve Analysis for Assessing the Capability of Radiation-Induced CD8 T-Lymphocyte Apoptosis to Predict Late Toxicities after Adjuvant Radiotherapy of Breast Cancer Patients

Late fibrosis can occur in breast cancer patients treated with curative-intent radiotherapy. Predicting this toxicity is of clinical interest in order to adapt the irradiation dose delivered. Radiation-induced CD8 T-lymphocyte apoptosis (RILA) had been proven to be associated with less grade ≥2 late radiation-induced toxicities in patients with miscellaneous cancers. Tobacco smoking status and adjuvant hormonotherapy were also identified as potential factors related to late-breast-fibrosis-free survival. This article evaluates the predictive performance of the RILA using a ROC curve analysis that takes into account the dynamic nature of fibrosis occurrence. This time-dependent ROC curve approach is also applied to evaluate the ability of the RILA combined with the other previously identified factors. Our analysis includes a Monte Carlo cross-validation procedure and the calculation of an expected cost of misclassification, which provides more importance to patients who have no risk of late fibrosis in order to be able to treat them with the maximal irradiation dose. Performance evaluation was assessed at 12, 24, 36 and 50 months. At 36 months, our results were comparable to those obtained in a previous study, thus underlying the predictive power of the RILA. Based on specificity and cost, RILA alone seemed to be the most performant, while its association with the other factors had better negative predictive value results.

A semi-parametric approach for time-dependent ROC curves with nonignorable missing biomarker

The main purpose of this paper is to survey the statistical inference for covariate-specific time-dependent receiver operating characteristic (ROC) curves with nonignorable missing continuous biomarker values. To construct time-dependent ROC curves, we consider a joint model which assumes that the failure time depends on the continuous biomarker and the covariates through a Cox proportional hazards model and that the continuous biomarker depends on the covariates through a semiparametric location model. Assuming a purely parametric model on the propensity score, we utilize instrumental variables to deal with the identifiable issue and estimate the unknown parameters of the propensity score by a simple and efficient method. In addition, when the propensity score is estimated, we develop HT and AIPW approaches to estimate our interested quantities. In the presence of nonignorable missing biomarker, our AIPW estimators of the interested quantities are still doubly robust when the true propensity score is a special parametric logistic model. At last, simulation studies are conducted to assess the performance of our proposed approaches, and a real data analysis is also carried out to illustrate its application.

MELD 3.0 adequately predicts mortality and renal replacement therapy requirements in patients with alcohol-associated hepatitis

Background & Aims

Model for End-Stage Liver Disease (MELD) score better predicts mortality in alcohol-associated hepatitis (AH) but could underestimate severity in women and malnourished patients. Using a global cohort, we assessed the ability of the MELD 3.0 score to predict short-term mortality in AH.

Methods

This was a retrospective cohort study of patients admitted to hospital with AH from 2009 to 2019. The main outcome was all-cause 30-day mortality. We compared the AUC using DeLong's method and also performed a time-dependent AUC with competing risks analysis.

Results

A total of 2,124 patients were included from 28 centres from 10 countries on three continents (median age 47.2 ± 11.2 years, 29.9% women, 71.3% with underlying cirrhosis). The median MELD 3.0 score at admission was 25 (20-33), with an estimated survival of 73.7% at 30 days. The MELD 3.0 score had a better performance in predicting 30-day mortality (AUC:0.761, 95%CI:0.732-0.791) compared with MELD sodium (MELD-Na; AUC: 0.744, 95% CI: 0.713-0.775; p = 0.042) and Maddrey's discriminant function (mDF) (AUC: 0.724, 95% CI: 0.691-0.757; p = 0.013). However, MELD 3.0 did not perform better than traditional MELD (AUC: 0.753, 95% CI: 0.723-0.783; p = 0.300) and Age-Bilirubin-International Normalised Ratio-Creatinine (ABIC) (AUC:0.757, 95% CI: 0.727-0.788; p = 0.765). These results were consistent in competing-risk analysis, where MELD 3.0 (AUC: 0.757, 95% CI: 0.724-0.790) predicted better 30-day mortality compared with MELD-Na (AUC: 0.739, 95% CI: 0.708-0.770; p = 0.028) and mDF (AUC:0.717, 95% CI: 0.687-0.748; p = 0.042). The MELD 3.0 score was significantly better in predicting renal replacement therapy requirements during admission compared with the other scores (AUC: 0.844, 95% CI: 0.805-0.883).

Conclusions

MELD 3.0 demonstrated better performance compared with MELD-Na and mDF in predicting 30-day and 90-day mortality, and was the best predictor of renal replacement therapy requirements during admission for AH. However, further prospective studies are needed to validate its extensive use in AH.

Impact and implications

Severe AH has high short-term mortality. The establishment of treatments and liver transplantation depends on mortality prediction. We evaluated the performance of the new MELD 3.0 score to predict short-term mortality in AH in a large global cohort. MELD 3.0 performed better in predicting 30- and 90-day mortality compared with MELD-Na and mDF, but was similar to MELD and ABIC scores. MELD 3.0 was the best predictor of renal replacement therapy requirements. Thus, further prospective studies are needed to support the wide use of MELD 3.0 in AH.

CA-125 Early Dynamics to Predict Overall Survival in Women with Newly Diagnosed Advanced Ovarian Cancer Based on Meta-Analysis Data

(1) Background: Cancer antigen 125 (CA-125) is a protein produced by ovarian cancer cells that is used for patients' monitoring. However, the best ways to analyze its decline and prognostic role are poorly quantified. (2) Methods: We leveraged individual patient data from the Gynecologic Cancer Intergroup (GCIG) meta-analysis (N = 5573) to compare different approaches summarizing the early trajectory of CA-125 before the prediction time (called the landmark time) at 3 or 6 months after treatment initiation in order to predict overall survival. These summaries included observed and estimated measures obtained by a linear mixed model (LMM). Their performances were evaluated by 10-fold cross-validation with the Brier score and the area under the ROC (AUC). (3) Results: The estimated value and the last observed value at 3 months were the best measures used to predict overall survival, with an AUC of 0.75 CI 95% [0.70; 0.80] at 24 and 36 months and 0.74 [0.69; 0.80] and 0.75 [0.69; 0.80] at 48 months, respectively, considering that CA-125 over 6 months did not improve the AUC, with 0.74 [0.68; 0.78] at 24 months and 0.71 [0.65; 0.76] at 36 and 48 months. (4) Conclusions: A 3-month surveillance provided reliable individual information on overall survival until 48 months for patients receiving first-line chemotherapy.

Assessing Performance and Clinical Usefulness in Prediction Models With Survival Outcomes: Practical Guidance for Cox Proportional Hazards Models

Risk prediction models need thorough validation to assess their performance. Validation of models for survival outcomes poses challenges due to the censoring of observations and the varying time horizon at which predictions can be made. This article describes measures to evaluate predictions and the potential improvement in decision making from survival models based on Cox proportional hazards regression.

As a motivating case study, the authors consider the prediction of the composite outcome of recurrence or death (the "event") in patients with breast cancer after surgery. They developed a simple Cox regression model with 3 predictors, as in the Nottingham Prognostic Index, in 2982 women (1275 events over 5 years of follow-up) and externally validated this model in 686 women (285 events over 5 years). Improvement in performance was assessed after the addition of progesterone receptor as a prognostic biomarker.

The model predictions can be evaluated across the full range of observed follow-up times or for the event occurring by the end of a fixed time horizon of interest. The authors first discuss recommended statistical measures that evaluate model performance in terms of discrimination, calibration, or overall performance. Further, they evaluate the potential clinical utility of the model to support clinical decision making according to a net benefit measure. They provide SAS and R code to illustrate internal and external validation.

The authors recommend the proposed set of performance measures for transparent reporting of the validity of predictions from survival models.

Exploring the role of neutrophil-to-lymphocyte ratio and blood chemistry in head and neck adenoid cystic carcinomas treated with carbon ion radiotherapy

BACKGROUND AND PURPOSE

In recent years, there is an emerging interest in the prognostic role of chemistry blood biomarkers in oncological patients but their role in adenoid cystic carcinomas (ACCs) is still unknown. This study aims to assess the prognostic significance of baseline neutrophil-to-lymphocyte ratio (NLR) and blood chemistry in a series of head and neck ACC patients treated with carbon ion radiotherapy (CIRT).

MATERIAL AND METHODS

We retrospectively retrieved the data of 49 consecutive head and neck ACC patients treated with CIRT. Univariable and multivariable Cox proportional hazard regression (Cox-ph) analyses were performed to look for a potential association of NLR, and other blood biomarker values, with disease-free survival (DFS), Local Control (LC), Metastasis Free Survival (MFS) and overall survival (OS).

RESULTS

No significant association between NLR > 2,5 and DFS, LC, MFS and OS was found with univariable analysis although a trend was reported for DFS (Hazard ratio [HR]: 2,10, 95 % CI: 0,85 - 5,08, p-value = 0,11). Patients with hemoglobin (hb) ≤ 14 g/dL showed significantly better DFS, MFS and OS. Multivariable regression Cox-ph analysis for DFS, adjusted for margin status, clinical target volume and Absolute Number of Monocytes, reported the following statistically significant HRs, for both NLR > 2,5 and hb > 14 g/dL respectively: 4,850 (95 % CI = 1,408 - 16,701, p = 0,012) and 3,032 (95 % CI = 1,095 - 8,393, p = 0,033). Moreover, hb > 14 with HR = 3,69 (95 % CI: 1,23 - 11,07, p-value = 0,02), was a negative independent prognostic predictor for MFS.

CONCLUSIONS

Pre-treatment NLR and hb values seem to be independent prognostic predictor for clinical outcomes in head and neck ACC patients. If their role will be validated in a larger prospective cohort, they might be worthwhile for a pre-treatment risk stratification in patients treated with CIRT.

Survival prediction models: an introduction to discrete-time modeling

Background

Prediction models for time-to-event outcomes are commonly used in biomedical research to obtain subject-specific probabilities that aid in making important clinical care decisions. There are several regression and machine learning methods for building these models that have been designed or modified to account for the censoring that occurs in time-to-event data. Discrete-time survival models, which have often been overlooked in the literature, provide an alternative approach for predictive modeling in the presence of censoring with limited loss in predictive accuracy. These models can take advantage of the range of nonparametric machine learning classification algorithms and their available software to predict survival outcomes.

Methods

Discrete-time survival models are applied to a person-period data set to predict the hazard of experiencing the failure event in pre-specified time intervals. This framework allows for any binary classification method to be applied to predict these conditional survival probabilities. Using time-dependent performance metrics that account for censoring, we compare the predictions from parametric and machine learning classification approaches applied within the discrete time-to-event framework to those from continuous-time survival prediction models. We outline the process for training and validating discrete-time prediction models, and demonstrate its application using the open-source R statistical programming environment.

Results

Using publicly available data sets, we show that some discrete-time prediction models achieve better prediction performance than the continuous-time Cox proportional hazards model. Random survival forests, a machine learning algorithm adapted to survival data, also had improved performance compared to the Cox model, but was sometimes outperformed by the discrete-time approaches. In comparing the binary classification methods in the discrete time-to-event framework, the relative performance of the different methods varied depending on the data set.

Conclusions

We present a guide for developing survival prediction models using discrete-time methods and assessing their predictive performance with the aim of encouraging their use in medical research settings. These methods can be applied to data sets that have continuous time-to-event outcomes and multiple clinical predictors. They can also be extended to accommodate new binary classification algorithms as they become available. We provide R code for fitting discrete-time survival prediction models in a github repository.

Supplementary Information

The online version contains supplementary material available at (10.1186/s12874-022-01679-6).

Prognostic accuracy of inflammatory markers in predicting risk of ICU admission for COVID-19: application of time-dependent receiver operating characteristic curves

Objective

Intensive care unit (ICU) admission occurs at different times during hospitalization among patients with COVID-19. We aimed to evaluate the time-dependent receive operating characteristic (ROC) curve and area under the ROC curve, AUC(t), and accuracy of baseline levels of inflammatory markers C-reactive protein (CRP) and neutrophil-to-lymphocyte ratio (NLR) in predicting time to an ICU admission event in patients with severe COVID-19 infection.

Methods

In this observational study, we evaluated 724 patients with confirmed severe COVID-19 referred to Ayatollah Rohani Hospital, affiliated with Babol University of Medical Sciences, Iran.

Results

The AUC(t) of CRP and NLR reached 0.741 (95% confidence interval [CI]: 0.661–0.820) and 0.690 (95% CI: 0.607–0.772), respectively, in the first 3 days after hospital admission. The optimal cutoff values of CRP and NLR for stratification of ICU admission outcomes in patients with severe COVID-19 were 78 mg/L and 5.13, respectively. The risk of ICU admission was significantly greater for patients with these cutoff values (CRP hazard ratio = 2.98; 95% CI: 1.58–5.62; NLR hazard ratio = 2.90; 95% CI: 1.45–5.77).

Conclusions

Using time-dependent ROC curves, CRP and NLR values at hospital admission were important predictors of ICU admission. This approach is more efficient than using standard ROC curves.

External validation of the MSKCC nomogram to estimate five-year overall survival after surgery for stage I-III colon cancer in a Dutch population

INTRODUCTION

The Memorial Sloan Kettering Cancer Centre (MSKCC) nomogram has been developed to estimate five-year overall survival (OS) after curative-intent surgery of colon cancer based on age, sex, T stage, differentiation grade, number of positive and examined regional lymph nodes. This is the first evaluation of the performance of the MSKCC model in a European population regarding prediction of OS.

MATERIAL AND METHODS

Population-based data from patients with stage I-III colon cancer diagnosed between 2010 and 2016 were obtained from the Netherlands Cancer Registry (NCR) for external validation of the MSKCC prediction model. Five-year survival probabilities were estimated for all patients in our dataset by using the MSKCC prediction equation. Histogram density plots were created to depict the distribution of the estimated probability and prognostic index. The performance of the model was evaluated in terms of its overall performance, discrimination, and calibration.

RESULTS

A total of 39,805 patients were included. Five-year OS was 71.9% (95% CI 71.5; 72.3) (11,051 events) with a median follow up of 5.6 years (IQR 4.1; 7.7). The Brier score was 0.10 (95% CI 0.10; 0.10). The C-index was 0.75 (95% CI 0.75; 0.76). The calibration measures and plot indicated that the model slightly overestimated observed mortality (observed/expected ratio = 0.86 [95% CI 0.86; 0.87], calibration intercept = -0.14 [95% CI -0.16; -0.11], and slope 1.07 [95% CI 1.05; 1.09], ICI = 0.04, E50 = 0.04, and E90 = 0.05).

CONCLUSIONS

The external validation of the MSKCC prediction nomogram in a large Dutch cohort supports the use of this practical tool in the European patient population. These personalised estimated survival probabilities may support clinicians when informing patients about prognosis. Adding potential relevant prognostic factors to the model, such as primary tumour location, might further improve the model.

Prediction of disability-free survival in healthy older people

Prolonging survival in good health is a fundamental societal goal. However, the leading determinants of disability-free survival in healthy older people have not been well established. Data from ASPREE, a bi-national placebo-controlled trial of aspirin with 4.7 years median follow-up, was analysed. At enrolment, participants were healthy and without prior cardiovascular events, dementia or persistent physical disability. Disability-free survival outcome was defined as absence of dementia, persistent disability or death. Selection of potential predictors from amongst 25 biomedical, psychosocial and lifestyle variables including recognized geriatric risk factors, utilizing a machine-learning approach. Separate models were developed for men and women. The selected predictors were evaluated in a multivariable Cox proportional hazards model and validated internally by bootstrapping. We included 19,114 Australian and US participants aged ≥65 years (median 74 years, IQR 71.6-77.7). Common predictors of a worse prognosis in both sexes included higher age, lower Modified Mini-Mental State Examination score, lower gait speed, lower grip strength and abnormal (low or elevated) body mass index. Additional risk factors for men included current smoking, and abnormal eGFR. In women, diabetes and depression were additional predictors. The biased-corrected areas under the receiver operating characteristic curves for the final prognostic models at 5 years were 0.72 for men and 0.75 for women. Final models showed good calibration between the observed and predicted risks. We developed a prediction model in which age, cognitive function and gait speed were the strongest predictors of disability-free survival in healthy older people.Trial registration Clinicaltrials.gov (NCT01038583).

Dynamic Prediction of Near-Term Overall Survival in Patients with Advanced NSCLC Based on Real-World Data

Simple Summary

Patients near the end of life often receive aggressive care, which may be of low value. For patients with advanced cancers, it is standard clinical practice to estimate the prognosis to inform treatment decisions and improve end-of-life care. However, clinical estimates of prognosis may be imprecise and rapidly become out-of-date if clinical factors that evolve over time are not incorporated. Patient prognosis is commonly estimated based on a clinician’s subjective assessment of patient reserve, such as performance status. We propose a spline-smoothed landmarking approach to dynamically estimate survival probabilities based on objective, evolving patient features. The proposed method allows predictions at any time during the patient disease course and demonstrates dramatically improved prediction accuracy compared to methods using clinical features at a fixed time. The proposed approaches can assist clinicians and patients in appropriately regulating treatments to improve outcomes and quality of life.

Abstract

Patients with terminal cancers commonly receive aggressive and sub-optimal treatment near the end of life, which may not be beneficial in terms of duration or quality of life. To improve end-of-life care, it is essential to develop methods that can accurately predict the short-term risk of death. However, most prediction models for patients with cancer are static in the sense that they only use patient features at a fixed time. We proposed a dynamic prediction model (DPM) that can incorporate time-dependent predictors. We apply this method to patients with advanced non-small-cell lung cancer from a real-world database. Inverse probability of censoring weighted AUC with bootstrap inference was used to compare predictions among models. We found that increasing ECOG performance status and decreasing albumin had negative prognostic associations with overall survival (OS). Moreover, the negative prognostic implications strengthened over the patient disease course. DPMs using both time-independent and time-dependent predictors substantially improved short-term prediction accuracy compared to Cox models using only predictors at a fixed time. The proposed model can be broadly applied for prediction based on longitudinal data, including an estimation of the dynamic effects of time-dependent features on OS and updating predictions at any follow-up time.

Censoring-robust time-dependent receiver operating characteristic curve estimators

Time-dependent receiver operating characteristic curves are often used to evaluate the classification performance of continuous measures when considering time-to-event data. When one is interested in evaluating the predictive performance of multiple covariates, it is common to use the Cox proportional hazards model to obtain risk scores; however, previous work has shown that when the model is mis-specified, the estimand corresponding to the partial likelihood estimator depends on the censoring distribution. In this manuscript, we show that when the risk score model is mis-specified, the AUC will also depend on the censoring distribution, leading to either over- or under-estimation of the risk score's predictive performance. We propose the use of censoring-robust estimators to remove the dependence on the censoring distribution and provide empirical results supporting the use of censoring-robust risk scores.

Receiver operating characteristic (ROC) movies, universal ROC (UROC) curves, and coefficient of predictive ability (CPA)

Throughout science and technology, receiver operating characteristic (ROC) curves and associated area under the curve ($$\mathrm{AUC}$$ AUC ) measures constitute powerful tools for assessing the predictive abilities of features, markers and tests in binary classification problems. Despite its immense popularity, ROC analysis has been subject to a fundamental restriction, in that it applies to dichotomous (yes or no) outcomes only. Here we introduce ROC movies and universal ROC (UROC) curves that apply to just any linearly ordered outcome, along with an associated coefficient of predictive ability ($${\mathrm{CPA}}$$ CPA ) measure. $${\mathrm{CPA}}$$ CPA equals the area under the UROC curve, and admits appealing interpretations in terms of probabilities and rank based covariances. For binary outcomes $${\mathrm{CPA}}$$ CPA equals $$\mathrm{AUC}$$ AUC , and for pairwise distinct outcomes $${\mathrm{CPA}}$$ CPA relates linearly to Spearman’s coefficient, in the same way that the C index relates linearly to Kendall’s coefficient. ROC movies, UROC curves, and $${\mathrm{CPA}}$$ CPA nest and generalize the tools of classical ROC analysis, and are bound to supersede them in a wealth of applications. Their usage is illustrated in data examples from biomedicine and meteorology, where rank based measures yield new insights in the WeatherBench comparison of the predictive performance of convolutional neural networks and physical-numerical models for weather prediction.

Gadobenate dimeglumine-enhanced biliary imaging from the hepatobiliary phase can predict progression in patients with liver cirrhosis

OBJECTIVES

To determine the value of gadobenate dimeglumine (Gd-BOPTA)-enhanced biliary imaging from the hepatobiliary phase in predicting hepatic decompensation and insufficiency for patients with cirrhosis.

METHODS

This single-center retrospective study included 270 patients who underwent Gd-BOPTA-enhanced magnetic resonance imaging. The relative enhancement ratios of the biliary system (REB) and liver parenchyma (REL) in patients with normal liver function without underlying chronic liver disease and three groups of patients with Child-Pugh A, Child-Pugh B, and Child-Pugh C disease were measured. After a mean follow-up of 38.5 ± 22.5 months, prognostic factors were evaluated using the Cox proportional hazards regression model. Receiver operating characteristic (ROC) curve analyses were performed to assess the capacity of the REB and REL to predict the development of hepatic decompensation and insufficiency.

RESULTS

During the follow-up period, nine of 79 patients with Child-Pugh A disease developed hepatic decompensation. The REB was a significant predictive factor (hazard ratio (HR) = 0.40 (0.19-0.84); p = 0.016), but the REL showed no association with hepatic decompensation. Moreover, the areas under the ROC curves (AUCs) were 0.83 and 0.52 for the REB and REL, respectively. Thirty-eight of 207 patients with cirrhosis developed hepatic insufficiency. The REB was a significant predictive factor (HR = 0.24 (0.13-0.46); p < 0.0001), but the REL did not show statistically significant association with hepatic insufficiency. The AUCs were 0.82 and 0.57 for the REB and REL, respectively.

CONCLUSIONS

Gd-BOPTA-enhanced biliary imaging from the hepatobiliary phase was valuable in predicting hepatic decompensation and insufficiency for cirrhotic patients.

KEY POINTS

• Gd-BOPTA-enhanced biliary imaging was a significant predictive factor for hepatic decompensation in patients with cirrhosis. • Gd-BOPTA-enhanced biliary imaging was a significant predictive factor for hepatic insufficiency in patients with cirrhosis. • Gd-BOPTA-enhanced biliary imaging showed superior predictive values for adverse clinical outcomes compared to liver parenchymal imaging at the hepatobiliary phase.

Smoothed time-dependent receiver operating characteristic curve for right censored survival data

The prediction reliability is of primary concern in many clinical studies when the objective is to develop new predictive models or improve existing risk scores. In fact, before using a model in any clinical decision making, it is very important to check its ability to discriminate between subjects who are at risk of, for example, developing certain disease in a near future from those who will not. To that end, the time-dependent receiver operating characteristic (ROC) curve is the most commonly used method in practice. Several approaches have been proposed in the literature to estimate the ROC nonparametrically in the context of survival data. But, except one recent approach, all the existing methods provide a nonsmooth ROC estimator whereas, by definition, the ROC curve is smooth. In this article we propose and study a new nonparametric smooth ROC estimator based on a weighted kernel smoother. More precisely, our approach relies on a well-known kernel method used to estimate cumulative distribution functions of random variables with bounded supports. We derived some asymptotic properties for the proposed estimator. As bandwidth is the main parameter to be set, we present and study different methods to appropriately select one. A simulation study is conducted, under different scenarios, to prove the consistency of the proposed method and to compare its finite sample performance with a competitor. The results show that the proposed method performs better and appear to be quite robust to bandwidth choice. As for inference purposes, our results also reveal the good performances of a proposed nonparametric bootstrap procedure. Furthermore, we illustrate the method using a real data example.

Assessing cure status prediction from survival data using receiver operating characteristic curves

Survival analysis relies on the hypothesis that, if the follow-up is long enough, the event of interest will eventually be observed for all observations. This assumption, however, is often not realistic. The survival data then contain a cure fraction. A common approach to modelling and analysing this type of data consists in using cure models. Two types of information can therefore be obtained: the survival at a given time and the cure status, both possibly modelled as a function of the covariates. The cure status is often of interest to medical practitioners, and one is usually interested in predicting it based on markers. Receiver operating characteristic, Roc, curves are one way to evaluate the predicted performance; however, the classical Roc curve method is not appropriate since the cure status is partially unobserved due to the presence of censoring in survival data. We propose a Roc curve estimator that aims to evaluate the cured/noncured status classification performance from cure survival data. This estimator, which handles the presence of censoring, decomposes sensitivity and specificity by means of the definition of conditional probability, and estimates these two quantities by means of weighted empirical distribution functions. The mixture cure model is used to calculate the weights. Based on simulations, we demonstrate good performance of the proposed method, and compare it with the classical Roc curve nonparametric estimator that would be obtained if the cure status was fully observed. We also compare our proposed method with the Roc curves of Heagerty et al. (2000) for classical survival analysis. Finally, we illustrate the methodology on a breast cancer dataset.

Two-stage receiver operating-characteristic curve estimator for cohort studies

The receiver operating-characteristic (ROC) curve is a graphical statistical tool routinely used for studying the classification accuracy in both, diagnostic and prognosis problems. Given the different nature of these situations, ROC curve estimation has been separately considered for binary (diagnostic) and time-to-event (prognosis) outcomes, even for data coming from the same study design. In this work, the authors propose a two-stage ROC curve estimator which allows to link both contexts through a general prediction model (first-stage) and the empirical cumulative estimator of the distribution function (second-stage) of the considered test (marker) on the total population. The so-called two-stage Mixed-Subject (sMS) approach proves its behavior on both, large-samples (theoretically) and finite-samples (via Monte Carlo simulations). Besides, a useful asymptotic distribution for the concomitant area under the curve is also computed. Results show the ability of the proposed estimator to fit non-standard situations by considering flexible predictive models. Two real-world examples, one with binary and one with time-dependent outcomes, help us to a better understanding of the proposed methodology on usual practical circumstances. The R code used for the practical implementation of the proposed methodology and its documentation is provided as supplementary material.

The area between ROC curves, a non-parametric method to evaluate a biomarker for patient treatment selection

Treatment selection markers are generally sought for when the benefit of an innovative treatment in comparison with a reference treatment is considered, and this benefit is suspected to vary according to the characteristics of the patients. Classically, such quantitative markers are detected through testing a marker-by-treatment interaction in a parametric regression model. Most alternative methods rely on modeling the risk of event occurrence in each treatment arm or the benefit of the innovative treatment over the marker values, but with assumptions that may be difficult to verify. Herein, a simple non-parametric approach is proposed to detect and assess the general capacity of a quantitative marker for treatment selection when no overall difference in efficacy could be demonstrated between two treatments in a clinical trial. This graphical method relies on the area between treatment-arm-specific receiver operating characteristic curves (ABC), which reflects the treatment selection capacity of the marker. A simulation study assessed the inference properties of the ABC estimator and compared them with other parametric and non-parametric indicators. The simulations showed that the estimate of the ABC had low bias, power comparable to parametric indicators, and that its confidence interval had a good coverage probability (better than the other non-parametric indicator in some cases). Thus, the ABC is a good alternative to parametric indicators. The ABC method was applied to data of the PETACC-8 trial that investigated FOLFOX4 versus FOLFOX4 + cetuximab in stage III colon adenocarcinoma. It enabled the detection of a treatment selection marker: the DDR2 gene.

Terminally Differentiated Effector Memory CD8+ T Cells Identify Kidney Transplant Recipients at High Risk of Graft Failure

BACKGROUND

Identifying biomarkers to predict kidney transplant failure and to define new therapeutic targets requires more comprehensive understanding of the immune response to chronic allogeneic stimulation.

METHODS

We investigated the frequency and function of CD8⁺ T cell subsets-including effector memory (EM) and terminally differentiated EM (TEMRA) CD8⁺ T cells-in blood samples from 284 kidney transplant recipients recruited 1 year post-transplant and followed for a median of 8.3 years. We also analyzed CD8⁺ T cell reactivity to donor-specific PBMCs in 24 patients who had received living-donor kidney transplants.

RESULTS

Increased frequency of circulating TEMRA CD8⁺ T cells at 1 year post-transplant associated with increased risk of graft failure during follow-up. This association remained after adjustment for a previously reported composite of eight clinical variables, the Kidney Transplant Failure Score. In contrast, increased frequency of EM CD8⁺ T cells associated with reduced risk of graft failure. A distinct TEMRA CD8⁺ T cell subpopulation was identified that was characterized by expression of FcγRIIIA (CD16) and by high levels of proinflammatory cytokine secretion and cytotoxic activity. Although donor-specific stimulation induced a similar rapid, early response in EM and TEMRA CD8⁺ T cells, CD16 engagement resulted in selective activation of TEMRA CD8⁺ T cells, which mediated antibody-dependent cytotoxicity.

CONCLUSIONS

At 1 year post-transplant, the composition of memory CD8⁺ T cell subsets in blood improved prediction of 8-year kidney transplant failure compared with a clinical-variables score alone. A subpopulation of TEMRA CD8⁺ T cells displays a novel dual mechanism of activation mediated by engagement of the T-cell receptor or of CD16. These findings suggest that TEMRA CD8⁺ T cells play a pivotal role in humoral and cellular rejection and reveal the potential value of memory CD8⁺ T cell monitoring for predicting risk of kidney transplant failure.

Prognostic Value of Lymph Node-To-Primary Tumor Standardized Uptake Value Ratio in Esophageal Squamous Cell Carcinoma Treated with Definitive Chemoradiotherapy

We aimed to investigate the prognostic value of the relative maximum standardized uptake value (SUV) of metastatic lymph node (LN) compared with that of primary tumor (SUV_LN / SUV_Tumor) based on a pretreatment [¹⁸F]-FDG PET/CT scan in patients with clinically node-positive esophageal squamous cell carcinoma (cN+ ESCC) treated with definitive chemoradiotherapy (dCRT). We retrospectively evaluated cN+ ESCC patients who underwent a PET/CT scan before dCRT. Time-dependent receiver operating characteristics analysis was performed to identify the optimal cutoff value for SUV_LN / SUV_Tumor. Prognostic influences of SUV_LN / SUV_Tumor on distant metastasis-free survival (DMFS) and overall survival (OS) were evaluated using the Kaplan-Meier method and log-rank test for univariate analysis and Cox's proportional hazards regression model for multivariate analysis. We identified 112 patients with newly diagnosed cN+ ESCC. After a median follow-up of 32.0 months, 50 (44.6%) patients had distant failure and 84 (75.0%) patients died. Patients with high SUV_LN / SUV_Tumor (≥ 0.39) experienced worse outcomes than low SUV_LN / SUV_Tumor (< 0.39) (two-year DMFS: 26% vs. 70%, p < 0.001; two-year OS: 21% vs. 48%, p = 0.001). Multivariate analysis showed that SUV_LN / SUV_Tumor was an independent prognostic factor for both DMFS (adjusted HR 2.24, 95% CI 1.34-3.75, p = 0.002) and OS (adjusted HR 1.61, 95% CI 1.03-2.53, p = 0.037). Pretreatment of SUV_LN / SUV_Tumor is a simple and useful marker for prognosticating DMFS and OS in cN+ ESCC patients treated with dCRT, which may help in tailoring treatment and designing future clinical trials.

A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes

It is widely recommended that any developed-diagnostic or prognostic-prediction model is externally validated in terms of its predictive performance measured by calibration and discrimination. When multiple validations have been performed, a systematic review followed by a formal meta-analysis helps to summarize overall performance across multiple settings, and reveals under which circumstances the model performs suboptimal (alternative poorer) and may need adjustment. We discuss how to undertake meta-analysis of the performance of prediction models with either a binary or a time-to-event outcome. We address how to deal with incomplete availability of study-specific results (performance estimates and their precision), and how to produce summary estimates of the c-statistic, the observed:expected ratio and the calibration slope. Furthermore, we discuss the implementation of frequentist and Bayesian meta-analysis methods, and propose novel empirically-based prior distributions to improve estimation of between-study heterogeneity in small samples. Finally, we illustrate all methods using two examples: meta-analysis of the predictive performance of EuroSCORE II and of the Framingham Risk Score. All examples and meta-analysis models have been implemented in our newly developed R package "metamisc".

The c-index is not proper for the evaluation of $t$-year predicted risks

We show that the widely used concordance index for time to event outcome is not proper when interest is in predicting a $t$-year risk of an event, for example 10-year mortality. In the situation with a fixed prediction horizon, the concordance index can be higher for a misspecified model than for a correctly specified model. Impropriety happens because the concordance index assesses the order of the event times and not the order of the event status at the prediction horizon. The time-dependent area under the receiver operating characteristic curve does not have this problem and is proper in this context.

Assessment of the 2017 AHA/ACC/HRS Guideline Recommendations for Implantable Cardioverter-Defibrillator Implantation in Cardiac Sarcoidosis

BACKGROUND

Implantable cardioverter-defibrillators are used to prevent sudden cardiac death in patients with cardiac sarcoidosis. The most recent recommendations for implantable cardioverter-defibrillator implantation in these patients are in the 2017 American Heart Association/American College of Cardiology/Heart Rhythm Society Guideline for Management of Patients With Ventricular Arrhythmias and the Prevention of Sudden Cardiac Death. These recommendations, based on observational studies or expert opinion, have not been assessed. We aimed to assess them.

METHODS

We performed a large retrospective cohort study of patients with biopsy-proven sarcoidosis and known or suspected cardiac sarcoidosis that underwent cardiovascular magnetic resonance imaging. Patients were followed for a composite end point of significant ventricular arrhythmia or sudden cardiac death. The discriminatory performance of the Guideline recommendations was tested using time-dependent receiver operating characteristic analyses. The optimal cutoff for the extent of late gadolinium enhancement predictive of the composite end point was determined using the Youden index.

RESULTS

In 290 patients, the class I and IIa recommendations identified all patients who experienced the composite end point during a median follow-up of 3.0 years. Patients meeting class I recommendations had a significantly higher incidence of the composite end point than those meeting class IIa recommendations. Left ventricular ejection fraction (LVEF) >35% with >5.7% late gadolinium enhancement on cardiovascular magnetic resonance imaging was as sensitive as and significantly more specific than LVEF >35% with any late gadolinium enhancement. Patients meeting 2 class IIa recommendations, LVEF >35% with the need for a permanent pacemaker and LVEF >35% with late gadolinium enhancement >5.7%, had high annualized event rates. Excluding 2 class IIa recommendations, LVEF >35% with syncope and LVEF >35% with inducible ventricular arrhythmia, resulted in improved discrimination for the composite end point.

CONCLUSIONS

We assessed the Guideline recommendations for implantable cardioverter-defibrillator implantation in patients with known or suspected cardiac sarcoidosis and identified topics for future research.

On the validity of time-dependent AUC estimation in the presence of cure fraction

During the last decades, several approaches have been proposed to estimate the time-dependent area under the receiver operating characteristic curve (AUC) of risk tools derived from survival data. The validity of these estimators relies on some regularity assumptions among which a survival function being proper. In practice, this assumption is not always satisfied because a fraction of the population may not be susceptible to experience the event of interest even for long follow-up. Studying the sensitivity of the proposed estimators to the violation of this assumption is of substantial interest. In this paper, we investigate the performance of a nonparametric simple estimator, developed for classical survival data, in the case when the population exhibits a cure fraction. Motivated from the current practice of deriving risk tools in oncology and cardiovascular disease prevention, we also assess the loss, in terms of predictive performance, when deriving risk tools from survival models that do not acknowledge the presence of cure. The simulation results show that the investigated method is valid even under the presence of cure. They also show that risk tools derived from survival models that ignore the presence of cure have smaller AUC compared to those derived from survival models that acknowledge the presence of cure. This was also attested with a real data analysis from a breast cancer study.

Association Between Tumor Size Kinetics and Survival in Patients With Urothelial Carcinoma Treated With Atezolizumab: Implication for Patient Follow-Up

We characterized the association between tumor size kinetics and survival in patients with advanced urothelial carcinoma treated with atezolizumab (anti-programmed death-ligand 1, Tecentriq) using a joint model. The model, developed on data from 309 patients of a phase II clinical trial, identified the time-to-tumor growth and the instantaneous changes in tumor size as the best on-treatment predictors of survival. On the validation dataset containing data from 457 patients from a phase III study, the model predicted individual survival probability using 3-month or 6-month tumor size follow-up data with an area under the receptor-occupancy curve between 0.75 and 0.84, as compared with values comprised between 0.62 and 0.75 when the model included only information available at treatment initiation. Including tumor size kinetics in a relevant statistical framework improves the prediction of survival probability during immunotherapy treatment and may be useful to identify most-at-risk patients in "real-time."

Performance of prediction models on survival outcomes of colorectal cancer with surgical resection: A systematic review and meta-analysis

Prediction models allow accurate estimate of individualized prognosis. Increasing numbers of models on survival of CRC patients with surgical resection are being published. However, their performance and potential clinical utility have been unclear. A systematic search in MEDLINE and Embase databases (until 9th April 2018) was performed. Original model development studies and external validation studies predicting any survival outcomes from CRC (follow-up ≥1 year after surgery) were included. We conducted random-effects meta-analyses in external validation studies to estimate the performance of each model. A total of 83 original prediction models and 52 separate external validation studies were identified. We identified five models (Basingstoke score, Fong score, Nordinger score, Peritoneal Surface Disease Severity Score and Valentini nomogram) that were validated in at least two external datasets with a median summarized C-statistic of 0.67 (range: 0.57-0.74). These models can potentially assist clinical decision-making. Besides developing new models, future research should also focus on validating existing prediction models and investigating their real-word impact and cost-effectiveness for CRC prognosis in clinical practice.

A Bayesian method to estimate the optimal threshold of a marker used to select patients' treatment

The use of a quantitative treatment selection marker to choose between two treatment options requires the estimate of an optimal threshold above which one of these two treatments is preferred. Herein, the optimal threshold expression is based on the definition of a utility function which aims to quantify the expected utility of the population (e.g. life expectancy, quality of life) by taking into account both efficacy (success or failure) and toxicity of each treatment option. Therefore, the optimal threshold is the marker value that maximizes the expected utility of the population. A method modelling the marker distribution in patient subgroups defined by the received treatment and the outcome is proposed to calculate the parameters of the utility function so as to estimate the optimal threshold and its 95% credible interval using the Bayesian inference. The simulation study found that the method had low bias and coverage probability close to 95% in multiple settings, but also the need of large sample size to estimate the optimal threshold in some settings. The method is then applied to the PETACC-8 trial that compares the efficacy of chemotherapy with a combined chemotherapy + anti-epidermal growth factor receptor in stage III colorectal cancer.

Quantifying and estimating the predictive accuracy for censored time-to-event data with competing risks

This paper focuses on quantifying and estimating the predictive accuracy of prognostic models for time-to-event outcomes with competing events. We consider the time-dependent discrimination and calibration metrics, including the receiver operating characteristics curve and the Brier score, in the context of competing risks. To address censoring, we propose a unified nonparametric estimation framework for both discrimination and calibration measures, by weighting the censored subjects with the conditional probability of the event of interest given the observed data. The proposed method can be extended to time-dependent predictive accuracy metrics constructed from a general class of loss functions. We apply the methodology to a data set from the African American Study of Kidney Disease and Hypertension to evaluate the predictive accuracy of a prognostic risk score in predicting end-stage renal disease, accounting for the competing risk of pre-end-stage renal disease death, and evaluate its numerical performance in extensive simulation studies.

Dynamic Prediction of Renal Failure Using Longitudinal Biomarkers in a Cohort Study of Chronic Kidney Disease

In longitudinal studies, prognostic biomarkers are often measured longitudinally. It is of both scientific and clinical interest to predict the risk of clinical events, such as disease progression or death, using these longitudinal biomarkers as well as other time-dependent and time-independent information about the patient. The prediction is dynamic in the sense that it can be made at any time during the follow-up, adapting to the changing at-risk population and incorporating the most recent longitudinal data. One approach is to build a joint model of longitudinal predictor variables and time to the clinical event, and draw predictions from the posterior distribution of the time to event conditional on longitudinal history. Another approach is to use the landmark model, which is a system of prediction models that evolve with the follow-up time. We review the pros and cons of the two approaches, and present a general analytical framework using the landmark approach. The proposed framework allows the measurement times of longitudinal data to be irregularly spaced and differ between subjects. We propose a unified kernel weighting approach for estimating the model parameters, calculating predicted probabilities, and evaluating prediction accuracy through double time-dependent Receiver Operating Characteristics (ROC) curves. We illustrate the proposed analytical framework using the African American Study of Kidney Disease and Hypertension (AASK) to develop a landmark model for dynamic prediction of end stage renal diseases or death among patients with chronic kidney disease.

Smooth time-dependent receiver operating characteristic curve estimators

The receiver operating characteristic curve is a popular graphical method often used to study the diagnostic capacity of continuous (bio)markers. When the considered outcome is a time-dependent variable, two main extensions have been proposed: the cumulative/dynamic receiver operating characteristic curve and the incident/dynamic receiver operating characteristic curve. In both cases, the main problem for developing appropriate estimators is the estimation of the joint distribution of the variables time-to-event and marker. As usual, different approximations lead to different estimators. In this article, the authors explore the use of a bivariate kernel density estimator which accounts for censored observations in the sample and produces smooth estimators of the time-dependent receiver operating characteristic curves. The performance of the resulting cumulative/dynamic and incident/dynamic receiver operating characteristic curves is studied by means of Monte Carlo simulations. Additionally, the influence of the choice of the required smoothing parameters is explored. Finally, two real-applications are considered. An R package is also provided as a complement to this article.

A sudden death risk score specifically for hypertension: based on 25 648 individual patient data from six randomized controlled trials

OBJECTIVE

To construct a sudden death risk score specifically for hypertension (HYSUD) patients with or without cardiovascular history.

METHODS

Data were collected from six randomized controlled trials of antihypertensive treatments with 8044 women and 17 604 men differing in age ranges and blood pressure eligibility criteria. In total, 345 sudden deaths (1.35%) occurred during a mean follow-up of 5.16 years. Risk factors of sudden death were examined using a multivariable Cox proportional hazards model adjusted on trials. The model was transformed to an integer system, with points added for each factor according to its association with sudden death risk.

RESULTS

Antihypertensive treatment was not associated with a reduction of the sudden death risk and had no interaction with other factors, allowing model development on both treatment and placebo groups. A risk score of sudden death in 5 years was built with seven significant risk factors: age, sex, SBP, serum total cholesterol, cigarette smoking, diabetes, and history of myocardial infarction. In terms of discrimination performance, HYSUD model was adequate with areas under the receiver operating characteristic curve of 77.74% (confidence interval 95%, 74.13-81.35) for the derivation set, of 77.46% (74.09-80.83) for the validation set, and of 79.17% (75.94-82.40) for the whole population.

CONCLUSION

Our work provides a simple risk-scoring system for sudden death prediction in hypertension, using individual data from six randomized controlled trials of antihypertensive treatments. HYSUD score could help assessing a hypertensive individual's risk of sudden death and optimizing preventive therapeutic strategies for these patients.

Standardized and weighted time-dependent receiver operating characteristic curves to evaluate the intrinsic prognostic capacities of a marker by taking into account confounding factors

Time-dependent receiver operating characteristic curves allow to evaluate the capacity of a marker to discriminate between subjects who experience the event up to a given prognostic time from those who are free of this event. In this article, we propose an inverse probability weighting estimator of a standardized and weighted time-dependent receiver operating characteristic curve. This estimator provides a measure of the prognostic capacities by taking into account potential confounding factors. We illustrate the robustness of the estimator by a simulation-based study and its usefulness by two applications in kidney transplantation.

Time-dependent ROC curve analysis in medical research: current methods and applications

BACKGROUND

ROC (receiver operating characteristic) curve analysis is well established for assessing how well a marker is capable of discriminating between individuals who experience disease onset and individuals who do not. The classical (standard) approach of ROC curve analysis considers event (disease) status and marker value for an individual as fixed over time, however in practice, both the disease status and marker value change over time. Individuals who are disease-free earlier may develop the disease later due to longer study follow-up, and also their marker value may change from baseline during follow-up. Thus, an ROC curve as a function of time is more appropriate. However, many researchers still use the standard ROC curve approach to determine the marker capability ignoring the time dependency of the disease status or the marker.

METHODS

We comprehensively review currently proposed methodologies of time-dependent ROC curves which use single or longitudinal marker measurements, aiming to provide clarity in each methodology, identify software tools to carry out such analysis in practice and illustrate several applications of the methodology. We have also extended some methods to incorporate a longitudinal marker and illustrated the methodologies using a sequential dataset from the Mayo Clinic trial in primary biliary cirrhosis (PBC) of the liver.

RESULTS

From our methodological review, we have identified 18 estimation methods of time-dependent ROC curve analyses for censored event times and three other methods can only deal with non-censored event times. Despite the considerable numbers of estimation methods, applications of the methodology in clinical studies are still lacking.

CONCLUSIONS

The value of time-dependent ROC curve methods has been re-established. We have illustrated the methods in practice using currently available software and made some recommendations for future research.

A simple method to estimate the time-dependent receiver operating characteristic curve and the area under the curve with right censored data

The time-dependent receiver operating characteristic curve is often used to study the diagnostic accuracy of a single continuous biomarker, measured at baseline, on the onset of a disease condition when the disease onset may occur at different times during the follow-up and hence may be right censored. Due to right censoring, the true disease onset status prior to the pre-specified time horizon may be unknown for some patients, which causes difficulty in calculating the time-dependent sensitivity and specificity. We propose to estimate the time-dependent sensitivity and specificity by weighting the censored data by the conditional probability of disease onset prior to the time horizon given the biomarker, the observed time to event, and the censoring indicator, with the weights calculated nonparametrically through a kernel regression on time to event. With this nonparametric weighting adjustment, we derive a novel, closed-form formula to calculate the area under the time-dependent receiver operating characteristic curve. We demonstrate through numerical study and theoretical arguments that the proposed method is insensitive to misspecification of the kernel bandwidth, produces unbiased and efficient estimators of time-dependent sensitivity and specificity, the area under the curve, and other estimands from the receiver operating characteristic curve, and outperforms several other published methods currently implemented in R packages.

Pretreatment count of peripheral neutrophils, monocytes, and lymphocytes as independent prognostic factor in patients with head and neck cancer

BACKGROUND

The purpose of this study was to analyze the prognostic value of pretreatment count of peripheral neutrophils, lymphocytes, monocytes, and neutrophil-to-lymphocyte ratio (NLR) in patients with head and neck squamous cell carcinoma (HNSCC).

METHODS

Local, regional, and distant recurrence-free survival and disease-specific survival were analyzed according to the count of neutrophils, lymphocytes, monocytes, and NLR.

RESULTS

We observed a decrease in disease-specific survival as the quartile category of neutrophils, monocytes, and NLR increased. In the case of lymphocytes, patients in the lower quartile had lower disease-specific survival. Considering the disease-specific survival as the dependent variable, a recursive partitioning analysis classified the patients according to the neutrophil and monocyte counts.

CONCLUSION

High pretreatment count of peripheral neutrophils and/or monocytes was independently related with worse prognosis in patients with HNSCC. Classification based on pretreatment neutrophil and monocyte counts enabled the identification of different prognostic profiles. © 2016 Wiley Periodicals, Inc. Head Neck 39: 219-226, 2017.

A targeted real-time early warning score (TREWScore) for septic shock

Sepsis is a leading cause of death in the United States, with mortality highest among patients who develop septic shock. Early aggressive treatment decreases morbidity and mortality. Although automated screening tools can detect patients currently experiencing severe sepsis and septic shock, none predict those at greatest risk of developing shock. We analyzed routinely available physiological and laboratory data from intensive care unit patients and developed "TREWScore," a targeted real-time early warning score that predicts which patients will develop septic shock. TREWScore identified patients before the onset of septic shock with an area under the ROC (receiver operating characteristic) curve (AUC) of 0.83 [95% confidence interval (CI), 0.81 to 0.85]. At a specificity of 0.67, TREWScore achieved a sensitivity of 0.85 and identified patients a median of 28.2 [interquartile range (IQR), 10.6 to 94.2] hours before onset. Of those identified, two-thirds were identified before any sepsis-related organ dysfunction. In comparison, the Modified Early Warning Score, which has been used clinically for septic shock prediction, achieved a lower AUC of 0.73 (95% CI, 0.71 to 0.76). A routine screening protocol based on the presence of two of the systemic inflammatory response syndrome criteria, suspicion of infection, and either hypotension or hyperlactatemia achieved a lower sensitivity of 0.74 at a comparable specificity of 0.64. Continuous sampling of data from the electronic health records and calculation of TREWScore may allow clinicians to identify patients at risk for septic shock and provide earlier interventions that would prevent or mitigate the associated morbidity and mortality.

Nonparametric estimation of time-dependent ROC curves conditional on a continuous covariate

The receiver-operating characteristic (ROC) curve is the most widely used measure for evaluating the performance of a diagnostic biomarker when predicting a binary disease outcome. The ROC curve displays the true positive rate (or sensitivity) and the false positive rate (or 1-specificity) for different cut-off values used to classify an individual as healthy or diseased. In time-to-event studies, however, the disease status (e.g. death or alive) of an individual is not a fixed characteristic, and it varies along the study. In such cases, when evaluating the performance of the biomarker, several issues should be taken into account: first, the time-dependent nature of the disease status; and second, the presence of incomplete data (e.g. censored data typically present in survival studies). Accordingly, to assess the discrimination power of continuous biomarkers for time-dependent disease outcomes, time-dependent extensions of true positive rate, false positive rate, and ROC curve have been recently proposed. In this work, we present new nonparametric estimators of the cumulative/dynamic time-dependent ROC curve that allow accounting for the possible modifying effect of current or past covariate measures on the discriminatory power of the biomarker. The proposed estimators can accommodate right-censored data, as well as covariate-dependent censoring. The behavior of the estimators proposed in this study will be explored through simulations and illustrated using data from a cohort of patients who suffered from acute coronary syndrome.

Comparison of Pathologic Response Evaluation Systems after Anthracycline with/without Taxane-Based Neoadjuvant Chemotherapy among Different Subtypes of Breast Cancers

Purpose

Several methods are used to assess the pathologic response of breast cancer after neoadjuvant chemotherapy (NAC) to predict clinical outcome. However, the clinical utility of these systems for each molecular subtype of breast cancer is unclear. Therefore, we applied six pathologic response assessment systems to specific subtypes of breast cancer and compared the results.

Patients and Methods

Five hundred and eighty eight breast cancer patients treated with anthracycline with/without taxane-based NAC were retrospectively analyzed, and the ypTNM stage, residual cancer burden (RCB), residual disease in breast and nodes (RDBN), tumor response ratio, Sataloff’s classification, and Miller—Payne grading system were evaluated. The results obtained for each assessment system were analyzed in terms of patient survival.

Results

In triple-negative tumors, all systems were significantly associated with disease-free survival and Kaplan-Meier survival curves for disease-free survival were clearly separated by all assessment methods. For HR+/HER2- tumors, systems assessing the residual tumor (ypTNM stage, RCB, and RDBN) had prognostic significance. However, for HER2+ tumors, the association between patient survival and the pathologic response assessment results varied according to the system used, and none resulted in distinct Kaplan—Meier curves.

Conclusion

Most of the currently available pathologic assessment systems used after anthracycline with/without taxane-based NAC effectively classified triple-negative breast cancers into groups showing different prognoses. The pathologic assessment systems evaluating residual tumors only also had prognostic significance in HR+/HER2- tumors. However, new assessment methods are required to effectively evaluate the pathologic response of HR+/HER2+ and HR-/HER2+ tumors to anthracycline with/without taxane-based NAC.

Receiver operating characteristic curve estimation for time to event with semicompeting risks and interval censoring

Semicompeting risks and interval censoring are frequent in medical studies, for instance when a disease may be diagnosed only at times of visit and disease onset is in competition with death. To evaluate the ability of markers to predict disease onset in this context, estimators of discrimination measures must account for these two issues. In recent years, methods for estimating the time-dependent receiver operating characteristic curve and the associated area under the ROC curve have been extended to account for right censored data and competing risks. In this paper, we show how an approximation allows to use the inverse probability of censoring weighting estimator for semicompeting events with interval censored data. Then, using an illness-death model, we propose two model-based estimators allowing to rigorously handle these issues. The first estimator is fully model based whereas the second one only uses the model to impute missing observations due to censoring. A simulation study shows that the bias for inverse probability of censoring weighting remains modest and may be less than the one of the two parametric estimators when the model is misspecified. We finally recommend the nonparametric inverse probability of censoring weighting estimator as main analysis and the imputation estimator based on the illness-death model as sensitivity analysis.