On comparing 2 correlated C indices with censored survival data

Screening tools for predicting mortality of adults with suspected sepsis: an international sepsis cohort validation study

OBJECTIVES

We evaluated the performance of commonly used sepsis screening tools across prospective sepsis cohorts in the USA, Cambodia and Ghana.

DESIGN

Prospective cohort studies.

SETTING AND PARTICIPANTS

From 2014 to 2021, participants with two or more SIRS (Systemic Inflammatory Response Syndrome) criteria and suspected infection were enrolled in emergency departments and medical wards at hospitals in Cambodia and Ghana and hospitalised participants with suspected infection were enrolled in the USA. Cox proportional hazards regression was performed, and Harrell's C-statistic calculated to determine 28-day mortality prediction performance of the quick Sequential Organ Failure Assessment (qSOFA) score ≥2, SIRS score ≥3, National Early Warning Score (NEWS) ≥5, Modified Early Warning Score (MEWS) ≥5 or Universal Vital Assessment (UVA) score ≥2. Screening tools were compared with baseline risk (age and sex) with the Wald test.

RESULTS

The cohorts included 567 participants (42.9% women) including 187 participants from Kumasi, Ghana, 200 participants from Takeo, Cambodia and 180 participants from Durham, North Carolina in the USA. The pooled mortality was 16.4% at 28 days. The mortality prediction accuracy increased from baseline risk with the MEWS (C-statistic: 0.63, 95% CI 0.58 to 0.68; p=0.002), NEWS (C-statistic: 0.68; 95% CI 0.64 to 0.73; p<0.001), qSOFA (C-statistic: 0.70, 95% CI 0.64 to 0.75; p<0.001), UVA score (C-statistic: 0.73, 95% CI 0.69 to 0.78; p<0.001), but not with SIRS (0.60; 95% CI 0.54 to 0.65; p=0.13). Within individual cohorts, only the UVA score in Ghana performed better than baseline risk (C-statistic: 0.77; 95% CI 0.71 to 0.83; p<0.001).

CONCLUSIONS

Among the cohorts, MEWS, NEWS, qSOFA and UVA scores performed better than baseline risk, largely driven by accuracy improvements in Ghana, while SIRS scores did not improve prognostication accuracy. Prognostication scores should be validated within the target population prior to clinical use.

Discriminatory Ability and Clinical Utility of the AJCC7 and AJCC8 Staging Systems for Breast Cancer in a Middle-Income Setting

(1) Background: Differences in access to biomarker testing and cancer treatment in resource-limited settings may affect the clinical utility of the AJCC8 staging system compared to the anatomical AJCC7 system. (2) Methods: A total of 4151 Malaysian women who were newly diagnosed with breast cancer from 2010 to 2020 were followed-up until December 2021. All patients were staged using the AJCC7 and AJCC8 systems. Overall survival (OS) and relative survival (RS) were determined. Concordance-index was used to compare the discriminatory ability between the two systems. (3) Results: Migration from the AJCC7 to AJCC8 staging system resulted in the downstaging of 1494 (36.0%) patients and the upstaging of 289 (7.0%) patients. Approximately 5% of patients could not be staged using the AJCC8 classification. Five-year OS varied between 97% (Stage IA) and 66% (Stage IIIC) for AJCC7, and 96% (Stage IA) and 60% (Stage IIIC) for AJCC8. Concordance-indexes for predicting OS using the AJCC7 and AJCC8 models were 0.720 (0.694-0.747) and 0.745 (0.716-0.774), and for predicting RS they were 0.692 (0.658-0.728) and 0.710 (0.674-0.748), respectively. (4) Conclusions: Given the comparable discriminatory ability between the two staging systems in predicting the stage-specific survival of women with breast cancer in the current study, the continued use of the AJCC7 staging system in resource-limited settings seems pragmatic and justifiable.

Validation of the Meet-URO score in patients with metastatic renal cell carcinoma receiving first-line nivolumab and ipilimumab in the Italian Expanded Access Program

BACKGROUND

The Meet-URO score allowed a more accurate prognostication than the International Metastatic RCC Database Consortium (IMDC) for patients with pre-treated metastatic renal cell carcinoma (mRCC) by adding the pre-treatment neutrophil-to-lymphocyte ratio and presence of bone metastases.

MATERIALS AND METHODS

A post hoc analysis was carried out to validate the Meet-URO score on the overall survival (OS) of patients with IMDC intermediate-poor-risk mRCC treated with first-line nivolumab plus ipilimumab within the prospective Italian Expanded Access Programme (EAP). We additionally considered progression-free survival (PFS) and disease response rates. Harrell's c-index was calculated to compare the accuracy of survival prediction.

RESULTS

Overall the EAP included 306 patients, with a median follow-up of 12.2 months, median OS was not reached, 1-year OS was 66.8% and median PFS was 7.9 months. By univariable analysis, both the IMDC score and the two additional variables of the Meet-URO score were associated with either OS or PFS (P < 0.001 for all comparisons). The four Meet-URO risk groups (G) had 1-year OS of 92%, 72%, 50% and 21% for G2 (29.1% of patients), G3 (28.8%), G4 (33.0%) and G5 (9.1%), respectively. OS was significantly shorter in each consecutive G (P = 0.001 for G3, P < 0.001 for both G4 and G5 compared to G2). Similarly, Meet-URO Gs 2-5 showed decreasing median PFS and response rates. The Meet-URO score showed the highest c-index for both OS (0.73) and PFS (0.67). Limitations include the post hoc nature of this analysis and the lack of a comparative arm to assess predictive value.

CONCLUSION

The Meet-URO score appeared to show better prognostic classification than the IMDC alone in patients with mRCC at IMDC intermediate-poor risk treated with first-line nivolumab and ipilimumab.

Development of a Spine X-Ray-Based Fracture Prediction Model Using a Deep Learning Algorithm

BACKGRUOUND

Since image-based fracture prediction models using deep learning are lacking, we aimed to develop an X-ray-based fracture prediction model using deep learning with longitudinal data.

METHODS

This study included 1,595 participants aged 50 to 75 years with at least two lumbosacral radiographs without baseline fractures from 2010 to 2015 at Seoul National University Hospital. Positive and negative cases were defined according to whether vertebral fractures developed during follow-up. The cases were divided into training (n=1,416) and test (n=179) sets. A convolutional neural network (CNN)-based prediction algorithm, DeepSurv, was trained with images and baseline clinical information (age, sex, body mass index, glucocorticoid use, and secondary osteoporosis). The concordance index (C-index) was used to compare performance between DeepSurv and the Fracture Risk Assessment Tool (FRAX) and Cox proportional hazard (CoxPH) models.

RESULTS

Of the total participants, 1,188 (74.4%) were women, and the mean age was 60.5 years. During a mean follow-up period of 40.7 months, vertebral fractures occurred in 7.5% (120/1,595) of participants. In the test set, when DeepSurv learned with images and clinical features, it showed higher performance than FRAX and CoxPH in terms of C-index values (DeepSurv, 0.612; 95% confidence interval [CI], 0.571 to 0.653; FRAX, 0.547; CoxPH, 0.594; 95% CI, 0.552 to 0.555). Notably, the DeepSurv method without clinical features had a higher C-index (0.614; 95% CI, 0.572 to 0.656) than that of FRAX in women.

CONCLUSION

DeepSurv, a CNN-based prediction algorithm using baseline image and clinical information, outperformed the FRAX and CoxPH models in predicting osteoporotic fracture from spine radiographs in a longitudinal cohort.

A nonparametric approach to confidence intervals for concordance index and difference between correlated indices

Concordance refers to the probability that subjects with high values on one variable also have high values on another variable. This index has wide application in practice, as a measure of effect size in group-comparison studies, an index of accuracy in diagnostic studies, and a discrimination index for prediction models. Herein, we provide a unified framework for statistical inference involving concordance indices for standard variables of binary, ordinal, and continuous types. In particular, we develop confidence interval procedures for a single concordance index and differences between two correlated indices. Simulation results show that procedures based on logit-transformation for a single index and Fisher's z-transformation for a difference between indices perform very well in terms of coverage and tail errors even when the sample size is as small as 30, unless the concordance is high and the standard is a binary variable for which at least 50 subjects are needed. We illustrate the procedures for a variety of standard variables with previously published data. Illustrative SAS code is provided.

Circulating Cystatin C Is an Independent Risk Marker for Cardiovascular Outcomes, Development of Renal Impairment, and Long-Term Mortality in Patients With Stable Coronary Heart Disease: The LIPID Study

Background

Elevated plasma cystatin C levels reflect reduced renal function and increased cardiovascular risk. Less is known about whether the increased risk persists long‐term or is independent of renal function and other important biomarkers.

Methods and Results

Cystatin C and other biomarkers were measured at baseline (in 7863 patients) and 1 year later (in 6106 patients) in participants in the LIPID (Long‐Term Intervention with Pravastatin in Ischemic Disease) study, who had a previous acute coronary syndrome. Outcomes were ascertained during the study (median follow‐up, 6 years) and long‐term (median follow‐up, 16 years). Glomerular filtration rate (GFR) was estimated using Chronic Kidney Disease Epidemiology Collaboration equations (first GFR‐creatinine, then GFR‐creatinine‐cystatin C). Over 6 years, in fully adjusted multivariable time‐to‐event models, with respect to the primary end point of coronary heart disease mortality or nonfatal myocardial infarction, for comparison of Quartile 4 versus 1 of baseline cystatin C, the hazard ratio was 1.37 (95% CI, 1.07–1.74; P=0.01), and for major cardiovascular events was 1.47 (95% CI, 1.19–1.82; P<0.001). Over 16 years, the association of baseline cystatin C with coronary heart disease, cardiovascular, and all‐cause mortality persisted (each P<0.001) and remained significant after adjustment for estimated GFR‐creatinine‐cystatin C. Cystatin C also predicted the development of chronic kidney disease for 6 years (odds ratio, 6.61; 95% CI, 4.28–10.20) independently of estimated GFR‐creatinine and other risk factors. However, this association was no longer significant after adjustment for estimated GFR‐creatinine‐cystatin C.

Conclusions

Cystatin C independently predicted major cardiovascular events, development of chronic kidney disease, and cardiovascular and all‐cause mortality. Prediction of long‐term mortality was independent of improved estimation of GFR.

Registration

URL: https://anzctr.org.au; Unique identifier: ACTRN12616000535471.

Associations Between Clinical Outcomes and a Recently Proposed Adult Congenital Heart Disease Anatomic and Physiological Classification System

Background

American Heart Association and American College of Cardiology consensus guidelines introduce an adult congenital heart disease anatomic and physiological (AP) classification system. We assessed the association between AP classification and clinical outcomes.

Methods and Results

Data were collected for 1000 outpatients with ACHD prospectively enrolled between 2012 and 2019. AP classification was assigned based on consensus definitions. Primary outcomes were (1) all‐cause mortality and (2) a composite of all‐cause mortality or nonelective cardiovascular hospitalization. Cox regression models were developed for AP classification, each component variable, and additional clinical models. Discrimination was assessed using the Harrell C statistic. Over a median follow‐up of 2.5 years (1.4–3.9 years), the composite outcome occurred in 185 participants, including 49 deaths. Moderately or severely complex anatomic class (class II/III) and severe physiological stage (stage D) had increased risk of the composite outcome (AP class IID and IIID hazard ratio, 4.46 and 3.73, respectively, versus IIC). AP classification discriminated moderately between patients who did and did not suffer the composite outcome (C statistic, 0.69 [95% CI, 0.67–0.71]), similar to New York Heart Association functional class and NT‐proBNP (N‐terminal pro‐B‐type natriuretic peptide); it was more strongly associated with mortality (C statistic, 0.81 [95% CI, 0.78–0.84]), as were NT‐proBNP and functional class. A model with AP class and NT‐proBNP provided the strongest discrimination for the composite outcome (C statistic, 0.73 [95% CI, 0.71–0.75]) and mortality (C statistic, 0.85 [95% CI, 0.82–0.88]).

Conclusions

The addition of physiological stage modestly improves the discriminative ability of a purely anatomic classification, but simpler approaches offer equivalent prognostic information. The AP system may be improved by addition of key variables, such as circulating biomarkers, and by avoiding categorization of continuous variables.

The ROC of Cox proportional hazards cure models with application in cancer studies

With recent advancement in cancer screening and treatment, many patients with cancers are identified at early stage and clinically cured. Importantly, uncured patients should be treated timely before the cancer progresses to advanced stages for which therapeutic options are rather limited. It is also crucial to identify uncured subjects among patients with early-stage cancers for clinical trials to develop effective adjuvant therapies. Thus, it is of interest to develop statistical predictive models with as high accuracy as possible in predicting the latent cure status. The receiver operating characteristic curve (ROC) and the area under the ROC curve (AUC) are among the most widely used statistical metrics for assessing predictive accuracy or discriminatory power for a dichotomous outcome (cured/uncured). Yet the conventional AUC cannot be directly used due to incompletely observed cure status. In this article, we proposed new estimates of the ROC curve and its AUC for predicting latent cure status in Cox proportional hazards (PH) cure models and transformation cure models. We developed explicit formulas to estimate sensitivity, specificity, the ROC and its AUC without requiring to know the patient cure status. We also developed EM type estimates to approximate sensitivity, specificity, ROC and AUC conditional on observed data. Numerical studies were used to assess their finite-sample performance of the proposed methods. Both methods are consistent and have similar efficiency as shown in our numerical studies. A melanoma dataset was used to demonstrate the utility of the proposed estimates of the ROC curve for the latent cure status. We also have developed an [Formula: see text] package called [Formula: see text] to efficiently compute the proposed estimates.

A numerical strategy to evaluate performance of predictive scores via a copula-based approach

Assessing and comparing the performance of correlated predictive scores are of current interest in precision medicine. Given the limitations of available theoretical approaches for assessing and comparing the predictive accuracy, numerical methods are highly desired which, however, have not been systematically developed due to technical challenges. The main challenges include the lack of a general strategy on effectively simulating many kinds of correlated predictive scores each with some given level of predictive accuracy in either concordance index or the area under a receiver operating characteristic curve area under the curves (AUC). To fill in this important knowledge gap, this paper is to provide a general copula-based numeric framework for assessing and comparing predictive performance of correlated predictive or risk scores. The new algorithms are designed to effectively simulate correlated predictive scores with given levels of predictive accuracy as measured in terms of concordance indices or time-dependent AUC for predicting survival outcomes. The copula-based numerical strategy is convenient for numerically evaluating and comparing multiple measures of predictive accuracy of correlated risk scores and for investigating finite-sample properties of test statistics and confidence intervals as well as assessing for optimism of given performance measures using cross-validation or bootstrap.

Clinical Significance of Papillary Muscles on Left Ventricular Mass Quantification Using Cardiac Magnetic Resonance Imaging: Reproducibility and Prognostic Value in Fabry Disease

PURPOSE

Accurate and reproducible assessment of left ventricular mass (LVM) is important in Fabry disease. However, it is unclear whether papillary muscles should be included in LVM assessed by cardiac magnetic resonance imaging (MRI). The purpose of this study was to evaluate the reproducibility and predictive value of LVM in patients with Fabry disease using different analysis approaches.

MATERIALS AND METHODS

A total of 92 patients (44±15 y, 61 women) with confirmed Fabry disease who had undergone cardiac MRI at a single tertiary referral hospital were included in this retrospective study. LVM was assessed at end-diastole using 2 analysis approaches, including and excluding papillary muscles. Adverse cardiac events were assessed as a composite end point, defined as ventricular tachycardia, bradycardia requiring device implantation, severe heart failure, and cardiac death. Statistical analysis included Cox proportional hazard models, Akaike information criterion, intraclass correlation coefficients, and Bland-Altman analysis.

RESULTS

Left ventricular end-diastolic volume, end-systolic volume, ejection fraction, and LVM all differed significantly between analysis approaches. LVM was significantly higher when papillary muscles were included versus excluded (157±71 vs. 141±62 g, P<0.001). Mean papillary mass was 16±11 g, accounting for 10%±3% of total LVM. LVM with pap illary muscles excluded had slightly better predictive value for the composite end point compared with LVM with papillary muscles included based on the model goodness-of-fit (Akaike information criterion 140 vs. 142). Interobserver agreement was slightly better for LVM with papillary muscles excluded compared with included (intraclass correlation coefficient 0.993 [95% confidence interval: 0.985, 0.996] vs. 0.989 [95% confidence interval: 0.975, 0.995]) with less bias and narrower limits of agreement.

CONCLUSIONS

Inclusion or exclusion of papillary muscles has a significant effect on LVM quantified by cardiac MRI, and therefore, a standardized analysis approach should be used for follow-up. Exclusion of papillary muscles from LVM is a reasonable approach in patients with Fabry disease given slightly better predictive value and reproducibility.

Multi-Parametric Deep Learning Model for Prediction of Overall Survival after Postoperative Concurrent Chemoradiotherapy in Glioblastoma Patients

This study aimed to investigate the performance of a deep learning-based survival-prediction model, which predicts the overall survival (OS) time of glioblastoma patients who have received surgery followed by concurrent chemoradiotherapy (CCRT). The medical records of glioblastoma patients who had received surgery and CCRT between January 2011 and December 2017 were retrospectively reviewed. Based on our inclusion criteria, 118 patients were selected and semi-randomly allocated to training and test datasets (3:1 ratio, respectively). A convolutional neural network–based deep learning model was trained with magnetic resonance imaging (MRI) data and clinical profiles to predict OS. The MRI was reconstructed by using four pulse sequences (22 slices) and nine images were selected based on the longest slice of glioblastoma by a physician for each pulse sequence. The clinical profiles consist of personal, genetic, and treatment factors. The concordance index (C-index) and integrated area under the curve (iAUC) of the time-dependent area-under-the-curve curves of each model were calculated to evaluate the performance of the survival-prediction models. The model that incorporated clinical and radiomic features showed a higher C-index (0.768 (95% confidence interval (CI): 0.759, 0.776)) and iAUC (0.790 (95% CI: 0.783, 0.797)) than the model using clinical features alone (C-index = 0.693 (95% CI: 0.685, 0.701); iAUC = 0.723 (95% CI: 0.716, 0.731)) and the model using radiomic features alone (C-index = 0.590 (95% CI: 0.579, 0.600); iAUC = 0.614 (95% CI: 0.607, 0.621)). These improvements to the C-indexes and iAUCs were validated using the 1000-times bootstrapping method; all were statistically significant (p < 0.001). This study suggests the synergistic benefits of using both clinical and radiomic parameters. Furthermore, it indicates the potential of multi-parametric deep learning models for the survival prediction of glioblastoma patients.

Geriatric assessment domains to predict overall survival in older cancer patients: An analysis of functional status, comorbidities, and nutritional status as prognostic factors

Cancer treatments for older patients must account for heterogeneity in health and functional status. Guidelines advocate the use of geriatric assessments (GAs), but comprehensive assessments are laborious and the utility of specific GA domains remains unclear. The identification of specific domains as prognostic factors may support survival predictions and treatment decisions. We aimed to evaluate the associations between several GA domains and overall survival in older cancer patients. We linked cancer registry data and administrative claims data from cancer patients residing in Osaka Prefecture, Japan. The subjects were patients aged ≥70 years who received a diagnosis of gastric, colorectal, or lung cancer between 2010 and 2014 at 36 designated cancer care hospitals. The following three GA domains were assessed at cancer diagnosis: functional status through activities of daily living (ADL), comorbidities, and nutritional status through body mass index. Cox proportional hazards models were constructed for the three cancer types to estimate each domain's prognostic effect while adjusting for gender, age, and cancer stage. Adjusted hazard ratios (HRs) for all-cause mortality were calculated. We identified 5,559, 4,746, and 4,837 patients with gastric, colorectal, and lung cancer respectively. ADL impairment (HRs: 1.39-3.34, 1.64-2.86, and 1.24-3.21 for gastric, colorectal, and lung cancer, respectively), comorbidities (1.32-1.58, 1.33-1.97, and 1.19-1.29 for gastric, colorectal, and lung cancer, respectively), and underweight (1.36, 1.51, and 1.54 for gastric, colorectal, and lung cancer, respectively) were significantly associated with poorer overall survival. In contrast, overweight was significantly associated with improved overall survival (HRs: 0.82 and 0.89 for gastric and lung cancer respectively). The addition of the three domains increased the models' C-statistics (0.816 to 0.836, 0.764 to 0.787, and 0.759 to 0.783 for gastric, colorectal, and lung cancer respectively). Incorporating these factors into initial patient evaluations during diagnosis may aid prognostic predictions and treatment strategies in geriatric oncology.

Data for a population based cohort study on abnormal findings of electrocardiograms (ECG), recorded during follow-up periodic examinations, and their association with long-term cardiovascular morbidity and all-cause mortality

In this Data in Brief article, we provide data of the cohort and statistical methods of the research- "Incidental abnormal ECG findings and long-term cardiovascular morbidity and all-cause mortality: a population based prospective study" (Goldman et al., 2019). Extended description of statistical analysis as well as data of cohort baseline characteristics and baseline ECG incidental abnormal findings of 2601 Israeli men and women without known cardiovascular disease (CVD) is presented. The cohort is part of the Israel study of Glucose Intolerance, Obesity and Hypertension (GOH) (Dankner et al., 2007). Furthermore, we provide the data on the performance assessment of the 23 - year CVD-risk and the 31- year all-cause mortality prediction models, which includes Receiver Operating Characteristic (ROC) curves, reclassification-based measures and calibration curve.

Application of Concordance Probability Estimate to Predict Conversion from Mild Cognitive Impairment to Alzheimer's Disease

Subjects with mild cognitive impairment (MCI) have a substantially increased risk of developing dementia due to Alzheimer's disease (AD). Identifying MCI subjects who have high progression risk to AD is important in clinical management. Existing risk prediction models of AD among MCI subjects generally use either the AUC or Harrell's C-statistic to evaluate predictive accuracy. AUC is aimed at binary outcome and Harrell's C-statistic depends on the unknown censoring distribution. Gönen & Heller's K-index, also known as concordance probability estimate (CPE), is another measure of overall predictive accuracy for Cox proportional hazards (PH) models, which does not depend on censoring distribution. As a comprehensive example, using Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset, we built a Cox PH model to predict the conversion from MCI to AD where the prognostic accuracy was evaluated using K-index.