A Tutorial on Evaluating the Time-Varying Discrimination Accuracy of Survival Models Used in Dynamic Decision Making

Contribution of body mass index, waist circumference, and 25-OH-D3 on the risk of pre-diabetes mellitus in the Chinese population

This study aimed to investigate the associations between body mass index (BMI), waist circumference (WC), 25-hydroxy-vitamin D3 (25-OH-D3), and the risk of pre-diabetes mellitus (PDM), as well as their predictive values in identifying PDM. A total of 1688 participants were included in this cross-sectional investigation. Spearman's correlation analysis was used to assess the relationships between candidate indicators and PDM. The impact of indicators on PDM risk was determined by multivariate logistic regression. The receiver operating characteristic (ROC) analysis was performed to evaluate the prognostic value of indicators. Our study indicated a positive correlation between WC, BMI, and 25-OH-D3 and PDM. WC (OR = 1.05, 95% CI = 1.04-1.06, p < 0.001), BMI (OR = 1.11, 95% CI = 1.08-1.15, p < 0.001), and 25-OH-D3 (OR = 1.01, 95% CI = 1.00-1.02, p = 0.037) and an increased risk of PDM. Additionally, the ROC analysis demonstrated that WC (AUC = 0.651, Specificity = 55.00%, Sensitivity = 67.900%) had a higher diagnostic value for predicting PDM compared to the other variables (BMI, 25-OH-D3, TG, TC, LDL-C, HDL-C, and UA). A cut-off value of WC > 80.5 cm predicted PDM with both good sensitivity and specificity. Additionally, the cut-off value of waist circumference (WC) for men with prediabetes was 86.500, while for women with prediabetes, it was 76.500.

Improving pressure injury risk assessment using real-world data from skilled nursing facilities: A cohort study

This study aimed to improve the predictive accuracy of the Braden assessment for pressure injury risk in skilled nursing facilities (SNFs) by incorporating real-world data and training a survival model. A comprehensive analysis of 126 384 SNF stays and 62 253 in-house pressure injuries was conducted using a large calibrated wound database. This study employed a time-varying Cox Proportional Hazards model, focusing on variations in Braden scores, demographic data and the history of pressure injuries. Feature selection was executed through a forward-backward process to identify significant predictive factors. The study found that sensory and moisture Braden subscores were minimally contributive and were consequently discarded. The most significant predictors of increased pressure injury risk were identified as a recent (within 21 days) decrease in Braden score, low subscores in nutrition, friction and activity, and a history of pressure injuries. The model demonstrated a 10.4% increase in predictive accuracy compared with traditional Braden scores, indicating a significant improvement. The study suggests that disaggregating Braden scores and incorporating detailed wound histories and demographic data can substantially enhance the accuracy of pressure injury risk assessments in SNFs. This approach aligns with the evolving trend towards more personalized and detailed patient care. These findings propose a new direction in pressure injury risk assessment, potentially leading to more effective and individualized care strategies in SNFs. The study highlights the value of large-scale data in wound care, suggesting its potential to enhance quantitative approaches for pressure injury risk assessment and supporting more accurate, data-driven clinical decision-making.

Prognoses Associated With Palliative Performance Scale Scores in Modern Palliative Care Practice

Importance

The Palliative Performance Scale (PPS) is one of the most widely used prognostic tools for patients with serious illness. However, current prognostic estimates associated with PPS scores are based on data that are over a decade old.

Objective

To generate updated prognostic estimates by PPS score, care setting, and illness category, and examine how well PPS predicts short- and longer-term survival.

Design, Setting, and Participants

This prognostic study was conducted at a large academic medical center with robust inpatient and outpatient palliative care practices using electronic health record data linked with data from California Vital Records. Eligible participants included patients who received a palliative care consultation between January 1, 2018, and December 31, 2020. Data analysis was conducted from November 2022 to February 2024.

Exposure

Palliative care consultation with a PPS score documented.

Main Outcomes and Measures

The primary outcomes were predicted 1-, 6-, and 12-month mortality and median survival of patients by PPS score in the inpatient and outpatient settings, and performance of the PPS across a range of survival times. In subgroup analyses, mortality risk by PPS score was estimated in patients with cancer vs noncancer illnesses and those seen in-person vs by video telemedicine in the outpatient setting.

Results

Overall, 4779 patients (mean [SD] age, 63.5 [14.8] years; 2437 female [51.0%] and 2342 male [49.0%]) had a palliative care consultation with a PPS score documented. Of these patients, 2276 were seen in the inpatient setting and 3080 were seen in the outpatient setting. In both the inpatient and outpatient settings, 1-, 6-, and 12-month mortality were higher and median survival was shorter for patients with lower PPS scores. Prognostic estimates associated with PPS scores were substantially longer (2.3- to 11.7-fold) than previous estimates commonly used by clinicians. The PPS had good ability to discriminate between patients who lived and those who died in the inpatient setting (integrated time-dependent area under the curve [iAUC], 0.74) but its discriminative ability was lower in the outpatient setting (iAUC, 0.67). The PPS better predicted 1-month survival than longer-term survival. Mortality rates were higher for patients with cancer than other serious illnesses at most PPS levels.

Conclusions and Relevance

In this prognostic study, prognostic estimates associated with PPS scores were substantially longer than previous estimates commonly used by clinicians. Based on these findings, an online calculator was updated to assist clinicians in reaching prognostic estimates that are more consistent with modern palliative care practice and specific to the patient's setting and diagnosis group.

Development and Standardization of an Osteoradionecrosis Classification System in Head and Neck Cancer: Implementation of a Risk-Based Model

PURPOSE

Osteoradionecrosis of the jaw (ORN) can manifest in varying severity. The aim of this study is to identify ORN risk factors and develop a novel classification to depict the severity of ORN.

METHODS

Consecutive patients with head and neck cancer (HNC) treated with curative-intent intensity-modulated radiation therapy (IMRT) (≥45 Gy) from 2011 to 2017 were included. Occurrence of ORN was identified from in-house prospective dental and clinical databases and charts. Multivariable logistic regression model was used to identify risk factors and stratify patients into high-risk and low-risk groups. A novel ORN classification system was developed to depict ORN severity by modifying existing systems and incorporating expert opinion. The performance of the novel system was compared with 15 existing systems for their ability to identify and predict serious ORN event (jaw fracture or requiring jaw resection).

RESULTS

ORN was identified in 219 of 2,732 (8%) consecutive patients with HNC. Factors associated with high risk of ORN were oral cavity or oropharyngeal primaries, received IMRT dose ≥60 Gy, current/ex-smokers, and/or stage III to IV periodontal condition. The ORN rate for high-risk versus low-risk patients was 12.7% versus 3.1% (P < .001) with an AUC of 0.71. Existing ORN systems overclassified serious ORN events and failed to recognize maxillary ORN. A novel ORN classification system, ClinRad, was proposed on the basis of vertical extent of bone necrosis and presence/absence of exposed bone/fistula. This system detected serious ORN events in 5.7% of patients and statistically outperformed existing systems.

CONCLUSION

We identified risk factors for ORN and proposed a novel ORN classification system on the basis of vertical extent of bone necrosis and presence/absence of exposed bone/fistula. It outperformed existing systems in depicting the seriousness of ORN and may facilitate clinical care and clinical trials.

Mortality prediction and influencing factors for intensive care unit patients with acute tubular necrosis: random survival forest and cox regression analysis

Background: Patients with acute tubular necrosis (ATN) not only have severe renal failure, but also have many comorbidities, which can be life-threatening and require timely treatment. Identifying the influencing factors of ATN and taking appropriate interventions can effectively shorten the duration of the disease to reduce mortality and improve patient prognosis. Methods: Mortality prediction models were constructed by using the random survival forest (RSF) algorithm and the Cox regression. Next, the performance of both models was assessed by the out-of-bag (OOB) error rate, the integrated brier score, the prediction error curve, and area under the curve (AUC) at 30, 60 and 90 days. Finally, the optimal prediction model was selected and the decision curve analysis and nomogram were established. Results: RSF model was constructed under the optimal combination of parameters (mtry = 10, nodesize = 88). Vasopressors, international normalized ratio (INR)_min, chloride_max, base excess_min, bicarbonate_max, anion gap_min, and metastatic solid tumor were identified as risk factors that had strong influence on mortality in ATN patients. Uni-variate and multivariate regression analyses were used to establish the Cox regression model. Nor-epinephrine, vasopressors, INR_min, severe liver disease, and metastatic solid tumor were identified as important risk factors. The discrimination and calibration ability of both predictive models were demonstrated by the OOB error rate and the integrated brier score. However, the prediction error curve of Cox regression model was consistently lower than that of RSF model, indicating that Cox regression model was more stable and reliable. Then, Cox regression model was also more accurate in predicting mortality of ATN patients based on the AUC at different time points (30, 60 and 90 days). The analysis of decision curve analysis shows that the net benefit range of Cox regression model at different time points is large, indicating that the model has good clinical effectiveness. Finally, a nomogram predicting the risk of death was created based on Cox model. Conclusion: The Cox regression model is superior to the RSF algorithm model in predicting mortality of patients with ATN. Moreover, the model has certain clinical utility, which can provide clinicians with some reference basis in the treatment of ATN and contribute to improve patient prognosis.

Personalised follow-up and management schema for patients with screen-detected pulmonary nodules: A dynamic modelling study

BACKGROUND

Selecting the time target for follow-up testing in lung cancer screening is challenging. We aim to devise dynamic, personalized lung cancer screening schema for patients with pulmonary nodules detected through low-dose computed tomography.

METHODS

We developed and validated dynamic models using data of pulmonary nodule patients (aged 55-74 years) from the National Lung Screening Trial. We predicted patient-specific risk profiles at baseline (R0) and updated the risk evaluation results in repeated screening rounds (R1 and R2). We used risk cutoffs to optimize time-dependent sensitivity at an early decision point (3 months) and time-dependent specificity at a late decision point (1 year).

RESULTS

In validation, area under receiver operating characteristic curve for predicting 12-month lung cancer onset was 0.867 (95 % confidence interval: 0.827-0.894) and 0.807 (0.765-0.948) at R0 and R1-R2, respectively. The personalized schema, compared with National Comprehensive Cancer Network (NCCN) guideline and Lung-RADS, yielded lower rates of delayed diagnosis (1.7% vs. 1.7% vs. 6.9 %) and over-testing (4.9% vs. 5.6% vs. 5.6 %) at R0, and lower rates of delayed diagnosis (0.0% vs. 18.2% vs. 18.2 %) and over-testing (2.6% vs. 8.3% vs. 7.3 %) at R2. Earlier test recommendation among cancer patients was more frequent using the personalized schema (vs. NCCN: 29.8% vs. 20.9 %, p = 0.0065; vs. Lung-RADS: 33.2% vs. 22.8 %, p = 0.0025), especially for women, patients aged ≥65 years, and part-solid or non-solid nodules.

CONCLUSIONS

The personalized schema is easy-to-implement and more accurate compared with rule-based protocols. The results highlight value of personalized approaches in realizing efficient nodule management.

Modeling multiple correlated end-organ disease trajectories: A tutorial for multistate and joint models with applications in diabetes complications

State-of-the-art biostatistics methods allow for the simultaneous modeling of several correlated non-fatal disease processes over time, but there is no clear guidance on the optimal analysis in most settings. An example occurs in diabetes, where it is not known with certainty how microvascular complications of the eyes, kidneys, and nerves co-develop over time. In this article, we propose and contrast two general model frameworks for studying complications (sequential state and parallel trajectory frameworks) and review multivariate methods for their analysis, focusing on multistate and joint modeling. We illustrate these methods in a tutorial format using the long-term follow-up from the Diabetes Control and Complications Trial and Epidemiology of Diabetes Interventions and Complications study public data repository. A formal comparison of prediction error and discrimination is included. Multistate models are particularly advantageous for determining the order and timing of complications, but require discretization of the longitudinal outcomes and possibly a very complex state space process. Intermittent observation of the states must be accounted for, and discretization is a probable disadvantage in this setting. In contrast, joint models can account for variations of continuous biomarkers over time and are particularly designed for modeling complex association structures between the complications and for performing dynamic predictions of an outcome of interest to inform clinical decisions (eg, a late-stage complication). We found that both models have helpful features that can better-inform our understanding of the complex trajectories that complications may take and can therefore help with decision making for patients presenting with diabetes complications.

Development and Validation of a Risk Score Predicting Death Without Transplant in Adult Heart Transplant Candidates

Importance

The US heart allocation system prioritizes medically urgent candidates with a high risk of dying without transplant. The current therapy-based 6-status system is susceptible to manipulation and has limited rank ordering ability.

Objective

To develop and validate a candidate risk score that incorporates current clinical, laboratory, and hemodynamic data.

Design, Setting, and Participants

A registry-based observational study of adult heart transplant candidates (aged ≥18 years) from the US heart allocation system listed between January 1, 2019, and December 31, 2022, split by center into training (70%) and test (30%) datasets. Adult candidates were listed between January 1, 2019, and December 31, 2022.

Main Outcomes and Measures

A US candidate risk score (US-CRS) model was developed by adding a predefined set of predictors to the current French Candidate Risk Score (French-CRS) model. Sensitivity analyses were performed, which included intra-aortic balloon pumps (IABP) and percutaneous ventricular assist devices (VAD) in the definition of short-term mechanical circulatory support (MCS) for the US-CRS. Performance of the US-CRS model, French-CRS model, and 6-status model in the test dataset was evaluated by time-dependent area under the receiver operating characteristic curve (AUC) for death without transplant within 6 weeks and overall survival concordance (c-index) with integrated AUC.

Results

A total of 16 905 adult heart transplant candidates were listed (mean [SD] age, 53 [13] years; 73% male; 58% White); 796 patients (4.7%) died without a transplant. The final US-CRS contained time-varying short-term MCS (ventricular assist-extracorporeal membrane oxygenation or temporary surgical VAD), the log of bilirubin, estimated glomerular filtration rate, the log of B-type natriuretic peptide, albumin, sodium, and durable left ventricular assist device. In the test dataset, the AUC for death within 6 weeks of listing for the US-CRS model was 0.79 (95% CI, 0.75-0.83), for the French-CRS model was 0.72 (95% CI, 0.67-0.76), and 6-status model was 0.68 (95% CI, 0.62-0.73). Overall c-index for the US-CRS model was 0.76 (95% CI, 0.73-0.80), for the French-CRS model was 0.69 (95% CI, 0.65-0.73), and 6-status model was 0.67 (95% CI, 0.63-0.71). Classifying IABP and percutaneous VAD as short-term MCS reduced the effect size by 54%.

Conclusions and Relevance

In this registry-based study of US heart transplant candidates, a continuous multivariable allocation score outperformed the 6-status system in rank ordering heart transplant candidates by medical urgency and may be useful for the medical urgency component of heart allocation.

Development and Standardization of a Classification System for Osteoradionecrosis: Implementation of a Risk-Based Model

Purpose

Osteoradionecrosis of the jaw (ORN) can manifest in varying severity. The aim of this study is to identify ORN risk factors and develop a novel classification to depict the severity of ORN.

Methods

Consecutive head-and-neck cancer (HNC) patients treated with curative-intent IMRT (≥ 45Gy) in 2011-2018 were included. Occurrence of ORN was identified from in-house prospective dental and clinical databases and charts. Multivariable logistic regression model was used to identify risk factors and stratify patients into high-risk and low-risk groups. A novel ORN classification system was developed to depict ORN severity by modifying existing systems and incorporating expert opinion. The performance of the novel system was compared to fifteen existing systems for their ability to identify and predict serious ORN event (jaw fracture or requiring jaw resection).

Results

ORN was identified in 219 out of 2732 (8%) consecutive HNC patients. Factors associated with high-risk of ORN were: oral-cavity or oropharyngeal primaries, received IMRT dose ≥60Gy, current/ex-smokers, and/or stage III-IV periodontal disease. The ORN rate for high-risk vs low-risk patients was 12.7% vs 3.1% (p<0.001) with an area-under-the-receiver-operating-curve (AUC) of 0.71. Existing ORN systems overclassified serious ORN events and failed to recognize maxillary ORN. A novel ORN classification system, RadORN, was proposed based on vertical extent of bone necrosis and presence/absence of exposed bone/fistula. This system detected serious ORN events in 5.7% of patients and statistically outperformed existing systems.

Conclusion

We identified risk factors for ORN, and proposed a novel ORN classification system based on vertical extent of bone necrosis and presence/absence of exposed bone/fistula. It outperformed existing systems in depicting the seriousness of ORN, and may facilitate clinical care and clinical trials.

Using Machine Learning to Leverage Biomarker Change and Predict Colorectal Cancer Recurrence

PURPOSE

The risk of colorectal cancer (CRC) recurrence after primary treatment varies across individuals and over time. Using patients' most up-to-date information, including carcinoembryonic antigen (CEA) biomarker profiles, to predict risk could improve personalized decision making.

METHODS

We used electronic health record data from an integrated health system on a cohort of patients diagnosed with American Joint Committee on Cancer stage I-III CRC between 2008 and 2013 (N = 3,970) and monitored until recurrence or end of follow-up. We addressed missingness in recurrence outcomes and longitudinal CEA measures, and engineered CEA features using current and past biomarker values for inclusion in a risk prediction model. We used a discrete time Superlearner model to evaluate various algorithms for predicting recurrence. We evaluated the time-varying discrimination and calibration of the algorithms and assessed the role of individual predictors.

RESULTS

Recurrence was documented in 448 (11.3%) patients. XGBoost with depth = 1 (XGB-D1) predicted recurrence substantially better than all other algorithms at all time points, with AUC ranging from 0.87 (95% CI, 0.86 to 0.88) at 6 months to 0.94 (95% CI, 0.92 to 0.96) at 54 months. The only variable used by XGB-D1 was 6-month change in log CEA. Predicted 1-year risk of recurrence was nearly zero for patients whose log CEA did not increase in the last 6 months, between 12.2% and 34.1% for patients whose log CEA increased between 0.10 and 0.40, and 43.6% for those with a log CEA increase >0.40. Compared with XGB, penalized regression approaches (lasso, ridge, and elastic net) performed poorly, with AUCs ranging from 0.58 to 0.69.

CONCLUSION

A flexible, machine learning approach that incorporated longitudinal CEA information yielded a simple and high-performing model for predicting recurrence on the basis of 6-month change in log CEA.

Evaluation of Plasma Biomarkers to Predict Major Adverse Kidney Events in Hospitalized Patients With COVID-19

RATIONALE & OBJECTIVE

Patients hospitalized with COVID-19 are at increased risk for major adverse kidney events (MAKE). We sought to identify plasma biomarkers predictive of MAKE in patients hospitalized with COVID-19.

STUDY DESIGN

Prospective cohort study.

SETTING & PARTICIPANTS

A total of 576 patients hospitalized with COVID-19 between March 2020 and January 2021 across 3 academic medical centers.

EXPOSURE

Twenty-six plasma biomarkers of injury, inflammation, and repair from first available blood samples collected during hospitalization.

OUTCOME

MAKE, defined as KDIGO stage 3 acute kidney injury (AKI), dialysis-requiring AKI, or mortality up to 60 days.

ANALYTICAL APPROACH

Cox proportional hazards regression to associate biomarker level with MAKE. We additionally applied the least absolute shrinkage and selection operator (LASSO) and random forest regression for prediction modeling and estimated model discrimination with time-varying C index.

RESULTS

The median length of stay for COVID-19 hospitalization was 9 (IQR, 5-16) days. In total, 95 patients (16%) experienced MAKE. Each 1 SD increase in soluble tumor necrosis factor receptor 1 (sTNFR1) and sTNFR2 was significantly associated with an increased risk of MAKE (adjusted HR [AHR], 2.30 [95% CI, 1.86-2.85], and AHR, 2.26 [95% CI, 1.73-2.95], respectively). The C index of sTNFR1 alone was 0.80 (95% CI, 0.78-0.84), and the C index of sTNFR2 was 0.81 (95% CI, 0.77-0.84). LASSO and random forest regression modeling using all biomarkers yielded C indexes of 0.86 (95% CI, 0.83-0.89) and 0.84 (95% CI, 0.78-0.91), respectively.

LIMITATIONS

No control group of hospitalized patients without COVID-19.

CONCLUSIONS

We found that sTNFR1 and sTNFR2 are independently associated with MAKE in patients hospitalized with COVID-19 and can both also serve as predictors for adverse kidney outcomes.

PLAIN-LANGUAGE SUMMARY

Patients hospitalized with COVID-19 are at increased risk for long-term adverse health outcomes, but not all patients suffer long-term kidney dysfunction. Identification of patients with COVID-19 who are at high risk for adverse kidney events may have important implications in terms of nephrology follow-up and patient counseling. In this study, we found that the plasma biomarkers soluble tumor necrosis factor receptor 1 (sTNFR1) and sTNFR2 measured in hospitalized patients with COVID-19 were associated with a greater risk of adverse kidney outcomes. Along with clinical variables previously shown to predict adverse kidney events in patients with COVID-19, both sTNFR1 and sTNFR2 are also strong predictors of adverse kidney outcomes.

Opportunistic detection of type 2 diabetes using deep learning from frontal chest radiographs

Deep learning (DL) models can harness electronic health records (EHRs) to predict diseases and extract radiologic findings for diagnosis. With ambulatory chest radiographs (CXRs) frequently ordered, we investigated detecting type 2 diabetes (T2D) by combining radiographic and EHR data using a DL model. Our model, developed from 271,065 CXRs and 160,244 patients, was tested on a prospective dataset of 9,943 CXRs. Here we show the model effectively detected T2D with a ROC AUC of 0.84 and a 16% prevalence. The algorithm flagged 1,381 cases (14%) as suspicious for T2D. External validation at a distinct institution yielded a ROC AUC of 0.77, with 5% of patients subsequently diagnosed with T2D. Explainable AI techniques revealed correlations between specific adiposity measures and high predictivity, suggesting CXRs' potential for enhanced T2D screening.

Copula modeling for the estimation of measures of marker classification and predictiveness performance with survival outcomes

The discriminative and predictive power of a continuous-valued marker for survival outcomes can be summarized using the receiver operating characteristic and predictiveness curves, respectively. In this paper, fully parametric and semi-parametric copula-based constructions of the joint model of the marker and the survival time are developed for characterizing, plotting, and analyzing both curves along with other underlying performance measures. The formulations require a copula function, a parametric specification for the margin of the marker, and either a parametric distribution or a non-parametric estimator for the margin of the time to event, to respectively characterize the fully parametric and semi-parametric joint models. Estimation is carried out using maximum likelihood and a two-stage procedure for the parametric and semi-parametric models, respectively. Resampling-based methods are used for computing standard errors and confidence bounds for the various parameters, curves, and associated measures. Graphical inspection of residuals from each conditional distribution is employed as a guide for choosing a copula from a set of candidates. The performance of the estimators of various classification and predictiveness measures is assessed in simulation studies, assuming different copula and censoring scenarios. The methods are illustrated with the analysis of two markers using the familiar primary biliary cirrhosis data set.

Quantifying and Interpreting the Prediction Accuracy of Models for the Time of a Cardiovascular Event-Moving Beyond C Statistic: A Review

Importance

For personalized or stratified medicine, it is critical to establish a reliable and efficient prediction model for a clinical outcome of interest. The goal is to develop a parsimonious model with fewer predictors for broad future application without compromising predictability. A general approach is to construct various empirical models via individual patients' specific baseline characteristics/biomarkers and then evaluate their relative merits. When the outcome of interest is the timing of a cardiovascular event, a commonly used metric to assess the adequacy of the fitted models is based on C statistics. These measures quantify a model's ability to separate those who develop events earlier from those who develop them later or not at all (discrimination), but they do not measure how closely model estimates match observed outcomes (prediction accuracy). Metrics that provide clinically interpretable measures to quantify prediction accuracy are needed.

Observations

C statistics measure the concordance between the risk scores derived from the model and the observed event time observations. However, C statistics do not quantify the model prediction accuracy. The integrated Brier Score, which calculates the mean squared distance between the empirical cumulative event-free curve and its individual patient's counterparts, estimates the prediction accuracy, but it is not clinically intuitive. A simple alternative measure is the average distance between the observed and predicted event times over the entire study population. This metric directly quantifies the model prediction accuracy and has often been used to evaluate the goodness of fit of the assumed models in settings other than survival data. This time-scale measure is easier to interpret than the C statistics or the Brier score.

Conclusions and Relevance

This article enhances our understanding of the model selection/evaluation process with respect to prediction accuracy. A simple, intuitive measure for quantifying such accuracy beyond C statistics can improve the reliability and efficiency of the selected model for personalized and stratified medicine.

Platelet RNA sequencing for cancer screening in patients with unprovoked venous thromboembolism: a prospective cohort study

BACKGROUND

Platelet RNA sequencing has been shown to accurately detect cancer in previous studies.

OBJECTIVES

To compare the diagnostic accuracy of platelet RNA sequencing with standard-of-care limited cancer screening in patients with unprovoked venous thromboembolism (VTE).

METHODS

Patients aged ≥40 years with unprovoked VTE were recruited at 13 centers and followed for 12 months for cancer. Participants underwent standard-of-care limited cancer screening, and platelet RNA sequencing analysis was performed centrally at study end for cases and selected controls. Sensitivity and specificity were calculated, using the predefined primary positivity threshold of 0.54 for platelet RNA sequencing aiming at 86% test sensitivity, and an additional predefined threshold of 0.89 aiming at 99% test specificity.

RESULTS

A total of 476 participants were enrolled, of whom 25 (5.3%) were diagnosed with cancer during 12-month follow-up. For each cancer patient, 3 cancer-free patients were randomly selected for the analysis. The sensitivity of limited screening was 72% (95% CI, 52-86) at a specificity of 91% (95% CI, 82-95). The area under the receiver operator characteristic for platelet RNA sequencing was 0.54 (95% CI, 0.41-0.66). At the primary positivity threshold, all patients had a positive test, for a sensitivity estimated at 100% (95% CI, 87-99) and a specificity of 8% (95% CI, 3.7-16.4). At the secondary threshold, sensitivity was 68% (95% CI, 48-83; p value compared with limited screening 0.71) at a specificity of 36% (95% CI, 26-47).

CONCLUSION

Platelet RNA sequencing had poor diagnostic accuracy for detecting occult cancer in patients with unprovoked VTE with the current algorithm.

Development and External Validation of a Mortality Prediction Model for Community-Dwelling Older Adults With Dementia

Importance

Estimating mortality risk in older adults with dementia is important for guiding decisions such as cancer screening, treatment of new and chronic medical conditions, and advance care planning.

Objective

To develop and externally validate a mortality prediction model in community-dwelling older adults with dementia.

Design, Setting, and Participants

This cohort study included community-dwelling participants (aged ≥65 years) in the Health and Retirement Study (HRS) from 1998 to 2016 (derivation cohort) and National Health and Aging Trends Study (NHATS) from 2011 to 2019 (validation cohort).

Exposures

Candidate predictors included demographics, behavioral/health factors, functional measures (eg, activities of daily living [ADL] and instrumental activities of daily living [IADL]), and chronic conditions.

Main Outcomes and Measures

The primary outcome was time to all-cause death. We used Cox proportional hazards regression with backward selection and multiple imputation for model development. Model performance was assessed by discrimination (integrated area under the receiver operating characteristic curve [iAUC]) and calibration (plots of predicted and observed mortality).

Results

Of 4267 participants with probable dementia in HRS, the mean (SD) age was 82.2 (7.6) years, 2930 (survey-weighted 69.4%) were female, and 785 (survey-weighted 12.1%) identified as Black. Median (IQR) follow-up time was 3.9 (2.0-6.8) years, and 3466 (81.2%) participants died by end of follow-up. The final model included age, sex, body mass index, smoking status, ADL dependency count, IADL difficulty count, difficulty walking several blocks, participation in vigorous physical activity, and chronic conditions (cancer, heart disease, diabetes, lung disease). The optimism-corrected iAUC after bootstrap internal validation was 0.76 (95% CI, 0.75-0.76) with time-specific AUC of 0.73 (95% CI, 0.70-0.75) at 1 year, 0.75 (95% CI, 0.73-0.77) at 5 years, and 0.84 (95% CI, 0.82-0.85) at 10 years. On external validation in NHATS (n = 2404), AUC was 0.73 (95% CI, 0.70-0.76) at 1 year and 0.74 (95% CI, 0.71-0.76) at 5 years. Calibration plots suggested good calibration across the range of predicted risk from 1 to 10 years.

Conclusions and Relevance

We developed and externally validated a mortality prediction model in community-dwelling older adults with dementia that showed good discrimination and calibration. The mortality risk estimates may help guide discussions regarding treatment decisions and advance care planning.

Genetic Determinants of Survival in Parkinson's Disease in the Asian Population

BACKGROUND

Patients with Parkinson's disease (PD) have reduced life expectancy compared to the general population. Genetic variation was shown to play a role in the heterogeneity of survival for patients with PD, although the underlying genetic background remains poorly studied.

OBJECTIVE

The aim was to explore the genetic determinants influencing the survival of PD.

METHODS

We performed a genome-wide association analysis using a Cox proportional hazards model in a longitudinal cohort of 1080 Chinese patients with PD. Furthermore, we built a clinical-genetic model to predict the survival of patients using clinical variables combined with polygenic risk score (PRS) of survival of PD.

RESULTS

The cohort was followed up for an average of 7.13 years, with 85 incidents of death. One locus rs12628329 (RPL3) was significantly associated with reduced survival time by ~10.8 months (P = 2.72E-08, β = 1.79, standard error = 0.32). Functional exploration suggested this variant could upregulate the expression of RPL3 and induce apoptosis and cell death. In addition, adding PRS of survival in the prediction model substantially improved survival predictability (concordance index [Cindex]: 0.936) compared with the clinical model (Cindex: 0.860).

CONCLUSIONS

These findings improve the current understanding of the genetic cause of survival of PD and provide a novel target RPL3 for further research on PD pathogenesis and potential therapeutic options. Our results also demonstrate the potential utility of PRS of survival in identifying patients with shorter survival and providing personalized clinical monitoring and treatment. © 2022 International Parkinson and Movement Disorder Society.

Performance assessment across different care settings of a heart failure hospitalisation risk-score for type 2 diabetes using administrative claims

Predicting the risk of cardiovascular complications, in particular heart failure hospitalisation (HHF), can improve the management of type 2 diabetes (T2D). Most predictive models proposed so far rely on clinical data not available at the higher Institutional level. Therefore, it is of interest to assess the risk of HHF in people with T2D using administrative claims data only, which are more easily obtainable and could allow public health systems to identify high-risk individuals. In this paper, the administrative claims of > 175,000 patients with T2D were used to develop a new risk score for HHF based on Cox regression. Internal validation on the administrative data cohort yielded satisfactory results in terms of discrimination (max AUROC = 0.792, C-index = 0.786) and calibration (Hosmer-Lemeshow test p value < 0.05). The risk score was then tested on data gathered from two independent centers (one diabetes outpatient clinic and one primary care network) to demonstrate its applicability to different care settings in the medium-long term. Thanks to the large size and broad demographics of the administrative dataset used for training, the proposed model was able to predict HHF without significant performance loss concerning bespoke models developed within each setting using more informative, but harder-to-acquire clinical variables.

A Scalable Risk Scoring System for COVID-19 Inpatients Based on Consumer-grade Wearables: Statistical Analysis and Model Development

Background

To provide effective care for inpatients with COVID-19, clinical practitioners need systems that monitor patient health and subsequently allow for risk scoring. Existing approaches for risk scoring in patients with COVID-19 focus primarily on intensive care units (ICUs) with specialized medical measurement devices but not on hospital general wards.

Objective

In this paper, we aim to develop a risk score for inpatients with COVID-19 in general wards based on consumer-grade wearables (smartwatches).

Methods

Patients wore consumer-grade wearables to record physiological measurements, such as the heart rate (HR), heart rate variability (HRV), and respiration frequency (RF). Based on Bayesian survival analysis, we validated the association between these measurements and patient outcomes (ie, discharge or ICU admission). To build our risk score, we generated a low-dimensional representation of the physiological features. Subsequently, a pooled ordinal regression with time-dependent covariates inferred the probability of either hospital discharge or ICU admission. We evaluated the predictive performance of our developed system for risk scoring in a single-center, prospective study based on 40 inpatients with COVID-19 in a general ward of a tertiary referral center in Switzerland.

Results

First, Bayesian survival analysis showed that physiological measurements from consumer-grade wearables are significantly associated with patient outcomes (ie, discharge or ICU admission). Second, our risk score achieved a time-dependent area under the receiver operating characteristic curve (AUROC) of 0.73-0.90 based on leave-one-subject-out cross-validation.

Conclusions

Our results demonstrate the effectiveness of consumer-grade wearables for risk scoring in inpatients with COVID-19. Due to their low cost and ease of use, consumer-grade wearables could enable a scalable monitoring system.

Trial Registration

Clinicaltrials.gov NCT04357834; https://www.clinicaltrials.gov/ct2/show/NCT04357834

Kidney injury after lung transplantation: Long-term mortality predicted by post-operative day-7 serum creatinine and few clinical factors

BACKGROUND

Acute kidney injury (AKI) after lung transplantation (LuTx) is associated with increased long-term mortality. In this prospective observational study, commonly used AKI-definitions were examined regarding prediction of long-term mortality and compared to simple use of the serum creatinine value at day 7 for patients who did not receive hemodialysis, and serum creatinine value immediately before initiation of hemodialysis (d7/preHD-sCr).

METHODS

185 patients with LuTx were prospectively enrolled from 2013-2014 at our center. Kidney injury was assessed within 7 days by: (1) the Kidney Disease Improving Global Outcomes criteria (KDIGO-AKI), (2) the Acute Disease Quality Initiative 16 Workgroup classification (ADQI-AKI) and (3) d7/preHD-sCr. Prediction of all-cause mortality was examined by Cox regression analysis, and clinical as well as laboratory factors for impaired kidney function post-LuTx were analyzed.

RESULTS

AKI according to KDIGO and ADQI-AKI occurred in 115 patients (62.2%) within 7 days after LuTx. Persistent ADQI-AKI, KDIGO-AKI stage 3 and higher d7/preHD-sCr were associated with higher mortality in the univariable analysis. In the multivariable analysis, d7/preHD-sCr in combination with body weight and intra- and postoperative platelet transfusions predicted mortality after LuTx with similar performance as models using KDIGO-AKI and ADQI-AKI (concordance index of 0.75 for d7/preHD-sCr vs., 0.74 and 0.73, respectively). Pre-transplant reduced renal function, diabetes, higher BMI, and intraoperative ECMO predicted higher d7/preHD-sCr (r2 = 0.354, p < 0.001).

CONCLUSION

Our results confirm the importance of AKI in lung transplant patients; however, a simple and pragmatic indicator of renal function, d7/preHD-sCr, predicts long-term mortality equally reliable as more complex AKI-definitions like KDIGO and ADQI.

A Framework for Using Real-World Data and Health Outcomes Modeling to Evaluate Machine Learning-Based Risk Prediction Models

OBJECTIVES

We propose a framework of health outcomes modeling with dynamic decision making and real-world data (RWD) to evaluate the potential utility of novel risk prediction models in clinical practice. Lung transplant (LTx) referral decisions in cystic fibrosis offer a complex case study.

METHODS

We used longitudinal RWD for a cohort of adults (n = 4247) from the Cystic Fibrosis Foundation Patient Registry to compare outcomes of an LTx referral policy based on machine learning (ML) mortality risk predictions to referral based on (1) forced expiratory volume in 1 second (FEV₁) alone and (2) heterogenous usual care (UC). We then developed a patient-level simulation model to project number of patients referred for LTx and 5-year survival, accounting for transplant availability, organ allocation policy, and heterogenous treatment effects.

RESULTS

Only 12% of patients (95% confidence interval 11%-13%) were referred for LTx over 5 years under UC, compared with 19% (18%-20%) under FEV₁ and 20% (19%-22%) under ML. Of 309 patients who died before LTx referral under UC, 31% (27%-36%) would have been referred under FEV₁ and 40% (35%-45%) would have been referred under ML. Given a fixed supply of organs, differences in referral time did not lead to significant differences in transplants, pretransplant or post-transplant deaths, or overall survival in 5 years.

CONCLUSIONS

Health outcomes modeling with RWD may help to identify novel ML risk prediction models with high potential real-world clinical utility and rule out further investment in models that are unlikely to offer meaningful real-world benefits.

Prognostic Significance of Urinary Biomarkers in Patients Hospitalized With COVID-19

RATIONALE & OBJECTIVE

Acute kidney injury (AKI) is common in patients with coronavirus disease 2019 (COVID-19) and associated with poor outcomes. Urinary biomarkers have been associated with adverse kidney outcomes in other settings and may provide additional prognostic information in patients with COVID-19. We investigated the association between urinary biomarkers and adverse kidney outcomes among patients hospitalized with COVID-19.

STUDY DESIGN

Prospective cohort study.

SETTING & PARTICIPANTS

Patients hospitalized with COVID-19 (n=153) at 2 academic medical centers between April and June 2020.

EXPOSURE

19 urinary biomarkers of injury, inflammation, and repair.

OUTCOME

Composite of KDIGO (Kidney Disease: Improving Global Outcomes) stage 3 AKI, requirement for dialysis, or death within 60 days of hospital admission. We also compared various kidney biomarker levels in the setting of COVID-19 versus other common AKI settings.

ANALYTICAL APPROACH

Time-varying Cox proportional hazards regression to associate biomarker level with composite outcome.

RESULTS

Out of 153 patients, 24 (15.7%) experienced the primary outcome. Twofold higher levels of neutrophil gelatinase-associated lipocalin (NGAL) (HR, 1.34 [95% CI, 1.14-1.57]), monocyte chemoattractant protein (MCP-1) (HR, 1.42 [95% CI, 1.09-1.84]), and kidney injury molecule 1 (KIM-1) (HR, 2.03 [95% CI, 1.38-2.99]) were associated with highest risk of sustaining primary composite outcome. Higher epidermal growth factor (EGF) levels were associated with a lower risk of the primary outcome (HR, 0.61 [95% CI, 0.47-0.79]). Individual biomarkers provided moderate discrimination and biomarker combinations improved discrimination for the primary outcome. The degree of kidney injury by biomarker level in COVID-19 was comparable to other settings of clinical AKI. There was evidence of subclinical AKI in COVID-19 patients based on elevated injury biomarker level in patients without clinical AKI defined by serum creatinine.

LIMITATIONS

Small sample size with low number of composite outcome events.

CONCLUSIONS

Urinary biomarkers are associated with adverse kidney outcomes in patients hospitalized with COVID-19 and may provide valuable information to monitor kidney disease progression and recovery.

Model-Based ROC Curve: Examining the Effect of Case Mix and Model Calibration on the ROC Plot

Background

The performance of risk prediction models is often characterized in terms of discrimination and calibration. The receiver-operating characteristic (ROC) curve is widely used for evaluating model discrimination. However, when comparing ROC curves across different samples, the effect of case mix makes the interpretation of discrepancies difficult. Further, compared with model discrimination, evaluating model calibration has not received the same level of attention. Current methods for examining model calibration require specification of smoothing or grouping factors.

Methods

We introduce the “model-based” ROC curve (mROC) to assess model calibration and the effect of case mix during external validation. The mROC curve is the ROC curve that should be observed if the prediction model is calibrated in the external population. We show that calibration-in-the-large and the equivalence of mROC and ROC curves are together sufficient conditions for the model to be calibrated. Based on this, we propose a novel statistical test for calibration that, unlike current methods, does not require any subjective specification of smoothing or grouping factors.

Results

Through a stylized example, we demonstrate how mROC separates the effect of case mix and model miscalibration when externally validating a risk prediction model. We present the results of simulation studies that confirm the properties of the new calibration test. A case study on predicting the risk of acute exacerbations of chronic obstructive pulmonary disease puts the developments in a practical context. R code for the implementation of this method is provided.

Conclusion

mROC can easily be constructed and used to interpret the effect of case mix and calibration on the ROC plot. Given the popularity of ROC curves among applied investigators, this framework can further promote assessment of model calibration.

Highlights

Mortality prediction in chronic obstructive pulmonary disease and obstructive sleep apnea

BACKGROUND

We aimed to assess mortality in chronic obstructive pulmonary disease (COPD), obstructive sleep apnea (OSA), and overlap syndrome, and evaluate which polysomnographic indices-apnea-hypopnea index (AHI) or hypoxemic load measurements-better predict mortality within 10 years.

METHODS

Adults with symptoms suggestive of sleep apnea and airway disease who underwent both polysomnography and spirometry plus bronchodilator response tests between 2000 and 2018 were included and divided into four groups according to presence of COPD and moderate-to-severe OSA (AHI ≥15/h). We estimated mortality using a Cox model adjusted for demographic/anthropometric covariates and comorbidities; this was called clinical model. To evaluate prognostic performance, we compared the concordance index (C-index) between clinical model and extended models, which incorporated one of polysomnographic indices-AHI, sleep time spent with SpO₂ < 90% (TS90), and mean and lowest SpO₂.

RESULTS

Among 355 participants, patients with COPD alone (57/355, 16.1%) and COPD-OSA overlap syndrome (37/355, 10.4%) had increased all-cause mortality than those who had neither disease (152/355, 42.8%) (adjusted HR, 2.98 and 3.19, respectively). The C-indices of extended models with TS90 (%) and mean SpO₂ were significantly higher than that of clinical model (0.765 vs. 0.737 and 0.756 vs. 0.737, respectively; all P < 0.05); however, the C-index of extended model with AHI was not (0.739 vs. 0.737; P = 0.15).

CONCLUSIONS

In this cohort with symptoms of sleep apnea and airway disease, patients with overlap syndrome had increased mortality, but not higher than in those with COPD alone. The measurement of hypoxemic load, not AHI, better predicted mortality.

A Deep Learning Approach to Predict Diabetes' Cardiovascular Complications From Administrative Claims

People with diabetes require lifelong access to healthcare services to delay the onset of complications. Their disease management processes generate great volumes of data across several domains, from clinical to administrative. Difficulties in accessing and processing these data hinder their secondary use in an institutional setting, even for highly desirable applications, such as the prediction of cardiovascular disease, the main driver of excess mortality in diabetes. Hence, in the present work, we propose a deep learning model for the prediction of major adverse cardiovascular events (MACE), developed and validated using the administrative claims of 214,676 diabetic patients of the Veneto region, in North East Italy. Specifically, we use a year of pharmacy and hospitalisation claims, together with basic patient's information, to predict the 4P-MACE composite endpoint, i.e., the first occurrence of death, heart failure, myocardial infarction, or stroke, with a variable prediction horizon of 1 to 5 years. Adapting to the time-to-event nature of this task, we cast our problem as a multi-outcome (4P-MACE and components), multi-label (1 to 5 years) classification task with a custom loss to account for the effect of censoring. Our model, purposefully specified to minimise data preparation costs, exhibits satisfactory performance in predicting 4P-MACE at all prediction horizons: AUROC from 0.812 (C.I.: 0.797 - 0.827) to 0.792 (C.I.: 0.781 - 0.802); C-index from 0.802 (C.I.: 0.788 - 0.816) to 0.770 (C.I.: 0.761 - 0.779). Components' prediction performance is also adequate, ranging from death's 0.877 1-year AUROC to stroke's 0.689 5-year AUROC.

Corneal Confocal Microscopy Predicts the Development of Diabetic Neuropathy: A Longitudinal Diagnostic Multinational Consortium Study

OBJECTIVE

Corneal nerve fiber length (CNFL) has been shown in research studies to identify diabetic peripheral neuropathy (DPN). In this longitudinal diagnostic study, we assessed the ability of CNFL to predict the development of DPN.

RESEARCH DESIGN AND METHODS

From a multinational cohort of 998 participants with type 1 and type 2 diabetes, we studied the subset of 261 participants who were free of DPN at baseline and completed at least 4 years of follow-up for incident DPN. The predictive validity of CNFL for the development of DPN was determined using time-dependent receiver operating characteristic (ROC) curves.

RESULTS

A total of 203 participants had type 1 and 58 had type 2 diabetes. Mean follow-up time was 5.8 years (interquartile range 4.2-7.0). New-onset DPN occurred in 60 participants (23%; 4.29 events per 100 person-years). Participants who developed DPN were older and had a higher prevalence of type 2 diabetes, higher BMI, and longer duration of diabetes. The baseline electrophysiology and corneal confocal microscopy parameters were in the normal range but were all significantly lower in participants who developed DPN. The time-dependent area under the ROC curve for CNFL ranged between 0.61 and 0.69 for years 1-5 and was 0.80 at year 6. The optimal diagnostic threshold for a baseline CNFL of 14.1 mm/mm² was associated with 67% sensitivity, 71% specificity, and a hazard ratio of 2.95 (95% CI 1.70-5.11; P < 0.001) for new-onset DPN.

CONCLUSIONS

CNFL showed good predictive validity for identifying patients at higher risk of developing DPN ∼6 years in the future.

Comparison of model fit and discriminatory ability of the 8th edition of the tumor-node-metastasis classification and the 9th edition of the Japanese classification to identify stage III colorectal cancer

BACKGROUND

The most widely accepted staging system for colorectal cancer (CRC) is the tumor-node-metastasis (TNM) classification. In Japan, the Japanese Classification of Colorectal, Appendiceal, and Anal Carcinoma (JCCRC) system is used. The two systems differ mainly in relation to tumor deposits (TD) and metastasis in the regional lymph nodes along the main feeding arteries and lateral pelvic lymph nodes (N3). Here, we investigated the prognostic ability of the two systems for stage III CRC.

METHODS

We reviewed 696 consecutive patients who underwent curative resection of stage III CRC at the National Cancer Center Hospital between May 2007 and April 2014. We examined the clinicopathological features of CRC and predicted overall survival (OS) and relapse-free survival (RFS) according to the 8th TNM and 9th JCCRC systems. The systems were compared using Akaike's information criterion (AIC), Harrell's concordance index (C-index), and time-dependent receiver-operating characteristic (ROC) curves.

RESULTS

The 9th JCCRC system was more clinically effective according to AIC (OS, 1199 vs. 1206; RFS, 2047 vs. 2057), showed better discriminatory ability according to the C-index (OS, 0.65 vs. 0.62; RFS, 0.62 vs. 0.58), and its time-dependent ROC curve was superior compared with the 8th TNM system.

CONCLUSION

These results suggest that the 9th JCCRC system has superior discriminative ability to the 8th TNM system, because the 9th JCCRC accounts for the presence of TD and N3 disease, which were both significant predictors of poor prognosis. Reconsidering the clinical value of these two factors in the TNM system could improve its clinical significance.

Concordance Measures and Time-Dependent ROC Methods

The receiver operating characteristic (ROC) curve displays sensitivity versus 1-specificity over a set of thresholds. The area under the ROC curve (AUC) is a global scalar summary of this curve. In the context of time-dependent ROC methods, we are interested in global scalar measures that summarize sequences of time-dependent AUCs over time. The concordance probability is a candidate for such purposes. The concordance probability can provide a global assessment of the discrimination ability of a test for an event that occurs at random times and may be right censored. If the test adequately differentiates between subjects who survive longer times and those who survive shorter times, this will assist clinical decisions. In this context the concordance probability may support assessment of precision medicine tools based on prognostic biomarkers models for overall survival. Definitions of time-dependent sensitivity and specificity are reviewed. Some connections between such definitions and concordance measures are also reviewed and we establish new connections via new measures of global concordance. We explore the relationship between such measures and their corresponding time-dependent AUC. To illustrate these concepts, an application in the context of Alzheimer's disease is presented.

Time-varying discrimination accuracy of longitudinal biomarkers for the prediction of mortality compared to assessment at fixed time point in severe burns patients

BACKGROUND

The progression of biomarkers over time is considered an indicator of disease progression and helps in the early detection of disease, thereby reducing disease-related mortality. Their ability to predict outcomes has been evaluated using conventional cross-sectional methods. This study investigated the prognostic performance of biomarkers over time.

METHODS

Patients aged > 18 years admitted to the burn intensive care unit within 24 h of a burn incident were enrolled. Information regarding longitudinal biomarkers, including white blood cells; platelet count; lactate, creatinine, and total bilirubin levels; and prothrombin time (PT), were retrieved from a clinical database. Time-dependent receiver operating characteristic curves using cumulative/dynamic and incident/dynamic (ID) approaches were used to evaluate prognostic performance.

RESULTS

Overall, 2259 patients were included and divided into survival and non-survival groups. By determining the area under the curve using the ID approach, platelets showed the highest c-index [0.930 (0.919-0.941)] across all time points. Conversely, the c-index of PT and creatinine levels were 0.862 (0.843-0.881) and 0.828 (0.809-0.848), respectively.

CONCLUSIONS

Platelet count was the best prognostic marker, followed by PT. Total bilirubin and creatinine levels also showed good prognostic ability. Although lactate was a strong predictor, it showed relatively poor prognostic performance in burns patients.

THBS2/CA19-9 Detecting Pancreatic Ductal Adenocarcinoma at Diagnosis Underperforms in Prediagnostic Detection: Implications for Biomarker Advancement

Pancreatic ductal adenocarcinoma (PDAC) is often diagnosed too late for effective therapy. The classic strategy for early detection biomarker advancement consists of initial retrospective phases of discovery and validation with tissue samples taken from individuals diagnosed with disease, compared with controls. Using this approach, we previously reported the discovery of a blood biomarker panel consisting of thrombospondin-2 (THBS2) and CA19-9 that together could discriminate resectable stage I and IIa PDAC as well as stages III and IV PDAC, with c-statistic values in the range of 0.96 to 0.97 in two phase II studies. We now report that in two studies of blood samples prospectively collected from 1 to 15 years prior to a PDAC diagnosis (Mayo Clinic and PLCO cohorts), THBS2 and/or CA19-9 failed to discriminate cases from healthy controls at the AUC = 0.8 needed. We conclude that PDAC progression may be heterogeneous and for some individuals can be more rapid than generally appreciated. It is important that PDAC early-detection studies incorporate high-risk, prospective prediagnostic cohorts into discovery and validation studies.Prevention Relevance: A blood biomarker panel of THBS2 and CA19-9 detects early stages of pancreatic ductal adenocarcinoma at diagnosis, but not when tested across a population up to 1 year earlier. Our findings suggest serial sampling over time, using prospectively collected samples for biomarker discovery, and more frequent screening of high-risk individuals.

Baseline and Dynamic Risk Predictors of Appropriate Implantable Cardioverter Defibrillator Therapy

Background Current approaches fail to separate patients at high versus low risk for ventricular arrhythmias owing to overreliance on a snapshot left ventricular ejection fraction measure. We used statistical machine learning to identify important cardiac imaging and time-varying risk predictors. Methods and Results Three hundred eighty-two cardiomyopathy patients (left ventricular ejection fraction ≤35%) underwent cardiac magnetic resonance before primary prevention implantable cardioverter defibrillator insertion. The primary end point was appropriate implantable cardioverter defibrillator discharge or sudden death. Patient characteristics; serum biomarkers of inflammation, neurohormonal status, and injury; and cardiac magnetic resonance-measured left ventricle and left atrial indices and myocardial scar burden were assessed at baseline. Time-varying covariates comprised interval heart failure hospitalizations and left ventricular ejection fractions. A random forest statistical method for survival, longitudinal, and multivariable outcomes incorporating baseline and time-varying variables was compared with (1) Seattle Heart Failure model scores and (2) random forest survival and Cox regression models incorporating baseline characteristics with and without imaging variables. Age averaged 57±13 years with 28% women, 66% white, 51% ischemic, and follow-up time of 5.9±2.3 years. The primary end point (n=75) occurred at 3.3±2.4 years. Random forest statistical method for survival, longitudinal, and multivariable outcomes with baseline and time-varying predictors had the highest area under the receiver operating curve, median 0.88 (95% CI, 0.75-0.96). Top predictors comprised heart failure hospitalization, left ventricle scar, left ventricle and left atrial volumes, left atrial function, and interleukin-6 level; heart failure accounted for 67% of the variation explained by the prediction, imaging 27%, and interleukin-6 2%. Serial left ventricular ejection fraction was not a significant predictor. Conclusions Hospitalization for heart failure and baseline cardiac metrics substantially improve ventricular arrhythmic risk prediction.

A practical perspective on the concordance index for the evaluation and selection of prognostic time-to-event models

Developing a prognostic model for biomedical applications typically requires mapping an individual's set of covariates to a measure of the risk that he or she may experience the event to be predicted. Many scenarios, however, especially those involving adverse pathological outcomes, are better described by explicitly accounting for the timing of these events, as well as their probability. As a result, in these cases, traditional classification or ranking metrics may be inadequate to inform model evaluation or selection. To address this limitation, it is common practice to reframe the problem in the context of survival analysis, and resort, instead, to the concordance index (C-index), which summarises how well a predicted risk score describes an observed sequence of events. A practically meaningful interpretation of the C-index, however, may present several difficulties and pitfalls. Specifically, we identify two main issues: i) the C-index remains implicitly, and subtly, dependent on time, and ii) its relationship with the number of subjects whose risk was incorrectly predicted is not straightforward. Failure to consider these two aspects may introduce undesirable and unwanted biases in the evaluation process, and even result in the selection of a suboptimal model. Hence, here, we discuss ways to obtain a meaningful interpretation in spite of these difficulties. Aiming to assist experimenters regardless of their familiarity with the C-index, we start from an introductory-level presentation of its most popular estimator, highlighting the latter's temporal dependency, and suggesting how it might be correctly used to inform model selection. We also address the nonlinearity of the C-index with respect to the number of correct risk predictions, elaborating a simplified framework that may enable an easier interpretation and quantification of C-index improvements or deteriorations.

Comparison of pre- and post-bronchodilator lung function as predictors of mortality: The HUNT Study

BACKGROUND AND OBJECTIVE

Post-bronchodilator (BD) lung function is recommended for the diagnosis of chronic obstructive pulmonary disease (COPD). However, often only pre-BD lung function is used in clinical practice or epidemiological studies. We aimed to compare the discrimination ability of pre-BD and post-BD lung function to predict all-cause mortality.

METHODS

Participants aged ≥40 years with airflow limitation (n = 2538) and COPD (n = 1262) in the second survey of the Nord-Trøndelag Health Study (HUNT2, 1995-1997) were followed up until 31 December 2015. Survival analysis and time-dependent area under the receiver operating characteristic curves (AUC) were used to compare the discrimination ability of pre-BD and post-BD lung function (percent-predicted forced expiratory volume in the first second (FEV1 ) (ppFEV1 ), FEV1 z-score, FEV1 quotient (FEV1 Q), modified Global Initiative for Chronic Obstructive Lung Disease (GOLD) categories or GOLD grades).

RESULTS

Among 2538 participants, 1387 died. The AUC for pre-BD and post-BD ppFEV1 to predict mortality were 60.8 and 61.8 (P = 0.005), respectively, at 20 years' follow-up. The corresponding AUC for FEV1 z-score were 58.5 and 60.4 (P < 0.001), for FEV1 Q were 68.7 and 70.1 (P = 0.002) and for modified GOLD categories were 62.3 and 64.5 (P < 0.001). Among participants with COPD, the AUC for pre-BD and post-BD ppFEV1 were 57.0 and 58.8 (P < 0.001), respectively. The corresponding AUC for FEV1 z-score were 53.1 and 55.8 (P < 0.001), for FEV1 Q were 63.6 and 65.1 (P = 0.037) and for GOLD grades were 56.0 and 57.0 (P = 0.268).

CONCLUSION

Mortality was better predicted by post-BD than by pre-BD lung function; however, they differed only by a small margin. The discrimination ability using GOLD grades among COPD participants was similar.

GOLD Classifications, COPD Hospitalization, and All-Cause Mortality in Chronic Obstructive Pulmonary Disease: The HUNT Study

Purpose

The Global Initiative for Chronic Obstructive Lung Disease (GOLD) has published three classifications of COPD from 2007 to 2017. No studies have investigated the ability of these classifications to predict COPD-related hospitalizations. We aimed to compare the discrimination ability of the GOLD 2007, 2011, and 2017 classifications to predict COPD hospitalization and all-cause mortality.

Patients and Methods

We followed 1300 participants with COPD aged ≥40 years who participated in the HUNT Study (1995-1997) through to December 31, 2015. Survival analysis and time-dependent area under receiver operating characteristics curves (AUC) were used to compare the discrimination abilities of the GOLD classifications.

Results

Of the 1300 participants, 522 were hospitalized due to COPD and 896 died over 20.4 years of follow-up. In adjusted models, worsening GOLD 2007, GOLD 2011, or GOLD 2017 categories were associated with higher hazards for COPD hospitalization and all-cause mortality, except for the GOLD 2017 classification and all-cause mortality (ptrend=0.114). In crude models, the AUCs (95% CI) for the GOLD 2007, GOLD 2011, and GOLD 2017 for COPD hospitalization were 63.1 (58.7-66.9), 60.9 (56.1-64.4), and 56.1 (54.0-58.1), respectively, at 20-years' follow-up. Corresponding estimates for all-cause mortality were 57.0 (54.8-59.1), 54.1 (52.1-56.0), and 52.6 (51.0-54.3). The differences in AUCs between the GOLD classifications to predict COPD hospitalization and all-cause mortality were constant over the follow-up time.

Conclusion

The GOLD 2007 classification was better than the GOLD 2011 and 2017 classifications at predicting COPD hospitalization and all-cause mortality.

Clinical risk prediction with random forests for survival, longitudinal, and multivariate (RF-SLAM) data analysis

Background

Clinical research and medical practice can be advanced through the prediction of an individual’s health state, trajectory, and responses to treatments. However, the majority of current clinical risk prediction models are based on regression approaches or machine learning algorithms that are static, rather than dynamic. To benefit from the increasing emergence of large, heterogeneous data sets, such as electronic health records (EHRs), novel tools to support improved clinical decision making through methods for individual-level risk prediction that can handle multiple variables, their interactions, and time-varying values are necessary.

Methods

We introduce a novel dynamic approach to clinical risk prediction for survival, longitudinal, and multivariate (SLAM) outcomes, called random forest for SLAM data analysis (RF-SLAM). RF-SLAM is a continuous-time, random forest method for survival analysis that combines the strengths of existing statistical and machine learning methods to produce individualized Bayes estimates of piecewise-constant hazard rates. We also present a method-agnostic approach for time-varying evaluation of model performance.

Results

We derive and illustrate the method by predicting sudden cardiac arrest (SCA) in the Left Ventricular Structural (LV) Predictors of Sudden Cardiac Death (SCD) Registry. We demonstrate superior performance relative to standard random forest methods for survival data. We illustrate the importance of the number of preceding heart failure hospitalizations as a time-dependent predictor in SCA risk assessment.

Conclusions

RF-SLAM is a novel statistical and machine learning method that improves risk prediction by incorporating time-varying information and accommodating a large number of predictors, their interactions, and missing values. RF-SLAM is designed to easily extend to simultaneous predictions of multiple, possibly competing, events and/or repeated measurements of discrete or continuous variables over time.Trial registration: LV Structural Predictors of SCD Registry (clinicaltrials.gov, NCT01076660), retrospectively registered 25 February 2010