Prediction errors for state occupation and transition probabilities in multi-state models

An augmented illness-death model for semi-competing risks with clinically immediate terminal events

Preeclampsia is a pregnancy-associated condition posing risks of both fetal and maternal mortality and morbidity that can only resolve following delivery and removal of the placenta. Because in its typical form preeclampsia can arise before delivery, but not after, these two events exemplify the time-to-event setting of "semi-competing risks" in which a non-terminal event of interest is subject to the occurrence of a terminal event of interest. The semi-competing risks framework presents a valuable opportunity to simultaneously address two clinically meaningful risk modeling tasks: (i) characterizing risk of developing preeclampsia, and (ii) characterizing time to delivery after onset of preeclampsia. However, some people with preeclampsia deliver immediately upon diagnosis, while others are admitted and monitored for an extended period before giving birth, resulting in two distinct trajectories following the non-terminal event, which we call "clinically immediate" and "non-immediate" terminal events. Though such phenomena arise in many clinical contexts, to-date there have not been methods developed to acknowledge the complex dependencies between such outcomes, nor leverage these phenomena to gain new insight into individualized risk. We address this gap by proposing a novel augmented frailty-based illness-death model with a binary submodel to distinguish risk of immediate terminal event following the non-terminal event. The model admits direct dependence of the terminal event on the non-terminal event through flexible regression specification, as well as indirect dependence via a shared frailty term linking each submodel. We develop an efficient Bayesian sampler for estimation and corresponding model fit metrics, and derive formulae for dynamic risk prediction. In an extended example using pregnancy outcome data from an electronic health record, we demonstrate the proposed model's direct applicability to address a broad range of clinical questions.

Calibration plots for multistate risk predictions models

INTRODUCTION

There is currently no guidance on how to assess the calibration of multistate models used for risk prediction. We introduce several techniques that can be used to produce calibration plots for the transition probabilities of a multistate model, before assessing their performance in the presence of random and independent censoring through a simulation.

METHODS

We studied pseudo-values based on the Aalen-Johansen estimator, binary logistic regression with inverse probability of censoring weights (BLR-IPCW), and multinomial logistic regression with inverse probability of censoring weights (MLR-IPCW). The MLR-IPCW approach results in a calibration scatter plot, providing extra insight about the calibration. We simulated data with varying levels of censoring and evaluated the ability of each method to estimate the calibration curve for a set of predicted transition probabilities. We also developed evaluated the calibration of a model predicting the incidence of cardiovascular disease, type 2 diabetes and chronic kidney disease among a cohort of patients derived from linked primary and secondary healthcare records.

RESULTS

The pseudo-value, BLR-IPCW, and MLR-IPCW approaches give unbiased estimates of the calibration curves under random censoring. These methods remained predominately unbiased in the presence of independent censoring, even if the censoring mechanism was strongly associated with the outcome, with bias concentrated in low-density regions of predicted transition probability.

CONCLUSIONS

We recommend implementing either the pseudo-value or BLR-IPCW approaches to produce a calibration curve, combined with the MLR-IPCW approach to produce a calibration scatter plot. The methods have been incorporated into the "calibmsm" R package available on CRAN.

Two-step interpretable modeling of ICU-AIs

We present a novel methodology for integrating high resolution longitudinal data with the dynamic prediction capabilities of survival models. The aim is two-fold: to improve the predictive power while maintaining the interpretability of the models. To go beyond the black box paradigm of artificial neural networks, we propose a parsimonious and robust semi-parametric approach (i.e., a landmarking competing risks model) that combines routinely collected low-resolution data with predictive features extracted from a convolutional neural network, that was trained on high resolution time-dependent information. We then use saliency maps to analyze and explain the extra predictive power of this model. To illustrate our methodology, we focus on healthcare-associated infections in patients admitted to an intensive care unit.

Modeling multiple correlated end-organ disease trajectories: A tutorial for multistate and joint models with applications in diabetes complications

State-of-the-art biostatistics methods allow for the simultaneous modeling of several correlated non-fatal disease processes over time, but there is no clear guidance on the optimal analysis in most settings. An example occurs in diabetes, where it is not known with certainty how microvascular complications of the eyes, kidneys, and nerves co-develop over time. In this article, we propose and contrast two general model frameworks for studying complications (sequential state and parallel trajectory frameworks) and review multivariate methods for their analysis, focusing on multistate and joint modeling. We illustrate these methods in a tutorial format using the long-term follow-up from the Diabetes Control and Complications Trial and Epidemiology of Diabetes Interventions and Complications study public data repository. A formal comparison of prediction error and discrimination is included. Multistate models are particularly advantageous for determining the order and timing of complications, but require discretization of the longitudinal outcomes and possibly a very complex state space process. Intermittent observation of the states must be accounted for, and discretization is a probable disadvantage in this setting. In contrast, joint models can account for variations of continuous biomarkers over time and are particularly designed for modeling complex association structures between the complications and for performing dynamic predictions of an outcome of interest to inform clinical decisions (eg, a late-stage complication). We found that both models have helpful features that can better-inform our understanding of the complex trajectories that complications may take and can therefore help with decision making for patients presenting with diabetes complications.

Multistate modeling and structure selection for multitype recurrent events and terminal event data

In cardiovascular disease studies, a large number of risk factors are measured but it often remains unknown whether all of them are relevant variables and whether the impact of these variables is changing with time or remains constant. In addition, more than one kind of cardiovascular disease events can be observed in the same patient and events of different types are possibly correlated. It is expected that different kinds of events are associated with different covariates and the forms of covariate effects also vary between event types. To tackle these problems, we proposed a multistate modeling framework for the joint analysis of multitype recurrent events and terminal event. Model structure selection is performed to identify covariates with time-varying coefficients, time-independent coefficients, and null effects. This helps in understanding the disease process as it can detect relevant covariates and identify the temporal dynamics of the covariate effects. It also provides a more parsimonious model to achieve better risk prediction. The performance of the proposed model and selection method is evaluated in numerical studies and illustrated on a real dataset from the Atherosclerosis Risk in Communities study.

Predicting COVID-19 progression in hospitalized patients in Belgium from a multi-state model

Objectives

To adopt a multi-state risk prediction model for critical disease/mortality outcomes among hospitalised COVID-19 patients using nationwide COVID-19 hospital surveillance data in Belgium.

Materials and methods

Information on 44,659 COVID-19 patients hospitalised between March 2020 and June 2021 with complete data on disease outcomes and candidate predictors was used to adopt a multi-state, multivariate Cox model to predict patients' probability of recovery, critical [transfer to intensive care units (ICU)] or fatal outcomes during hospital stay.

Results

Median length of hospital stay was 9 days (interquartile range: 5-14). After admission, approximately 82% of the COVID-19 patients were discharged alive, 15% of patients were admitted to ICU, and 15% died in the hospital. The main predictors of an increased probability for recovery were younger age, and to a lesser extent, a lower number of prevalent comorbidities. A patient's transition to ICU or in-hospital death had in common the following predictors: high levels of c-reactive protein (CRP) and lactate dehydrogenase (LDH), reporting lower respiratory complaints and male sex. Additionally predictors for a transfer to ICU included middle-age, obesity and reporting loss of appetite and staying at a university hospital, while advanced age and a higher number of prevalent comorbidities for in-hospital death. After ICU, younger age and low levels of CRP and LDH were the main predictors for recovery, while in-hospital death was predicted by advanced age and concurrent comorbidities.

Conclusion

As one of the very few, a multi-state model was adopted to identify key factors predicting COVID-19 progression to critical disease, and recovery or death.

The polytomous discrimination index for prediction involving multistate processes under intermittent observation

With the increasing importance of predictive modeling in health research comes the need for methods to rigorously assess predictive accuracy. We consider the problem of evaluating the accuracy of predictive models for nominal outcomes when outcome data are coarsened at random. We first consider the problem in the context of a multinomial response modeled by polytomous logistic regression. Attention is then directed to the motivating setting in which class membership corresponds to the state occupied in a multistate disease process at a time horizon of interest. Here, class (state) membership may be unknown at the time horizon since disease processes are under intermittent observation. We propose a novel extension to the polytomous discrimination index to address this and evaluate the predictive accuracy of an intensity-based model in the context of a study involving patients with arthritis from a registry at the University of Toronto Centre for Prognosis Studies in Rheumatic Diseases.

Classical Regression and Predictive Modeling

BACKGROUND

With the advent of personalized and stratified medicine, there has been much discussion about predictive modeling and the role of classical regression in modern medical research. We describe and distinguish the goals in these 2 frameworks for analysis.

METHODS

The assumptions underlying and utility of classical regression are reviewed for continuous and binary outcomes. The tenets of predictive modeling are then discussed and contrasted. Principles are illustrated by simulation and through application of methods to a neurosurgical study.

RESULTS

Classical regression can be used for insights into causal mechanisms if careful thought is given to the role of variables of interest and potential confounders. In predictive modeling, interest lies more in accuracy of predictions and so alternative metrics are used to judge adequacy of models and methods; methods which average predictions over several contending models can improve predictive performance but these do not admit a single risk score.

CONCLUSIONS

Both classical regression and predictive modeling have important roles in modern medical research. Understanding the distinction between the 2 frameworks for analysis is important to place them in their appropriate context and interpreting findings from published studies appropriately.

Impact of NPM1/FLT3-ITD genotypes defined by the 2017 European LeukemiaNet in patients with acute myeloid leukemia

Patients with acute myeloid leukemia (AML) harboring FLT3 internal tandem duplications (ITDs) have poor outcomes, in particular AML with a high (≥0.5) mutant/wild-type allelic ratio (AR). The 2017 European LeukemiaNet (ELN) recommendations defined 4 distinct FLT3-ITD genotypes based on the ITD AR and the NPM1 mutational status. In this retrospective exploratory study, we investigated the prognostic and predictive impact of the NPM1/FLT3-ITD genotypes categorized according to the 2017 ELN risk groups in patients randomized within the RATIFY trial, which evaluated the addition of midostaurin to standard chemotherapy. The 4 NPM1/FLT3-ITD genotypes differed significantly with regard to clinical and concurrent genetic features. Complete ELN risk categorization could be done in 318 of 549 trial patients with FLT3-ITD AML. Significant factors for response after 1 or 2 induction cycles were ELN risk group and white blood cell (WBC) counts; treatment with midostaurin had no influence. Overall survival (OS) differed significantly among ELN risk groups, with estimated 5-year OS probabilities of 0.63, 0.43, and 0.33 for favorable-, intermediate-, and adverse-risk groups, respectively (P < .001). A multivariate Cox model for OS using allogeneic hematopoietic cell transplantation (HCT) in first complete remission as a time-dependent variable revealed treatment with midostaurin, allogeneic HCT, ELN favorable-risk group, and lower WBC counts as significant favorable factors. In this model, there was a consistent beneficial effect of midostaurin across ELN risk groups.

Predicting the clinical trajectory in critically ill patients with sepsis: a cohort study

Background

To develop a mathematical model to estimate daily evolution of disease severity using routinely available parameters in patients admitted to the intensive care unit (ICU).

Methods

Over a 3-year period, we prospectively enrolled consecutive adults with sepsis and categorized patients as (1) being at risk for developing (more severe) organ dysfunction, (2) having (potentially still reversible) limited organ failure, or (3) having multiple-organ failure. Daily probabilities for transitions between these disease states, and to death or discharge, during the first 2 weeks in ICU were calculated using a multi-state model that was updated every 2 days using both baseline and time-varying information. The model was validated in independent patients.

Results

We studied 1371 sepsis admissions in 1251 patients. Upon presentation, 53 (4%) were classed at risk, 1151 (84%) had limited organ failure, and 167 (12%) had multiple-organ failure. Among patients with limited organ failure, 197 (17%) evolved to multiple-organ failure or died and 809 (70%) improved or were discharged alive within 14 days. Among patients with multiple-organ failure, 67 (40%) died and 91 (54%) improved or were discharged. Treatment response could be predicted with reasonable accuracy (c-statistic ranging from 0.55 to 0.81 for individual disease states, and 0.67 overall). Model performance in the validation cohort was similar.

Conclusions

This prediction model that estimates daily evolution of disease severity during sepsis may eventually support clinicians in making better informed treatment decisions and could be used to evaluate prognostic biomarkers or perform in silico modeling of novel sepsis therapies during trial design.

Clinical trial registration

ClinicalTrials.gov NCT01905033