Dynamic clinical prediction models for discrete time‐to‐event data with competing risks—A case study on the OUTCOMEREA database

Critical appraisal and assessment of bias among studies evaluating risk prediction models for in-hospital and 30-day mortality after percutaneous coronary intervention: a systematic review

OBJECTIVE

We systematically assessed prediction models for the risk of in-hospital and 30-day mortality in post-percutaneous coronary intervention (PCI) patients.

DESIGN

Systematic review and narrative synthesis.

DATA SOURCES

Searched PubMed, Web of Science, Embase, Cochrane Library, CINAHL, CNKI, Wanfang Database, VIP Database and SinoMed for literature up to 31 August 2023.

ELIGIBILITY CRITERIA

The included literature consists of studies in Chinese or English involving PCI patients aged ≥18 years. These studies aim to develop risk prediction models and include designs such as cohort studies, case-control studies, cross-sectional studies or randomised controlled trials. Each prediction model must contain at least two predictors. Exclusion criteria encompass models that include outcomes other than death post-PCI, literature lacking essential details on study design, model construction and statistical analysis, models based on virtual datasets, and publications such as conference abstracts, grey literature, informal publications, duplicate publications, dissertations, reviews or case reports. We also exclude studies focusing on the localisation applicability of the model or comparative effectiveness.

DATA EXTRACTION AND SYNTHESIS

Two independent teams of researchers developed standardised data extraction forms based on CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies to extract and cross-verify data. They used Prediction model Risk Of Bias Assessment Tool (PROBAST) to assess the risk of bias and applicability of the model development or validation studies included in this review.

RESULTS

This review included 28 studies with 38 prediction models, showing area under the curve values ranging from 0.81 to 0.987. One study had an unclear risk of bias, while 27 studies had a high risk of bias, primarily in the area of statistical analysis. The models constructed in 25 studies lacked clinical applicability, with 21 of these studies including intraoperative or postoperative predictors.

CONCLUSION

The development of in-hospital and 30-day mortality prediction models for post-PCI patients is in its early stages. Emphasising clinical applicability and predictive stability is vital. Future research should follow PROBAST's low risk-of-bias guidelines, prioritising external validation for existing models to ensure reliable and widely applicable clinical predictions.

PROSPERO REGISTRATION NUMBER

CRD42023477272.

Early prediction of ventilator-associated pneumonia with machine learning models: A systematic review and meta-analysis of prediction model performance✰

BACKGROUND

Machine learning-based prediction models can catalog, classify, and correlate large amounts of multimodal data to aid clinicians at diagnostic, prognostic, and therapeutic levels. Early prediction of ventilator-associated pneumonia (VAP) may accelerate the diagnosis and guide preventive interventions. The performance of a variety of machine learning-based prediction models were analyzed among adults undergoing invasive mechanical ventilation.

METHODS

This systematic review and meta-analysis was conducted in accordance with the Cochrane Collaboration. Machine learning-based prediction models were identified from a search of nine multi-disciplinary databases. Two authors independently selected and extracted data using predefined criteria and data extraction forms. The predictive performance, the interpretability, the technological readiness level, and the risk of bias of the included studies were evaluated.

RESULTS

Final analysis included 10 static prediction models using supervised learning. The pooled area under the receiver operating characteristics curve, sensitivity, and specificity for VAP were 0.88 (95 % CI 0.82-0.94, I2 98.4 %), 0.72 (95 % CI 0.45-0.98, I2 97.4 %) and 0.90 (95 % CI 0.85-0.94, I2 97.9 %), respectively. All included studies had either a high or unclear risk of bias without significant improvements in applicability. The care-related risk factors for the best performing models were the duration of mechanical ventilation, the length of ICU stay, blood transfusion, nutrition strategy, and the presence of antibiotics.

CONCLUSION

A variety of the prediction models, prediction intervals, and prediction windows were identified to facilitate timely diagnosis. In addition, care-related risk factors susceptible for preventive interventions were identified. In future, there is a need for dynamic machine learning models using time-depended predictors in conjunction with feature importance of the models to predict real-time risk of VAP and related outcomes to optimize bundled care.

Validation of discrete time-to-event prediction models in the presence of competing risks

Clinical prediction models play a key role in risk stratification, therapy assignment and many other fields of medical decision making. Before they can enter clinical practice, their usefulness has to be demonstrated using systematic validation. Methods to assess their predictive performance have been proposed for continuous, binary, and time-to-event outcomes, but the literature on validation methods for discrete time-to-event models with competing risks is sparse. The present paper tries to fill this gap and proposes new methodology to quantify discrimination, calibration, and prediction error (PE) for discrete time-to-event outcomes in the presence of competing risks. In our case study, the goal was to predict the risk of ventilator-associated pneumonia (VAP) attributed to Pseudomonas aeruginosa in intensive care units (ICUs). Competing events are extubation, death, and VAP due to other bacteria. The aim of this application is to validate complex prediction models developed in previous work on more recently available validation data.

Dynamic prediction: A challenge for biostatisticians, but greatly needed by patients, physicians and the public

Prognosis is usually expressed in terms of the probability that a patient will or will not have experienced an event of interest t years after diagnosis of a disease. This quantity, however, is of little informative value for a patient who is still event-free after a number of years. Such a patient would be much more interested in the conditional probability of being event-free in the upcoming t years, given that he/she did not experience the event in the s years after diagnosis, called "conditional survival." It is the simplest form of a dynamic prediction and can be dealt with using straightforward extensions of standard time-to-event analyses in clinical cohort studies. For a healthy individual, a related problem with further complications is the so-called "age-conditional probability of developing cancer" in the next t years. Here, the competing risk of dying from other diseases has to be taken into account. For both situations, the hazard function provides the central dynamic concept, which can be further extended in a natural way to build dynamic prediction models that incorporate both baseline and time-dependent characteristics. Such models are able to exploit the most current information accumulating over time in order to accurately predict the further course or development of a disease. In this article, the biostatistical challenges as well as the relevance and importance of dynamic prediction are illustrated using studies of multiple myeloma, a hematologic malignancy with a formerly rather poor prognosis which has improved over the last few years.