Validation of probabilistic predictions

Prediction of Long-Term Physical, Mental, and Cognitive Problems Following Critical Illness: Development and External Validation of the PROSPECT Prediction Model

OBJECTIVES

ICU survivors often suffer from long-lasting physical, mental, and cognitive health problems after hospital discharge. As several interventions that treat or prevent these problems already start during ICU stay, patients at high risk should be identified early. This study aimed to develop a model for early prediction of post-ICU health problems within 48 hours after ICU admission.

DESIGN

Prospective cohort study in seven Dutch ICUs.

SETTING/PATIENTS

ICU patients older than 16 years and admitted for greater than or equal to 12 hours between July 2016 and March 2020.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

Outcomes were physical problems (fatigue or ≥ 3 new physical symptoms), mental problems (anxiety, depression, or post-traumatic stress disorder), and cognitive impairment. Patient record data and questionnaire data were collected at ICU admission, and after 3 and 12 months, of 2,476 patients. Several models predicting physical, mental, or cognitive problems and a composite score at 3 and 12 months were developed using variables collected within 48 hours after ICU admission. Based on performance and clinical feasibility, a model, PROSPECT, predicting post-ICU health problems at 3 months was chosen, including the predictors of chronic obstructive pulmonary disease, admission type, expected length of ICU stay greater than or equal to 2 days, and preadmission anxiety and fatigue. Internal validation using bootstrapping on data of the largest hospital ( n = 1,244) yielded a C -statistic of 0.73 (95% CI, 0.70-0.76). External validation was performed on data ( n = 864) from the other six hospitals with a C -statistic of 0.77 (95% CI, 0.73-0.80).

CONCLUSIONS

The developed and externally validated PROSPECT model can be used within 48 hours after ICU admission for identifying patients with an increased risk of post-ICU problems 3 months after ICU admission. Timely preventive interventions starting during ICU admission and follow-up care can prevent or mitigate post-ICU problems in these high-risk patients.

Comparison of likelihood penalization and variance decomposition approaches for clinical prediction models: A simulation study

Logistic regression is one of the most commonly used approaches to develop clinical risk prediction models. Developers of such models often rely on approaches that aim to minimize the risk of overfitting and improve predictive performance of the logistic model, such as through likelihood penalization and variance decomposition techniques. We present an extensive simulation study that compares the out-of-sample predictive performance of risk prediction models derived using the elastic net, with Lasso and ridge as special cases, and variance decomposition techniques, namely, incomplete principal component regression and incomplete partial least squares regression. We varied the expected events per variable, event fraction, number of candidate predictors, presence of noise predictors, and the presence of sparse predictors in a full-factorial design. Predictive performance was compared on measures of discrimination, calibration, and prediction error. Simulation metamodels were derived to explain the performance differences within model derivation approaches. Our results indicate that, on average, prediction models developed using penalization and variance decomposition approaches outperform models developed using ordinary maximum likelihood estimation, with penalization approaches being consistently superior over the variance decomposition approaches. Differences in performance were most pronounced on the calibration of the model. Performance differences regarding prediction error and concordance statistic outcomes were often small between approaches. The use of likelihood penalization and variance decomposition techniques methods was illustrated in the context of peripheral arterial disease.

External validation of models for predicting cumulative live birth over multiple complete cycles of IVF treatment

STUDY QUESTION

Can two prediction models developed using data from 1999 to 2009 accurately predict the cumulative probability of live birth per woman over multiple complete cycles of IVF in an updated UK cohort?

SUMMARY ANSWER

After being updated, the models were able to estimate individualized chances of cumulative live birth over multiple complete cycles of IVF with greater accuracy.

WHAT IS KNOWN ALREADY

The McLernon models were the first to predict cumulative live birth over multiple complete cycles of IVF. They were converted into an online calculator called OPIS (Outcome Prediction In Subfertility) which has 3000 users per month on average. A previous study externally validated the McLernon models using a Dutch prospective cohort containing data from 2011 to 2014. With changes in IVF practice over time, it is important that the McLernon models are externally validated on a more recent cohort of patients to ensure that predictions remain accurate.

STUDY DESIGN, SIZE, DURATION

A population-based cohort of 91 035 women undergoing IVF in the UK between January 2010 and December 2016 was used for external validation. Data on frozen embryo transfers associated with these complete IVF cycles conducted from 1 January 2017 to 31 December 2017 were also collected.

PARTICIPANTS/MATERIALS, SETTING, METHODS

Data on IVF treatments were obtained from the Human Fertilisation and Embryology Authority (HFEA). The predictive performances of the McLernon models were evaluated in terms of discrimination and calibration. Discrimination was assessed using the c-statistic and calibration was assessed using calibration-in-the-large, calibration slope, and calibration plots. Where any model demonstrated poor calibration in the validation cohort, the models were updated using intercept recalibration, logistic recalibration, or model revision to improve model performance.

MAIN RESULTS AND THE ROLE OF CHANCE

Following exclusions, 91 035 women who underwent 144 734 complete cycles were included. The validation cohort had a similar distribution age profile to women in the development cohort. Live birth rates over all complete cycles of IVF per woman were higher in the validation cohort. After calibration assessment, both models required updating. The coefficients of the pre-treatment model were revised, and the updated model showed reasonable discrimination (c-statistic: 0.67, 95% CI: 0.66 to 0.68). After logistic recalibration, the post-treatment model showed good discrimination (c-statistic: 0.75, 95% CI: 0.74 to 0.76). As an example, in the updated pre-treatment model, a 32-year-old woman with 2 years of primary infertility has a 42% chance of having a live birth in the first complete ICSI cycle and a 77% chance over three complete cycles. In a couple with 2 years of primary male factor infertility where a 30-year-old woman has 15 oocytes collected in the first cycle, a single fresh blastocyst embryo transferred in the first cycle and spare embryos cryopreserved, the estimated chance of live birth provided by the post-treatment model is 46% in the first complete ICSI cycle and 81% over three complete cycles.

LIMITATIONS, REASONS FOR CAUTION

Two predictors from the original models, duration of infertility and previous pregnancy, which were not available in the recent HFEA dataset, were imputed using data from the older cohort used to develop the models. The HFEA dataset does not contain some other potentially important predictors, e.g. BMI, ethnicity, race, smoking and alcohol intake in women, as well as measures of ovarian reserve such as antral follicle count.

WIDER IMPLICATIONS OF THE FINDINGS

Both updated models show improved predictive ability and provide estimates which are more reflective of current practice and patient case mix. The updated OPIS tool can be used by clinicians to help shape couples' expectations by informing them of their individualized chances of live birth over a sequence of multiple complete cycles of IVF.

STUDY FUNDING/COMPETING INTEREST(S)

This study was supported by an Elphinstone scholarship scheme at the University of Aberdeen and Aberdeen Fertility Centre, University of Aberdeen. S.B. has a commitment of research funding from Merck. D.J.M. and M.B.R. declare support for the present manuscript from Elphinstone scholarship scheme at the University of Aberdeen and Assisted Reproduction Unit at Aberdeen Fertility Centre, University of Aberdeen. D.J.M. declares grants received by University of Aberdeen from NHS Grampian, The Meikle Foundation, and Chief Scientist Office in the past 3 years. D.J.M. declares receiving an honorarium for lectures from Merck. D.J.M. is Associate Editor of Human Reproduction Open and Statistical Advisor for Reproductive BioMed Online. S.B. declares royalties from Cambridge University Press for a book. S.B. declares receiving an honorarium for lectures from Merck, Organon, Ferring, Obstetric and Gynaecological Society of Singapore, and Taiwanese Society for Reproductive Medicine. S.B. has received support from Merck, ESHRE, and Ferring for attending meetings as speaker and is on the METAFOR and CAPRE Trials Data Monitoring Committee.

TRIAL REGISTRATION NUMBER

N/A.

Validation of prediction models for radiation-induced late rectal bleeding: evidence from a large pooled population of prostate cancer patients

PURPOSE

To validate published models for the risk estimate of grade≥1 (G1+), grade≥2 (G2+) and grade=3 (G3) late rectal bleeding (LRB) after radical radiotherapy for prostate cancer in a large pooled population from three prospective trials.

MATERIALS AND METHODS

The external validation population included patients from Europe, and Oceanian centres enrolled between 2003 and 2014. Patients received 3DCRT or IMRT at doses between 66-80 Gy. IMRT was administered with conventional or hypofractionated schemes (2.35-2.65 Gy/fr). LRB was prospectively scored using patient-reported questionnaires (LENT/SOMA scale) with a 3-year follow-up. All Normal Tissue Complication Probability (NTCP) models published until 2021 based on the Equivalent Uniform Dose (EUD) from the rectal Dose Volume Histogram (DVH) were considered for validation. Model performance in validation was evaluated through calibration and discrimination.

RESULTS

Sixteen NTCP models were tested on data from 1633 patients. G1+ LRB was scored in 465 patients (28.5%), G2+ in 255 patients (15.6%) and G3 in 112 patients (6.8%). The best performances for G2+ and G3 LRB highlighted the importance of the medium-high doses to the rectum (volume parameters n=0.24 and n=0.18, respectively). Good performance was seen for models of severe LRB. Moreover, a multivariate model with two clinical factors found the best calibration slope.

CONCLUSION

Five published NTCP models developed on non-contemporary cohorts were able to predict a relative increase in the toxicity response in a more recent validation population. Compared to QUANTEC findings, dosimetric results pointed toward mid-high doses of rectal DVH. The external validation cohort confirmed abdominal surgery and cardiovascular diseases as risk factors.

Dementia risk predictions from German claims data using methods of machine learning

INTRODUCTION

We examined whether German claims data are suitable for dementia risk prediction, how machine learning (ML) compares to classical regression, and what the important predictors for dementia risk are.

METHODS

We analyzed data from the largest German health insurance company, including 117,895 dementia-free people age 65+. Follow-up was 10 years. Predictors were: 23 age-related diseases, 212 medical prescriptions, 87 surgery codes, as well as age and sex. Statistical methods included logistic regression (LR), gradient boosting (GBM), and random forests (RFs).

RESULTS

Discriminatory power was moderate for LR (C-statistic = 0.714; 95% confidence interval [CI] = 0.708-0.720) and GBM (C-statistic = 0.707; 95% CI  = 0.700-0.713) and lower for RF (C-statistic = 0.636; 95% CI  = 0.628-0.643). GBM had the best model calibration. We identified antipsychotic medications and cerebrovascular disease but also a less-established specific antibacterial medical prescription as important predictors.

DISCUSSION

Our models from German claims data have acceptable accuracy and may provide cost-effective decision support for early dementia screening.

Development and Validation of a Predictive Tool for Postpartum Hemorrhage after Vaginal Delivery: A Prospective Cohort Study

Postpartum hemorrhage (PPH) is one of the leading causes of maternal morbidity worldwide. This study aimed to develop and validate a predictive model for PPH after vaginal deliveries, based on routinely available clinical and biological data. The derivation monocentric cohort included pregnant women with vaginal delivery at Brest University Hospital (France) between April 2013 and May 2015. Immediate PPH was defined as a blood loss of ≥500 mL in the first 24 h after delivery and measured with a graduated collector bag. A logistic model, using a combination of multiple imputation and variable selection with bootstrap, was used to construct a predictive model and a score for PPH. An external validation was performed on a prospective cohort of women who delivered between 2015 and 2019 at Brest University Hospital. Among 2742 deliveries, PPH occurred in 141 (5.1%) women. Eight factors were independently associated with PPH: pre-eclampsia (aOR 6.25, 95% CI 2.35−16.65), antepartum bleeding (aOR 2.36, 95% CI 1.43−3.91), multiple pregnancy (aOR 3.24, 95% CI 1.52−6.92), labor duration ≥ 8 h (aOR 1.81, 95% CI 1.20−2.73), macrosomia (aOR 2.33, 95% CI 1.36−4.00), episiotomy (aOR 2.02, 95% CI 1.40−2.93), platelet count < 150 Giga/L (aOR 2.59, 95% CI 1.47−4.55) and aPTT ratio ≥ 1.1 (aOR 2.01, 95% CI 1.25−3.23). The derived predictive score, ranging from 0 to 10 (woman at risk if score ≥ 1), demonstrated a good discriminant power (AUROC 0.69; 95% CI 0.65−0.74) and calibration. The external validation cohort was composed of 3061 vaginal deliveries. The predictive score on this independent cohort showed an acceptable ability to discriminate (AUROC 0.66; 95% CI 0.62−0.70). We derived and validated a robust predictive model identifying women at risk for PPH using in-depth statistical methodology. This score has the potential to improve the care of pregnant women and to take preventive actions on them.

Tailored Bayes: a risk modeling framework under unequal misclassification costs

Risk prediction models are a crucial tool in healthcare. Risk prediction models with a binary outcome (i.e., binary classification models) are often constructed using methodology which assumes the costs of different classification errors are equal. In many healthcare applications, this assumption is not valid, and the differences between misclassification costs can be quite large. For instance, in a diagnostic setting, the cost of misdiagnosing a person with a life-threatening disease as healthy may be larger than the cost of misdiagnosing a healthy person as a patient. In this article, we present Tailored Bayes (TB), a novel Bayesian inference framework which "tailors" model fitting to optimize predictive performance with respect to unbalanced misclassification costs. We use simulation studies to showcase when TB is expected to outperform standard Bayesian methods in the context of logistic regression. We then apply TB to three real-world applications, a cardiac surgery, a breast cancer prognostication task, and a breast cancer tumor classification task and demonstrate the improvement in predictive performance over standard methods.

The leap to ordinal: Detailed functional prognosis after traumatic brain injury with a flexible modelling approach

When a patient is admitted to the intensive care unit (ICU) after a traumatic brain injury (TBI), an early prognosis is essential for baseline risk adjustment and shared decision making. TBI outcomes are commonly categorised by the Glasgow Outcome Scale–Extended (GOSE) into eight, ordered levels of functional recovery at 6 months after injury. Existing ICU prognostic models predict binary outcomes at a certain threshold of GOSE (e.g., prediction of survival [GOSE > 1]). We aimed to develop ordinal prediction models that concurrently predict probabilities of each GOSE score. From a prospective cohort (n = 1,550, 65 centres) in the ICU stratum of the Collaborative European NeuroTrauma Effectiveness Research in TBI (CENTER-TBI) patient dataset, we extracted all clinical information within 24 hours of ICU admission (1,151 predictors) and 6-month GOSE scores. We analysed the effect of two design elements on ordinal model performance: (1) the baseline predictor set, ranging from a concise set of ten validated predictors to a token-embedded representation of all possible predictors, and (2) the modelling strategy, from ordinal logistic regression to multinomial deep learning. With repeated k-fold cross-validation, we found that expanding the baseline predictor set significantly improved ordinal prediction performance while increasing analytical complexity did not. Half of these gains could be achieved with the addition of eight high-impact predictors to the concise set. At best, ordinal models achieved 0.76 (95% CI: 0.74–0.77) ordinal discrimination ability (ordinal c-index) and 57% (95% CI: 54%– 60%) explanation of ordinal variation in 6-month GOSE (Somers’ D_xy). Model performance and the effect of expanding the predictor set decreased at higher GOSE thresholds, indicating the difficulty of predicting better functional outcomes shortly after ICU admission. Our results motivate the search for informative predictors that improve confidence in prognosis of higher GOSE and the development of ordinal dynamic prediction models.

Innovations in suicide prevention research (INSPIRE): a protocol for a population-based case-control study

BACKGROUND

Suicide deaths have been increasing for the past 20 years in the USA resulting in 45 979 deaths in 2020, a 29% increase since 1999. Lack of data linkage between entities with potential to implement large suicide prevention initiatives (health insurers, health institutions and corrections) is a barrier to developing an integrated framework for suicide prevention.

OBJECTIVES

Data linkage between death records and several large administrative datasets to (1) estimate associations between risk factors and suicide outcomes, (2) develop predictive algorithms and (3) establish long-term data linkage workflow to ensure ongoing suicide surveillance.

METHODS

We will combine six data sources from North Carolina, the 10th most populous state in the USA, from 2006 onward, including death certificate records, violent deaths reporting system, large private health insurance claims data, Medicaid claims data, University of North Carolina electronic health records and data on justice involved individuals released from incarceration. We will determine the incidence of death from suicide, suicide attempts and ideation in the four subpopulations to establish benchmarks. We will use a nested case-control design with incidence density-matched population-based controls to (1) identify short-term and long-term risk factors associated with suicide attempts and mortality and (2) develop machine learning-based predictive algorithms to identify individuals at risk of suicide deaths.

DISCUSSION

We will address gaps from prior studies by establishing an in-depth linked suicide surveillance system integrating multiple large, comprehensive databases that permit establishment of benchmarks, identification of predictors, evaluation of prevention efforts and establishment of long-term surveillance workflow protocols.

A Comparison of Methods to Detect Changes in Prediction Models

BACKGROUND

Prediction models inform decisions in many areas of medicine. Most models are fitted once and then applied to new (future) patients, despite the fact that model coefficients can vary over time due to changes in patients' clinical characteristics and disease risk. However, the optimal method to detect changes in model parameters has not been rigorously assessed.

METHODS

We simulated data, informed by post-lung transplant mortality data and tested the following two approaches for detecting model change: (1) the "Direct Approach," it compares coefficients of the model refit on recent data to those at baseline; and (2) "Calibration Regression," it fits a logistic regression model of the log-odds of the observed outcomes versus the linear predictor from the baseline model (i.e., the log-odds of the predicted probabilities obtained from the baseline model) and tests whether the intercept and slope differ from 0 and 1, respectively. Four scenarios were simulated using logistic regression for binary outcomes as follows: (1) we fixed all model parameters, (2) we varied the outcome prevalence between 0.1 and 0.2, (3) we varied the coefficient of one of the ten predictors between 0.2 and 0.4, and (4) we varied the outcome prevalence and coefficient of one predictor simultaneously.

RESULTS

Calibration regression tended to detect changes sooner than the Direct Approach, with better performance (e.g., larger proportion of true claims). When the sample size was large, both methods performed well. When two parameters changed simultaneously, neither method performed well.

CONCLUSION

Neither change detection method examined here proved optimal under all circumstances. However, our results suggest that if one is interested in detecting a change in overall incidence of an outcome (e.g., intercept), the Calibration Regression method may be superior to the Direct Approach. Conversely, if one is interested in detecting a change in other model covariates (e.g., slope), the Direct Approach may be superior.

Comparison of dynamic updating strategies for clinical prediction models

BACKGROUND

Prediction models inform many medical decisions, but their performance often deteriorates over time. Several discrete-time update strategies have been proposed in the literature, including model recalibration and revision. However, these strategies have not been compared in the dynamic updating setting.

METHODS

We used post-lung transplant survival data during 2010-2015 and compared the Brier Score (BS), discrimination, and calibration of the following update strategies: (1) never update, (2) update using the closed testing procedure proposed in the literature, (3) always recalibrate the intercept, (4) always recalibrate the intercept and slope, and (5) always refit/revise the model. In each case, we explored update intervals of every 1, 2, 4, and 8 quarters. We also examined how the performance of the update strategies changed as the amount of old data included in the update (i.e., sliding window length) increased.

RESULTS

All methods of updating the model led to meaningful improvement in BS relative to never updating. More frequent updating yielded better BS, discrimination, and calibration, regardless of update strategy. Recalibration strategies led to more consistent improvements and less variability over time compared to the other updating strategies. Using longer sliding windows did not substantially impact the recalibration strategies, but did improve the discrimination and calibration of the closed testing procedure and model revision strategies.

CONCLUSIONS

Model updating leads to improved BS, with more frequent updating performing better than less frequent updating. Model recalibration strategies appeared to be the least sensitive to the update interval and sliding window length.

Risk Prediction for Gastric Cancer Using GWAS-Identifie Polymorphisms, Helicobacter pylori Infection and Lifestyle-Related Risk Factors in a Japanese Population

Simple Summary

Gastric cancer remains the major cancer in Japan and worldwide. It is expected that practical intervention strategies for prevention, such as personalized approaches based on genetic risk models, will be developed. Here, we developed and validated a risk prediction model for gastric cancer using genetic, biological, and lifestyle-related risk factors. Results showed that the combination of selected GWAS-identified SNP polymorphisms and other predictors provided high discriminatory accuracy and good calibration in both the derivation and validation studies; however, the contribution of genetic factors to risk prediction was limited. The greatest contributor to risk prediction was ABCD classification (Helicobacter pylori infection-related factor).

Abstract

Background: As part of our efforts to develop practical intervention applications for cancer prevention, we investigated a risk prediction model for gastric cancer based on genetic, biological, and lifestyle-related risk factors. Methods: We conducted two independent age- and sex-matched case–control studies, the first for model derivation (696 cases and 1392 controls) and the second (795 and 795) for external validation. Using the derivation study data, we developed a prediction model by fitting a conditional logistic regression model using the predictors age, ABCD classification defined by H. pylori infection and gastric atrophy, smoking, alcohol consumption, fruit and vegetable intake, and 3 GWAS-identified polymorphisms. Performance was assessed with regard to discrimination (area under the curve (AUC)) and calibration (calibration plots and Hosmer–Lemeshow test). Results: A combination of selected GWAS-identified polymorphisms and the other predictors provided high discriminatory accuracy and good calibration in both the derivation and validation studies, with AUCs of 0.77 (95% confidence intervals: 0.75–0.79) and 0.78 (0.77–0.81), respectively. The calibration plots of both studies stayed close to the ideal calibration line. In the validation study, the environmental model (nongenetic model) was significantly more discriminative than the inclusive model, with an AUC value of 0.80 (0.77–0.82). Conclusion: The contribution of genetic factors to risk prediction was limited, and the ABCD classification (H. pylori infection-related factor) contributes most to risk prediction of gastric cancer.

Performance Metrics for the Comparative Analysis of Clinical Risk Prediction Models Employing Machine Learning

BACKGROUND

New methods such as machine learning techniques have been increasingly used to enhance the performance of risk predictions for clinical decision-making. However, commonly reported performance metrics may not be sufficient to capture the advantages of these newly proposed models for their adoption by health care professionals to improve care. Machine learning models often improve risk estimation for certain subpopulations that may be missed by these metrics.

METHODS AND RESULTS

This article addresses the limitations of commonly reported metrics for performance comparison and proposes additional metrics. Our discussions cover metrics related to overall performance, discrimination, calibration, resolution, reclassification, and model implementation. Models for predicting acute kidney injury after percutaneous coronary intervention are used to illustrate the use of these metrics.

CONCLUSIONS

We demonstrate that commonly reported metrics may not have sufficient sensitivity to identify improvement of machine learning models and propose the use of a comprehensive list of performance metrics for reporting and comparing clinical risk prediction models.

Long-term survival of mechanically ventilated patients with severe COVID-19: an observational cohort study

Background

Information is lacking regarding long-term survival and predictive factors for mortality in patients with acute hypoxemic respiratory failure due to coronavirus disease 2019 (COVID-19) and undergoing invasive mechanical ventilation. We aimed to estimate 180-day mortality of patients with COVID-19 requiring invasive ventilation, and to develop a predictive model for long-term mortality.

Methods

Retrospective, multicentre, national cohort study between March 8 and April 30, 2020 in 16 intensive care units (ICU) in Spain. Participants were consecutive adults who received invasive mechanical ventilation for COVID-19. Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection detected in positive testing of a nasopharyngeal sample and confirmed by real time reverse-transcriptase polymerase chain reaction (rt-PCR). The primary outcomes was 180-day survival after hospital admission. Secondary outcomes were length of ICU and hospital stay, and ICU and in-hospital mortality. A predictive model was developed to estimate the probability of 180-day mortality.

Results

868 patients were included (median age, 64 years [interquartile range [IQR], 56–71 years]; 72% male). Severity at ICU admission, estimated by SAPS3, was 56 points [IQR 50–63]. Prior to intubation, 26% received some type of noninvasive respiratory support. The unadjusted overall 180-day survival rates was 59% (95% CI 56–62%). The predictive factors measured during ICU stay, and associated with 180-day mortality were: age [Odds Ratio [OR] per 1-year increase 1.051, 95% CI 1.033–1.068)), SAPS3 (OR per 1-point increase 1.027, 95% CI 1.011–1.044), diabetes (OR 1.546, 95% CI 1.085–2.204), neutrophils to lymphocytes ratio (OR per 1-unit increase 1.008, 95% CI 1.001–1.016), failed attempt of noninvasive positive pressure ventilation prior to orotracheal intubation (OR 1.878 (95% CI 1.124–3.140), use of selective digestive decontamination strategy during ICU stay (OR 0.590 (95% CI 0.358–0.972) and administration of low dosage of corticosteroids (methylprednisolone 1 mg/kg) (OR 2.042 (95% CI 1.205–3.460).

Conclusion

The long-term survival of mechanically ventilated patients with severe COVID-19 reaches more than 50% and may help to provide individualized risk stratification and potential treatments.

Trial registration: ClinicalTrials.gov Identifier: NCT04379258. Registered 10 April 2020 (retrospectively registered)

Supplementary Information

The online version contains supplementary material available at 10.1186/s13613-021-00929-y.

Prediction of Cardiovascular Disease Risk Accounting for Future Initiation of Statin Treatment

Cardiovascular disease (CVD) risk-prediction models are used to identify high-risk individuals and guide statin initiation. However, these models are usually derived from individuals who might initiate statins during follow-up. We present a simple approach to address statin initiation to predict “statin-naive” CVD risk. We analyzed primary care data (2004–2017) from the UK Clinical Practice Research Datalink for 1,678,727 individuals (aged 40–85 years) without CVD or statin treatment history at study entry. We derived age- and sex-specific prediction models including conventional risk factors and a time-dependent effect of statin initiation constrained to 25% risk reduction (from trial results). We compared predictive performance and measures of public-health impact (e.g., number needed to screen to prevent 1 event) against models ignoring statin initiation. During a median follow-up of 8.9 years, 103,163 individuals developed CVD. In models accounting for (versus ignoring) statin initiation, 10-year CVD risk predictions were slightly higher; predictive performance was moderately improved. However, few individuals were reclassified to a high-risk threshold, resulting in negligible improvements in number needed to screen to prevent 1 event. In conclusion, incorporating statin effects from trial results into risk-prediction models enables statin-naive CVD risk estimation and provides moderate gains in predictive ability but had a limited impact on treatment decision-making under current guidelines in this population.

A simulation-based evaluation of machine learning models for clinical decision support: application and analysis using hospital readmission

The interest in applying machine learning in healthcare has grown rapidly in recent years. Most predictive algorithms requiring pathway implementations are evaluated using metrics focused on predictive performance, such as the c statistic. However, these metrics are of limited clinical value, for two reasons: (1) they do not account for the algorithm's role within a provider workflow; and (2) they do not quantify the algorithm's value in terms of patient outcomes and cost savings. We propose a model for simulating the selection of patients over time by a clinician using a machine learning algorithm, and quantifying the expected patient outcomes and cost savings. Using data on unplanned emergency department surgical readmissions, we show that factors such as the provider's schedule and postoperative prediction timing can have major effects on the pathway cohort size and potential cost reductions from preventing hospital readmissions.

Prognostic models of survival in patients with advanced incurable cancer: the PiPS2 observational study

BACKGROUND

The Prognosis in Palliative care Study (PiPS) prognostic survival models predict survival in patients with incurable cancer. PiPS-A (Prognosis in Palliative care Study - All), which involved clinical observations only, and PiPS-B (Prognosis in Palliative care Study - Blood), which additionally required blood test results, consist of 14- and 56-day models that combine to create survival risk categories: 'days', 'weeks' and 'months+'.

OBJECTIVES

The primary objectives were to compare PIPS-B risk categories against agreed multiprofessional estimates of survival and to validate PiPS-A and PiPS-B. The secondary objectives were to validate other prognostic models, to assess the acceptability of the models to patients, carers and health-care professionals and to identify barriers to and facilitators of clinical use.

DESIGN

This was a national, multicentre, prospective, observational, cohort study with a nested qualitative substudy using interviews with patients, carers and health-care professionals.

SETTING

Community, hospital and hospice palliative care services across England and Wales.

PARTICIPANTS

For the validation study, the participants were adults with incurable cancer, with or without capacity to consent, who had been recently referred to palliative care services and had sufficient English language. For the qualitative substudy, a subset of participants in the validation study took part, along with informal carers, patients who declined to participate in the main study and health-care professionals.

MAIN OUTCOME MEASURES

For the validation study, the primary outcomes were survival, clinical prediction of survival and PiPS-B risk category predictions. The secondary outcomes were predictions of PiPS-A and other prognostic models. For the qualitative substudy, the main outcomes were participants' views about prognostication and the use of prognostic models.

RESULTS

For the validation study, 1833 participants were recruited. PiPS-B risk categories were as accurate as agreed multiprofessional estimates of survival (61%; p = 0.851). Discrimination of the PiPS-B 14-day model (c-statistic 0.837, 95% confidence interval 0.810 to 0.863) and the PiPS-B 56-day model (c-statistic 0.810, 95% confidence interval 0.788 to 0.832) was excellent. The PiPS-B 14-day model showed some overfitting (calibration in the large -0.202, 95% confidence interval -0.364 to -0.039; calibration slope 0.840, 95% confidence interval 0.730 to 0.950). The PiPS-B 56-day model was well-calibrated (calibration in the large 0.152, 95% confidence interval 0.030 to 0.273; calibration slope 0.914, 95% confidence interval 0.808 to 1.02). PiPS-A risk categories were less accurate than agreed multiprofessional estimates of survival (p < 0.001). The PiPS-A 14-day model (c-statistic 0.825, 95% confidence interval 0.803 to 0.848; calibration in the large -0.037, 95% confidence interval -0.168 to 0.095; calibration slope 0.981, 95% confidence interval 0.872 to 1.09) and the PiPS-A 56-day model (c-statistic 0.776, 95% confidence interval 0.755 to 0.797; calibration in the large 0.109, 95% confidence interval 0.002 to 0.215; calibration slope 0.946, 95% confidence interval 0.842 to 1.05) had excellent or reasonably good discrimination and calibration. Other prognostic models were also validated. Where comparisons were possible, the other prognostic models performed less well than PiPS-B. For the qualitative substudy, 32 health-care professionals, 29 patients and 20 carers were interviewed. The majority of patients and carers expressed a desire for prognostic information and said that PiPS could be helpful. Health-care professionals said that PiPS was user friendly and may be helpful for decision-making and care-planning. The need for a blood test for PiPS-B was considered a limitation.

LIMITATIONS

The results may not be generalisable to other populations.

CONCLUSIONS

PiPS-B risk categories are as accurate as agreed multiprofessional estimates of survival. PiPS-A categories are less accurate. Patients, carers and health-care professionals regard PiPS as potentially helpful in clinical practice.

FUTURE WORK

A study to evaluate the impact of introducing PiPS into routine clinical practice is needed.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN13688211.

FUNDING

This project was funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 25, No. 28. See the NIHR Journals Library website for further project information.

Derivation of the first clinical diagnostic models for dehydration severity in patients over five years with acute diarrhea

Diarrheal diseases lead to an estimated 1.3 million deaths each year, with the majority of those deaths occurring in patients over five years of age. As the severity of diarrheal disease can vary widely, accurately assessing dehydration status remains the most critical step in acute diarrhea management. The objective of this study is to empirically derive clinical diagnostic models for assessing dehydration severity in patients over five years with acute diarrhea in low resource settings. We enrolled a random sample of patients over five years with acute diarrhea presenting to the icddr,b Dhaka Hospital. Two blinded nurses independently assessed patients for symptoms/signs of dehydration on arrival. Afterward, consecutive weights were obtained to determine the percent weight change with rehydration, our criterion standard for dehydration severity. Full and simplified ordinal logistic regression models were derived to predict the outcome of none (<3%), some (3-9%), or severe (>9%) dehydration. The reliability and accuracy of each model were assessed. Bootstrapping was used to correct for over-optimism and compare each model's performance to the current World Health Organization (WHO) algorithm. 2,172 patients were enrolled, of which 2,139 (98.5%) had complete data for analysis. The Inter-Class Correlation Coefficient (reliability) was 0.90 (95% CI = 0.87, 0.91) for the full model and 0.82 (95% CI = 0.77, 0.86) for the simplified model. The area under the Receiver-Operator Characteristic curve (accuracy) for severe dehydration was 0.79 (95% CI: 0.76-0.82) for the full model and 0.73 (95% CI: 0.70, 0.76) for the simplified model. The accuracy for both the full and simplified models were significantly better than the WHO algorithm (p<0.001). This is the first study to empirically derive clinical diagnostic models for dehydration severity in patients over five years. Once prospectively validated, the models may improve management of patients with acute diarrhea in low resource settings.

Sensitivity and specificity of tuberculosis signs and symptoms screening and adjunct role of social pathology characteristics in predicting bacteriologically confirmed tuberculosis in Myanmar

BACKGROUND

Globally, using tuberculosis signs and symptoms (TB-SS) as a screening tool has become less important due to its low sensitivity and specificity. We analyzed data from the Myanmar National Tuberculosis (TB) prevalence survey in 2010. The various TB screening models were developed to predict TB by using logistic regression analysis, and their performance on TB prediction was compared by the measures of overall performance, calibration and discrimination ability, and sensitivity and specificity to determine whether social pathology characteristics could be used as a TB screening tool.

RESULTS

Among 51,367 participants, 311 (0.6%) had bacteriologically confirmed TB, of which 37.2% were asymptomatic and 2% had a normal chest X-ray. Out of 32 various combinations of signs and symptoms, having any signs and symptoms gave the best sensitivity of 59.8% and specificity of 67.2%, but chest X-ray (CXR) alone gave the highest sensitivity (95.1%) and specificity (86.3%). The next best combination was cough only with a sensitivity of 24.4% and specificity of 85%. Other combinations had poor sensitivity (< 10%). Among various TB screening models, the overall performance R2 was higher in the combined models of social pathology and TB signs and symptoms as well as the social pathology model, compared to TB-SS models (> 10% versus < 3%), although all TB screening models were perfect to predict TB (Brier score = 0). The social pathology model shows a better calibration, more closer to 45° line of calibration plot with Hosmer-Lemeshow test p value = 0.787, than the combined models while it had a better discrimination ability in area under the curve, AUC = 80.4%, compared to TB-SS models with any signs and symptoms, AUC = 63.5% and with any cough, AUC = 57.1% (DeLong p value = 0.0001). Moreover, at the propensity score cutoff value ≥ 0.0053, the combined and social pathology models had sensitivity of ~ 80% and specificity of ~ 70%. The highest population attributable fraction to predict TB by social pathology characteristics was male gender (42.6%), age ≥ 55 years (31.0%), and underweight (30.4%).

CONCLUSION

Over one-third of bacteriologically confirmed TB was asymptomatic. The conventional TB-SS screening tool using any TB signs and symptoms had a lower sensitivity and specificity compared to CXR and social pathology screening tools. The social pathology characteristics as TB screening tool had good calibration and can improve the discrimination ability to predict TB than TB-SS screenings and should be encouraged.

Simulating Progression-Free and Overall Survival for First-Line Doublet Chemotherapy With or Without Bevacizumab in Metastatic Colorectal Cancer Patients Based on Real-World Registry Data

BACKGROUND

Simulation models utilizing real-world data have potential to optimize treatment sequencing strategies for specific patient subpopulations, including when conducting clinical trials is not feasible. We aimed to develop a simulation model to estimate progression-free survival (PFS) and overall survival for first-line doublet chemotherapy with or without bevacizumab for specific subgroups of metastatic colorectal cancer (mCRC) patients based on registry data.

METHODS

Data from 867 patients were used to develop two survival models and one logistic regression model that populated a discrete event simulation (DES). Discrimination and calibration were used for internal validation of these models separately and predicted and observed medians and Kaplan-Meier plots were compared for the integrated DES. Bootstrapping was performed to correct for optimism in the internal validation and to generate correlated sets of model parameters for use in a probabilistic analysis to reflect parameter uncertainty.

RESULTS

The survival models showed good calibration based on the regression slopes and modified Hosmer-Lemeshow statistics at 1 and 2 years, but not for short-term predictions at 0.5 years. Modified C-statistics indicated acceptable discrimination. The simulation estimated that median first-line PFS (95% confidence interval) of 219 (25%) patients could be improved from 175 days (156-199) to 269 days (246-294) if treatment would be targeted based on the highest expected PFS.

CONCLUSIONS

Extensive internal validation showed that DES accurately estimated the outcomes of treatment combination strategies for specific subpopulations, with outcomes suggesting treatment could be optimized. Although results based on real-world data are informative, they cannot replace randomized trials.

SEA-MAKE score as a tool for predicting major adverse kidney events in critically ill patients with acute kidney injury: results from the SEA-AKI study

BACKGROUND

Acute kidney injury (AKI) is a common problem in critically ill patients and associated with high rates of morbidity and mortality. Recently, Major Adverse Kidney Events (MAKE) were introduced as important kidney endpoints. If these endpoints can be predicted, then it may help the physicians to identify high-risk patients and provide the opportunity to have targeted preventive therapy. The objective of this study was to create a simplified scoring system to predict MAKE within 28 days among AKI patients in ICU.

METHODS

This is a prospective web-based multicenter cohort study that was conducted in adults who were admitted to the ICU in 17 centers across Thailand from 2013 to 2015. A predicting score was derived from the regression equation with Receiver Operating Characteristic (ROC) analysis to evaluate the diagnostic test and produce predictive models. Internal validation was obtained using the bootstrapping method.

RESULTS

From 5071 cases, 2856 (56%) had AKI. Among those with AKI, 1749 (61%) had MAKE. Among those that have MAKE, there were 1175 (41.4%) deaths, 414 (14.4%) were on dialysis and 1154 (40.7%) had non-recovery renal function. The simplified score points of low Glasgow coma scale was 3, tachypnea was 1, vasopressor use was 1, on mechanical ventilation was 2, oliguria was 2, serum creatinine rising ≥ 3 times was 5, high blood urea nitrogen was 3, low hematocrit was 2, and thrombocytopenia was 1. The area under ROC curve for optimism corrected performance was 0.80 (0.78, 0.81). When the cut-off value was 7, the sensitivity, specificity, positive likelihood ratio, and positive predictive values were 0.75, 0.76, 3.10, and 0.84, respectively. When the scoring system was calibrated, the α intercept and β slope were 1.001 and 0, respectively.

CONCLUSIONS

SEA-MAKE scoring system is a new simplified clinical tool that can be used to predict major adverse kidney events in AKI patients. The simplicity of the scoring system is highly likely to be used in resource-limited settings. However, external validation is necessary before widespread use.

Developing Predictive Models to Determine Patients in End-of-Life Care in Administrative Datasets

Introduction

In observational studies with mortality endpoints, one needs to consider how to account for subjects whose interventions appear to be part of ‘end-of-life’ care.

Objective

The objective of this study was to develop a diagnostic predictive model to identify those in end-of-life care at the time of a drug exposure.

Methods

We used data from four administrative claims datasets from 2000 to 2017. The index date was the date of the first prescription for the last new drug subjects received during their observation period. The outcome of end-of-life care was determined by the presence of one or more codes indicating terminal or hospice care. Models were developed using regularized logistic regression. Internal validation was through examination of the area under the receiver operating characteristic curve (AUC) and through model calibration in a 25% subset of the data held back from model training. External validation was through examination of the AUC after applying the model learned on one dataset to the three other datasets.

Results

The models showed excellent performance characteristics. Internal validation resulted in AUCs ranging from 0.918 (95% confidence interval [CI] 0.905–0.930) to 0.983 (95% CI 0.978–0.987) for the four different datasets. Calibration results were also very good, with slopes near unity. External validation also produced very good to excellent performance metrics, with AUCs ranging from 0.840 (95% CI 0.834–0.846) to 0.956 (95% CI 0.952–0.960).

Conclusion

These results show that developing diagnostic predictive models for determining subjects in end-of-life care at the time of a drug treatment is possible and may improve the validity of the risk profile for those treatments.

Electronic supplementary material

The online version of this article (10.1007/s40264-020-00906-7) contains supplementary material, which is available to authorized users.

A nomogram for predicting the response to exclusive enteral nutrition in adult patients with isolated colonic Crohn's disease

BACKGROUND

Isolated colonic Crohn's disease (cCD) responds less well to induction therapy with exclusive enteral nutrition (EEN) compared with ileal or ileocolonic disease in adult patients; therefore, we aimed to identify the factors that influence the response to EEN and develop a predictive nomogram model to optimize the use of EEN in cCD patients.

MATERIALS AND METHODS

Eighty-five cCD patients treated with EEN as first-line therapy at our center between 1 June 2012 and 30 June 2018 were retrospectively analyzed as the primary cohort. The primary endpoint was clinical remission after EEN therapy. Potential predictive factors for the efficacy of EEN were assessed by univariate and multivariate analyses, and a nomogram to predict the response to EEN therapy in cCD patients was designed. Another 19 cCD patients were retrospectively included in the validation cohort to verify the accuracy of the nomogram model.

RESULTS

The clinical remission rates for the primary cohort and validation cohort were 52.9% and 47.4%, respectively. Pancolitis was the greatest contributor to the risk of failure to respond to EEN [odds ratio (OR) = 4.896; 95% confidence interval (CI) = 1.223-19.607; p = 0.025], lean body mass index (LBMI), colonic lesion features, simple endoscopic scores for Crohn's disease, C-reactive protein before treatment and ∆prealbumin were also related to the efficacy of EEN in cCD. The nomogram model showed robust discrimination, with an area under the receiving operating characteristic curve of 0.906.

CONCLUSION

Several predictive factors for response to EEN therapy in cCD adult patients were identified, and a promising nomogram that can predict the effect of EEN in cCD was developed.

Models estimating risk of hepatocellular carcinoma in patients with alcohol or NAFLD-related cirrhosis for risk stratification

BACKGROUND & AIMS

Hepatocellular carcinoma (HCC) risk varies dramatically in patients with cirrhosis according to well-described, readily available predictors. We aimed to develop simple models estimating HCC risk in patients with alcohol-related liver disease (ALD)-cirrhosis or non-alcoholic fatty liver disease (NAFLD)-cirrhosis and calculate the net benefit that would be derived by implementing HCC surveillance strategies based on HCC risk as predicted by our models.

METHODS

We identified 7,068 patients with NAFLD-cirrhosis and 16,175 with ALD-cirrhosis who received care in the Veterans Affairs (VA) healthcare system in 2012. We retrospectively followed them for the development of incident HCC until January 2018. We used Cox proportional hazards regression to develop and internally validate models predicting HCC risk using baseline characteristics at entry into the cohort in 2012. We plotted decision curves of net benefit against HCC screening thresholds.

RESULTS

We identified 1,278 incident cases of HCC during a mean follow-up period of 3.7 years. Mean annualized HCC incidence was 1.56% in NAFLD-cirrhosis and 1.44% in ALD-cirrhosis. The final models estimating HCC were developed separately for NAFLD-cirrhosis and ALD-cirrhosis and included 7 predictors: age, gender, diabetes, body mass index, platelet count, serum albumin and aspartate aminotransferase to √alanine aminotransferase ratio. The models exhibited very good measures of discrimination and calibration and an area under the receiver operating characteristic curve of 0.75 for NAFLD-cirrhosis and 0.76 for ALD-cirrhosis. Decision curves showed higher standardized net benefit of risk-based screening using our prediction models compared to the screen-all approach.

CONCLUSIONS

We developed simple models estimating HCC risk in patients with NAFLD-cirrhosis or ALD-cirrhosis, which are available as web-based tools (www.hccrisk.com). Risk stratification can be used to inform risk-based HCC surveillance strategies in individual patients or healthcare systems or to identify high-risk patients for clinical trials.

LAY SUMMARY

Patients with cirrhosis of the liver are at risk of getting hepatocellular carcinoma (HCC or liver cancer) and therefore it is recommended that they undergo surveillance for HCC. However, the risk of HCC varies dramatically in patients with cirrhosis, which has implications on if and how patients get surveillance, how providers counsel patients about the need for surveillance, and how healthcare systems approach and prioritize surveillance. We used readily available predictors to develop models estimating HCC risk in patients with cirrhosis, which are available as web-based tools at www.hccrisk.com.

Sample size for binary logistic prediction models: Beyond events per variable criteria

Binary logistic regression is one of the most frequently applied statistical approaches for developing clinical prediction models. Developers of such models often rely on an Events Per Variable criterion (EPV), notably EPV ≥10, to determine the minimal sample size required and the maximum number of candidate predictors that can be examined. We present an extensive simulation study in which we studied the influence of EPV, events fraction, number of candidate predictors, the correlations and distributions of candidate predictor variables, area under the ROC curve, and predictor effects on out-of-sample predictive performance of prediction models. The out-of-sample performance (calibration, discrimination and probability prediction error) of developed prediction models was studied before and after regression shrinkage and variable selection. The results indicate that EPV does not have a strong relation with metrics of predictive performance, and is not an appropriate criterion for (binary) prediction model development studies. We show that out-of-sample predictive performance can better be approximated by considering the number of predictors, the total sample size and the events fraction. We propose that the development of new sample size criteria for prediction models should be based on these three parameters, and provide suggestions for improving sample size determination.

Radiomics in nuclear medicine: robustness, reproducibility, standardization, and how to avoid data analysis traps and replication crisis

Radiomics in nuclear medicine is rapidly expanding. Reproducibility of radiomics studies in multicentre settings is an important criterion for clinical translation. We therefore performed a meta-analysis to investigate reproducibility of radiomics biomarkers in PET imaging and to obtain quantitative information regarding their sensitivity to variations in various imaging and radiomics-related factors as well as their inherent sensitivity. Additionally, we identify and describe data analysis pitfalls that affect the reproducibility and generalizability of radiomics studies. After a systematic literature search, 42 studies were included in the qualitative synthesis, and data from 21 were used for the quantitative meta-analysis. Data concerning measurement agreement and reliability were collected for 21 of 38 different factors associated with image acquisition, reconstruction, segmentation and radiomics-specific processing steps. Variations in voxel size, segmentation and several reconstruction parameters strongly affected reproducibility, but the level of evidence remained weak. Based on the meta-analysis, we also assessed inherent sensitivity to variations of 110 PET image biomarkers. SUV_mean and SUV_max were found to be reliable, whereas image biomarkers based on the neighbourhood grey tone difference matrix and most biomarkers based on the size zone matrix were found to be highly sensitive to variations, and should be used with care in multicentre settings. Lastly, we identify 11 data analysis pitfalls. These pitfalls concern model validation and information leakage during model development, but also relate to reporting and the software used for data analysis. Avoiding such pitfalls is essential for minimizing bias in the results and to enable reproduction and validation of radiomics studies.

Trajectories of femorotibial cartilage thickness among persons with or at risk of knee osteoarthritis: development of a prediction model to identify progressors

OBJECTIVE

There is significant variability in the trajectory of structural progression across people with knee osteoarthritis (OA). We aimed to identify distinct trajectories of femorotibial cartilage thickness over 2 years and develop a prediction model to identify individuals experiencing progressive cartilage loss.

METHODS

We analysed data from the Osteoarthritis Initiative (OAI) (n = 1,014). Latent class growth analysis (LCGA) was used to identify trajectories of medial femorotibial cartilage thickness assessed on magnetic resonance imaging (MRI) at baseline, 1 and 2 years. Baseline characteristics were compared between trajectory-based subgroups and a prediction model was developed including those with frequent knee symptoms at baseline (n = 686). To examine clinical relevance of the trajectories, we assessed their association with concurrent changes in knee pain and incidence of total knee replacement (TKR) over 4 years.

RESULTS

The optimal model identified three distinct trajectories: (1) stable (87.7% of the population, mean change -0.08 mm, SD 0.19); (2) moderate cartilage loss (10.0%, -0.75 mm, SD 0.16) and (3) substantial cartilage loss (2.2%, -1.38 mm, SD 0.23). Higher Western Ontario & McMaster Universities Osteoarthritis Index (WOMAC) pain scores, family history of TKR, obesity, radiographic medial joint space narrowing (JSN) ≥1 and pain duration ≤1 year were predictive of belonging to either the moderate or substantial cartilage loss trajectory [area under the curve (AUC) 0.79, 95% confidence interval (CI) 0.74, 0.84]. The two progression trajectories combined were associated with pain progression (OR 1.99, 95% CI 1.34, 2.97) and incidence of TKR (OR 4.34, 1.62, 11.62).

CONCLUSIONS

A minority of individuals follow a progressive cartilage loss trajectory which was strongly associated with poorer clinical outcomes. If externally validated, the prediction model may help to select individuals who may benefit from cartilage-targeted therapies.

Performance of SAPS II according to ICU length of stay: Protocol for an observational study

BACKGROUND

Severity scores, including the Simplified Acute Physiology Score (SAPS) II, are widely used in the intensive care unit (ICU) to predict mortality outcomes using data from ICU admission or shortly hereafter. For patients with longer ICU length of stay (LOS), the predictive performance of admission-based severity scores may deteriorate compared to patients with shorter ICU LOS. This protocol and statistical analysis plan outlines a study that will assess the influence of ICU LOS on the performance of SAPS II for predicting 90-day post-ICU mortality.

METHODS

A Danish nationwide cohort study including adult (≥18 years) ICU patients admitted to a Danish ICU between 1 January 2012 and 30 June 2016. The study will be conducted using the Danish Intensive Care Database (DID), which contains data routinely, prospectively, and consecutively reported for all Danish ICU admissions. Discrimination of SAPS II for predicting 90-day post-ICU mortality will be assessed for the entire cohort and stratified according to ICU LOS. A first-level recalibration of SAPS II will be performed, and if adequate, standardised mortality ratios and calibration stratified according to ICU LOS will be reported.

CONCLUSIONS

The outlined large, nationwide cohort study will provide important, contemporary information about the influence of ICU LOS on severity score performance relevant for ICU clinicians, researchers, and administrators. Publication of the protocol and statistical analysis plan prior to study conduct ensures transparency, and limits the risk of publication bias, post hoc changes in analyses, and challenges with multiple comparisons.

Novel nomograms to predict recurrence and progression in primary non-muscle-invasive bladder cancer: validation of predictive efficacy in comparison with European Organization of Research and Treatment of Cancer scoring system

PURPOSE

To develop and validate novel nomograms to predict recurrence and progression after transurethral resection of bladder tumor (TURBT) in Korean patients with non-muscle-invasive bladder cancer (NMIBC).

METHODS

We retrospectively analyzed the clinical data on 970 newly diagnosed NMIBC patients after TURBT between 2000 and 2013 in a single institution. We used multivariate Cox proportional hazard models to identify the significant predictors of recurrence and progression, which resulted in the construction of the nomograms predicting the 5-year probability of recurrence and progression. We internally validated the nomograms using the area under the receiver-operating characteristics' curves and calibration plots. In addition, the clinical usefulness of each nomogram was assessed and compared with that of the European Organization of Research and Treatment of Cancer (EORTC)-scoring system using decision curve analysis (DCA).

RESULTS

The significant factors related to recurrence were gross hematuria at diagnosis, previous or concomitant upper urinary tract urothelial carcinoma (UTUC), pT1 tumor, high tumor grade, multiple tumors, and intravesical therapy. The significant predictors of progression were previous or concomitant UTUC, pT1 tumor, high tumor grade, carcinoma in situ, and lymphovascular invasion. The 5-year predictive accuracy of each nomogram was 65% for recurrence and 70% for progression, respectively. Compared with the EORTC-scoring system, the nomograms were generally well calibrated. On DCA, each nomogram presented better net benefit gains than did the EORTC-scoring system across a wide range of threshold probabilities.

CONCLUSIONS

Our novel nomograms are not completely accurate, but they show a reasonable level of discriminative ability, adequate calibration, and meaningful net benefit gain for the prediction of recurrence and progression after TURBT in Korean NMIBC patients. Additional external validation will be required to generalize the nomograms which we developed.

Development of models estimating the risk of hepatocellular carcinoma after antiviral treatment for hepatitis C

BACKGROUND & AIMS

Most patients with hepatitis C virus (HCV) infection will undergo antiviral treatment with direct-acting antivirals (DAAs) and achieve sustained virologic response (SVR). We aimed to develop models estimating hepatocellular carcinoma (HCC) risk after antiviral treatment.

METHODS

We identified 45,810 patients who initiated antiviral treatment in the Veterans Affairs (VA) national healthcare system from 1/1/2009 to 12/31/2015, including 29,309 (64%) DAA-only regimens and 16,501 (36%) interferon ± DAA regimens. We retrospectively followed patients until 6/15/2017 to identify incident cases of HCC. We used Cox proportional hazards regression to develop and internally validate models predicting HCC risk using baseline characteristics at the time of antiviral treatment.

RESULTS

We identified 1,412 incident cases of HCC diagnosed at least 180 days after initiation of antiviral treatment during a mean follow-up of 2.5 years (range 1.0-7.5 years). Models predicting HCC risk after antiviral treatment were developed and validated separately for four subgroups of patients: cirrhosis/SVR, cirrhosis/no SVR, no cirrhosis/SVR, no cirrhosis/no SVR. Four predictors (age, platelet count, serum aspartate aminotransferase/√alanine aminotransferase ratio and albumin) accounted for most of the models' predictive value, with smaller contributions from sex, race-ethnicity, HCV genotype, body mass index, hemoglobin and serum alpha-fetoprotein. Fitted models were well-calibrated with very good measures of discrimination. Decision curves demonstrated higher net benefit of using model-based HCC risk estimates to determine whether to recommend screening or not compared to the screen-all or screen-none strategies.

CONCLUSIONS

We developed and internally validated models that estimate HCC risk following antiviral treatment. These models are available as web-based tools that can be used to inform risk-based HCC surveillance strategies in individual patients.

LAY SUMMARY

Most patients with hepatitis C virus have been treated or will be treated with direct-acting antivirals. It is important that we can model the risk of hepatocellular carcinoma in these patients, so that we develop the optimum screening strategy that avoids unnecessary screening, while adequately screening those at increased risk. Herein, we have developed and validated models that are available as web-based tools that can be used to guide screening strategies.

The Prognosis in Palliative care Study II (PiPS2): study protocol for a multi-centre, prospective, observational, cohort study

BACKGROUND

More accurate methods of prognostication are likely to lead to improvements in the quality of care of patients approaching the ends of their lives. The Prognosis in Palliative care Scales (PiPS) are prognostic models of survival. The scores are calculated using simple clinical data and observations. There are two separate PiPS models; PiPS-A for patients without blood test results and PiPS-B for patients with blood test results. Both models predict whether a patient is likely to live for "days", "weeks" or "months" and have been shown to perform as well as clinicians' estimates of survival. PiPS-B has also been found to be significantly better than doctors' estimates of survival. We report here a protocol for the validation of PiPS and for the evaluation of the accuracy of other prognostic tools in a new, larger cohort of patients with advanced cancer.

METHODS

This is a national, multi-centre, prospective, observational cohort study, aiming to recruit 1778 patients via palliative care services across England and Wales. Eligible patients have advanced, incurable cancer and have recently been referred to palliative care services. Patients with or without capacity are included in the study. The primary outcome is the accuracy of PiPS predictions and the difference in accuracy between these predictions and the clinicians' estimates of survival; with PiPS-B being the main model of interest. The secondary outcomes include the accuracy of predictions by the Palliative Prognostic Index (PPI), Palliative Performance Scale (PPS), Palliative Prognostic score (PaP) and the Feliu Prognostic Nomogram (FPN) compared with actual patient survival and clinicians' estimates of survival. A nested qualitative sub-study using face-to-face interviews with patients, carers and clinicians is also being undertaken to assess the acceptability of the prognostic models and to identify barriers and facilitators to clinical use.

DISCUSSION

The study closed to recruitment at the end of April 2018 having exceeded the required sample size of 1778 patients. The qualitative sub-study is nearing completion. This demonstrates the feasibility of recruiting large numbers of participants to a prospective palliative care study.

TRIAL REGISTRATION

ISRCTN13688211 (registration date: 28/06/2016).

Use of risk-adjusted CUSUM charts to monitor 30-day mortality in Danish hospitals

Background

Monitoring hospital outcomes and clinical processes as a measure of clinical performance is an integral part of modern health care. The risk-adjusted cumulative sum (CUSUM) chart is a frequently used sequential analysis technique that can be implemented to monitor a wide range of different types of outcomes.

Objective

The aim of this study was to describe how risk-adjusted CUSUM charts based on population-based nationwide medical registers were used to monitor 30-day mortality in Danish hospitals and to give an example on how alarms of increased hospital mortality from the charts can guide further in-depth analyses.

Materials and methods

We used routinely collected administrative data from the Danish National Patient Registry and the Danish Civil Registration System to create risk-adjusted CUSUM charts. We monitored 30-day mortality after hospital admission with one of 77 selected diagnoses in 24 hospital units in Denmark in 2015. The charts were set to detect a 50% increase in 30-day mortality, and control limits were determined by simulations.

Results

Among 1,085,576 hospital admissions, 441,352 admissions had one of the 77 selected diagnoses as their primary diagnosis and were included in the risk-adjusted CUSUM charts. The charts yielded a total of eight alarms of increased mortality. The median of the hospitals’ estimated average time to detect a 50% increase in 30-day mortality was 50 days (interquartile interval, 43;54). In the selected example of an alarm, descriptive analyses indicated performance problems with 30-day mortality following hip fracture surgery and diagnosis of chronic obstructive pulmonary disease.

Conclusion

The presented implementation of risk-adjusted CUSUM charts can detect significant increases in 30-day mortality within 2 months, on average, in most Danish hospitals. Together with descriptive analyses, it was possible to use an alarm from a risk-adjusted CUSUM chart to identify potential performance problems.

Development and internal validation of the Simplified Mortality Score for the Intensive Care Unit (SMS-ICU)

BACKGROUND

Intensive care unit (ICU) mortality prediction scores deteriorate over time, and their complexity decreases clinical applicability and commonly causes problems with missing data. We aimed to develop and internally validate a new and simple score that predicts 90-day mortality in adults upon acute admission to the ICU: the Simplified Mortality Score for the Intensive Care Unit (SMS-ICU).

METHODS

We used data from an international cohort of 2139 patients acutely admitted to the ICU and 1947 ICU patients with severe sepsis/septic shock from 2009 to 2016. We performed multiple imputations for missing data and used binary logistic regression analysis with variable selection by backward elimination, followed by conversion to a simple point-based score. We assessed the apparent performance and validated the score internally using bootstrapping to present optimism-corrected performance estimates.

RESULTS

The SMS-ICU comprises seven variables available in 99.5% of the patients: two numeric variables: age and lowest systolic blood pressure, and five dichotomous variables: haematologic malignancy/metastatic cancer, acute surgical admission and use of vasopressors/inotropes, respiratory support and renal replacement therapy. Discrimination (area under the receiver operating characteristic curve) was 0.72 (95% CI: 0.71-0.74), overall performance (Nagelkerke's R² ) was 0.19 and calibration (intercept and slope) was 0.00 and 0.99, respectively. Optimism-corrected performance was similar to apparent performance.

CONCLUSIONS

The SMS-ICU predicted 90-day mortality with reasonable and stable performance. If performance remains adequate after external validation, the SMS-ICU could prove a valuable tool for ICU clinicians and researchers because of its simplicity and expected very low number of missing values.

Development of Clinical Prediction Models for Surgery and Complications in Crohn's Disease

BACKGROUND AND AIMS

Crohn's disease-related complications account for a substantial proportion of inflammatory bowel disease-associated health care expenditure. Identifying patients at risk for complications may allow for targeted use of early therapeutic interventions to offset this natural course. We aimed to develop risk prediction models for Crohn's disease-related surgery and complications.

METHODS

Using data from the Randomised Evaluation of an Algorithm for Crohn's Disease cluster-randomised clinical Trial [REACT], which involved 1898 patients from 40 community practices, separate prediction models were derived and internally validated for predicting Crohn's disease-related surgery and disease-related complications [defined as the first disease-related surgery, hospitalisation, or complication within 24 months]. Model performance was assessed in terms of discrimination and calibration, decision curves, and net benefit analyses.

RESULTS

There were 130 [6.8%] disease-related surgeries and 504 [26.6%] complications during the 24-month follow-up period. Selected baseline predictors of surgery included age, gender, disease location, Harvey-Bradshaw Index [HBI] score, stool frequency, antimetabolite or 5-aminosalicylate use, and the presence of a fistula, abscess, or abdominal mass. Selected predictors of complications included those same factors for surgery, plus corticosteroid or anti-tumour necrosis factor use, but excluded 5-aminosalicylate use. Discrimination ability, as measured by validated c-statistics, was 0.70 and 0.62 for the surgery and complication models, respectively. Score charts and nomograms were developed to facilitate future risk score calculation.

CONCLUSIONS

Separate risk prediction models for Crohn's disease-related surgery and complications were developed using clinical trial data involving community gastroenterology practices. These models could be used to guide Crohn's disease management. External validation is warranted.

A Point System to Forecast Hepatocellular Carcinoma Risk Before and After Treatment Among Persons with Chronic Hepatitis C

BACKGROUND

Risk of hepatocellular carcinoma (HCC) may be difficult to determine in the clinical setting.

AIM

Develop a scoring system to forecast HCC risk among patients with chronic hepatitis C.

METHODS

Using data from the Chronic Hepatitis Cohort Study collected during 2005-2014, we derived HCC risk scores for males and females using an extended Cox model with aspartate aminotransferase-to-platelet ratio index (APRI) as a time-dependent variables and mean Kaplan-Meier survival functions from patient data at two study sites, and used data collected at two separate sites for external validation. For model calibration, we used the Greenwood-Nam-D'Agostino goodness-of-fit statistic to examine differences between predicted and observed risk.

RESULTS

Of 12,469 patients (1628 with a history of sustained viral response [SVR]), 504 developed HCC; median follow-up was 6 years. Final predictors in the model included age, alcohol abuse, interferon-based treatment response, and APRI. Point values, ranging from -3 to 14 (males) and -3 to 12 (females), were established using hazard ratios of the predictors aligned with 1-, 3-, and 5-year Kaplan-Meier survival probabilities of HCC. Discriminatory capacity was high (c-index 0.82 males and 0.84 females) and external calibration demonstrated no differences between predicted and observed HCC risk for 1-, 3-, and 5-year forecasts among males (all p values >0.97) and for 3- and 5-year risk among females (all p values >0.87).

CONCLUSION

This scoring system, based on age, alcohol abuse history, treatment response, and APRI, can be used to forecast up to a 5-year risk of HCC among hepatitis C patients before and after SVR.

The Michigan Risk Score to predict peripherally inserted central catheter-associated thrombosis

Essentials How best to quantify thrombosis risk with peripherally inserted central catheters (PICC) is unknown. Data from a registry were used to develop the Michigan Risk Score (MRS) for PICC thrombosis. Five risk factors were associated with PICC thrombosis and used to develop a risk score. MRS was predictive of the risk of PICC thrombosis and can be useful in clinical practice.

SUMMARY

Background Peripherally inserted central catheters (PICCs) are associated with upper extremity deep vein thrombosis (DVT). We developed a score to predict risk of PICC-related thrombosis. Methods Using data from the Michigan Hospital Medicine Safety Consortium, image-confirmed upper-extremity DVT cases were identified. A logistic, mixed-effects model with hospital-specific random intercepts was used to identify factors associated with PICC-DVT. Points were assigned to each predictor, stratifying patients into four classes of risk. Internal validation was performed by bootstrapping with assessment of calibration and discrimination of the model. Results Of 23 010 patients who received PICCs, 475 (2.1%) developed symptomatic PICC-DVT. Risk factors associated with PICC-DVT included: history of DVT; multi-lumen PICC; active cancer; presence of another CVC when the PICC was placed; and white blood cell count greater than 12 000. Four risk classes were created based on thrombosis risk. Thrombosis rates were 0.9% for class I, 1.6% for class II, 2.7% for class III and 4.7% for class IV, with marginal predicted probabilities of 0.9% (0.7, 1.2), 1.5% (1.2, 1.9), 2.6% (2.2, 3.0) and 4.5% (3.7, 5.4) for classes I, II, III, and IV, respectively. The risk classification rule was strongly associated with PICC-DVT, with odds ratios of 1.68 (95% CI, 1.19, 2.37), 2.90 (95% CI, 2.09, 4.01) and 5.20 (95% CI, 3.65, 7.42) for risk classes II, III and IV vs. risk class I, respectively. Conclusion The Michigan PICC-DVT Risk Score offers a novel way to estimate risk of DVT associated with PICCs and can help inform appropriateness of PICC insertion.

Sediment quality benchmarks for assessing oil-related impacts to the deep-sea benthos

Paired sediment contaminant and benthic infaunal data from prior studies following the 2010 Deepwater Horizon (DWH) oil spill in the Gulf of Mexico were analyzed using logistic regression models (LRMs) to derive sediment quality benchmarks for assessing risks of oil-related impacts to the deep-sea benthos. Sediment total polycyclic aromatic hydrocarbon (PAH) and total petroleum hydrocarbon (TPH) concentrations were used as measures of oil exposure. Taxonomic richness (average number of taxa/sample) was selected as the primary benthic response variable. Data are from 37 stations (1300-1700 m water depth) in fine-grained sediments (92%-99% silt-clay) sampled within 200 km of the DWH wellhead (most within 40 km) in 2010 and 32 stations sampled in 2011 (29 of which were common to both years). Results suggest the likelihood of impacts to benthic macrofauna and meiofauna communities is low (<20%) at TPH concentrations of less than 606 mg kg^-1 (ppm dry weight) and 700 mg kg^-1 respectively, high (>80%) at concentrations greater than 2144 mg kg^-1 and 2359 mg kg^-1 respectively, and intermediate at concentrations in between. For total PAHs, the probability of impacts is low (<20%) at concentrations of less than 4.0 mg kg^-1 (ppm) for both macrofauna and meiofauna, high (>80%) at concentrations greater than 24 mg kg^-1 and 25 mg kg^-1 for macrofauna and meiofauna, respectively, and intermediate at concentrations in between. Although numerical sediment quality guidelines (SQGs) are available for total PAHs and other chemical contaminants based on bioeffect data for shallower estuarine, marine, and freshwater biota, to our knowledge, none have been developed for measures of total oil (e.g., TPH) or specifically for deep-sea benthic applications. The benchmarks presented herein provide valuable screening tools for evaluating the biological significance of observed oil concentrations in similar deep-sea sediments following future spills and as potential restoration targets to aid in managing recovery. Integr Environ Assess Manag 2017;13:840-851. Published 2017. This article is a US Government work and is in the public domain in the USA.

A Model to Predict Central-Line-Associated Bloodstream Infection Among Patients With Peripherally Inserted Central Catheters: The MPC Score

BACKGROUND Peripherally inserted central catheters (PICCs) are associated with central-line-associated bloodstream infections (CLABSIs). However, no tools to predict risk of PICC-CLABSI have been developed. OBJECTIVE To operationalize or prioritize CLABSI risk factors when making decisions regarding the use of PICCs using a risk model to estimate an individual's risk of PICC-CLABSI prior to device placement. METHODS Using data from the Michigan Hospital Medicine Safety consortium, patients that experienced PICC-CLABSI between January 2013 and October 2016 were identified. A Cox proportional hazards model with robust sandwich standard error estimates was then used to identify factors associated with PICC-CLABSI. Based on regression coefficients, points were assigned to each predictor and summed for each patient to create the Michigan PICC-CLABSI (MPC) score. The predictive performance of the score was assessed using time-dependent area-under-the-curve (AUC) values. RESULTS Of 23,088 patients that received PICCs during the study period, 249 patients (1.1%) developed a CLABSI. Significant risk factors associated with PICC-CLABSI included hematological cancer (3 points), CLABSI within 3 months of PICC insertion (2 points), multilumen PICC (2 points), solid cancers with ongoing chemotherapy (2 points), receipt of total parenteral nutrition (TPN) through the PICC (1 point), and presence of another central venous catheter (CVC) at the time of PICC placement (1 point). The MPC score was significantly associated with risk of CLABSI (P<.0001). For every point increase, the hazard ratio of CLABSI increased by 1.63 (95% confidence interval, 1.56-1.71). The area under the receiver-operating-characteristics curve was 0.67 to 0.77 for PICC dwell times of 6 to 40 days, which indicates good model calibration. CONCLUSION The MPC score offers a novel way to inform decisions regarding PICC use, surveillance of high-risk cohorts, and utility of blood cultures when PICC-CLABSI is suspected. Future studies validating the score are necessary. Infect Control Hosp Epidemiol 2017;38:1155-1166.

Simplified Mortality Score for the Intensive Care Unit (SMS-ICU): protocol for the development and validation of a bedside clinical prediction rule

INTRODUCTION

Mortality prediction scores are widely used in intensive care units (ICUs) and in research, but their predictive value deteriorates as scores age. Existing mortality prediction scores are imprecise and complex, which increases the risk of missing data and decreases the applicability bedside in daily clinical practice. We propose the development and validation of a new, simple and updated clinical prediction rule: the Simplified Mortality Score for use in the Intensive Care Unit (SMS-ICU).

METHODS AND ANALYSIS

During the first phase of the study, we will develop and internally validate a clinical prediction rule that predicts 90-day mortality on ICU admission. The development sample will comprise 4247 adult critically ill patients acutely admitted to the ICU, enrolled in 5 contemporary high-quality ICU studies/trials. The score will be developed using binary logistic regression analysis with backward stepwise elimination of candidate variables, and subsequently be converted into a point-based clinical prediction rule. The general performance, discrimination and calibration of the score will be evaluated, and the score will be internally validated using bootstrapping. During the second phase of the study, the score will be externally validated in a fully independent sample consisting of 3350 patients included in the ongoing Stress Ulcer Prophylaxis in the Intensive Care Unit trial. We will compare the performance of the SMS-ICU to that of existing scores.

ETHICS AND DISSEMINATION

We will use data from patients enrolled in studies/trials already approved by the relevant ethical committees and this study requires no further permissions. The results will be reported in accordance with the Transparent Reporting of multivariate prediction models for Individual Prognosis Or Diagnosis (TRIPOD) statement, and submitted to a peer-reviewed journal.

Algorithm for predicting death among older adults in the home care setting: study protocol for the Risk Evaluation for Support: Predictions for Elder-life in the Community Tool (RESPECT)

INTRODUCTION

Older adults living in the community often have multiple, chronic conditions and functional impairments. A challenge for healthcare providers working in the community is the lack of a predictive tool that can be applied to the broad spectrum of mortality risks observed and may be used to inform care planning.

OBJECTIVE

To predict survival time for older adults in the home care setting. The final mortality risk algorithm will be implemented as a web-based calculator that can be used by older adults needing care and by their caregivers.

DESIGN

Open cohort study using the Resident Assessment Instrument for Home Care (RAI-HC) data in Ontario, Canada, from 1 January 2007 to 31 December 2013.

PARTICIPANTS

The derivation cohort will consist of ∼437 000 older adults who had an RAI-HC assessment between 1 January 2007 and 31 December 2012. A split sample validation cohort will include ∼122 000 older adults with an RAI-HC assessment between 1 January and 31 December 2013.

MAIN OUTCOME MEASURES

Predicted survival from the time of an RAI-HC assessment. All deaths (n≈245 000) will be ascertained through linkage to a population-based registry that is maintained by the Ministry of Health in Ontario.

STATISTICAL ANALYSIS

Proportional hazards regression will be estimated after assessment of assumptions. Predictors will include sociodemographic factors, social support, health conditions, functional status, cognition, symptoms of decline and prior healthcare use. Model performance will be evaluated for 6-month and 12-month predicted risks, including measures of calibration (eg, calibration plots) and discrimination (eg, c-statistics). The final algorithm will use combined development and validation data.

ETHICS AND DISSEMINATION

Research ethics approval has been granted by the Sunnybrook Health Sciences Centre Review Board. Findings will be disseminated through presentations at conferences and in peer-reviewed journals.

TRIAL REGISTRATION NUMBER

NCT02779309, Pre-results.

A comprehensive validation of HBV-related acute-on-chronic liver failure models to assist decision-making in targeted therapeutics

This research utilized an external longitudinal dataset of hepatitis B virus-related acute-on-chronic liver failure (HBV-ACLF) to compare and validate various predictive models that support the current recommendations to select the most effective predictive risk models to estimate short- and long-term mortality and facilitate decision-making about preferable therapeutics for HBV-ACLF patients. Twelve ACLF prognostic models were developed after a systematic literature search using the longitudinal data of 232 HBV-ACLF patients on the waiting list for liver transplantation (LT). Four statistical measures, the constant (A) and slope (B) of the fitted line, the area under the curve (C) and the net benefit (D), were calculated to assess and compare the calibration, discrimination and clinical usefulness of the 12 predictive models. According to the model calibration and discrimination, the logistic regression models (LRM2) and the United Kingdom model of end-stage liver disease(UKELD) were selected as the best predictive models for both 3-month and 5-year outcomes. The decision curve summarizes the benefits of intervention relative to the costs of unnecessary treatment. After the comprehensive validation and comparison of the currently used models, LRM2 was confirmed as a markedly effective prognostic model for LT-free HBV-ACLF patients for assisting targeted and standardized therapeutic decisions.

Assessing calibration of prognostic risk scores

Current methods used to assess calibration are limited, particularly in the assessment of prognostic models. Methods for testing and visualizing calibration (e.g. the Hosmer-Lemeshow test and calibration slope) have been well thought out in the binary regression setting. However, extension of these methods to Cox models is less well known and could be improved. We describe a model-based framework for the assessment of calibration in the binary setting that provides natural extensions to the survival data setting. We show that Poisson regression models can be used to easily assess calibration in prognostic models. In addition, we show that a calibration test suggested for use in survival data has poor performance. Finally, we apply these methods to the problem of external validation of a risk score developed for the general population when assessed in a special patient population (i.e. patients with particular comorbidities, such as rheumatoid arthritis).