A discussion of calibration techniques for evaluating binary and categorical predictive models

Validation of serum gamma-glutamyl transferase activity and body weight information for identifying dairy calves that are too young to be transported to auction markets in Canada

Dairy calves are at risk of being stressed when transported during the first week of life. A new Canadian federal rule will forbid transportation of calves younger than 9 d old to auction market. However, in the absence of reliable information to determine birth date, other indirect methods would be of interest. This study aimed to determine the prediction accuracy of body weight, Brix refractometry, and serum gamma-glutamyl transferase (GGT) activity for determining if a calf was not fit to be transported (i.e., <9 d old). For this purpose, we used 284 calves with a known birth date from a cross-sectional and a prospective cohort study. A logistic regression model was built based on multivariable analysis as well as a misclassification cost term analysis. Because of the collinearity between GGT activity and Brix value and lower discrimination of Brix value, the GGT activity was retained for the main model. The final logistic regression model contained body weight and log-transformed GGT activity value. The misclassifications of the logistic model was minimized using a model probability threshold ≥0.55 with a sensitivity of 70.4% and a specificity of 77.3%. This probability threshold was relatively robust for various prevalence and false negative to false positive cost ratios. The prediction accuracy of this model was moderate at the individual level, but is helpful in calves with a reasonable suspicion of being less than 9 d old.

Development of an individualized risk calculator for poor functioning in young people victimized during childhood: A longitudinal cohort study

BACKGROUND

Childhood victimization elevates the average risk of developing functional impairment in adulthood. However, not all victimized children demonstrate poor outcomes. Although research has described factors that confer vulnerability or resilience, it is unknown if this knowledge can be translated to accurately identify the most vulnerable victimized children.

OBJECTIVE

To build and internally validate a risk calculator to identify those victimized children who are most at risk of functional impairment at age 18 years.

PARTICIPANTS

We utilized data from the Environmental Risk (E-Risk) Longitudinal Twin Study, a nationally-representative birth cohort of 2232 UK children born in 1994-95.

METHODS

Victimization exposure was assessed repeatedly between ages 5 and 12 years along with a range of individual-, family- and community-level predictors. Functional outcomes were assessed at age 18 years. We developed and evaluated a prediction model for psychosocial disadvantage and economic disadvantage using the Least Absolute Shrinkage and Selection Operator (LASSO) regularized regression with nested 10-fold cross-validation.

RESULTS

The model predicting psychosocial disadvantage following childhood victimization retained 12 of 22 predictors, had an area under the curve (AUC) of 0.65, and was well-calibrated within the range of 40-70% predicted risk. The model predicting economic disadvantage retained 10 of 22 predictors, achieved excellent discrimination (AUC = 0.80), and a high degree of calibration.

CONCLUSIONS

Prediction modelling techniques can be applied to estimate individual risk for poor functional outcomes in young adulthood following childhood victimization. Such risk prediction tools could potentially assist practitioners to target interventions, which is particularly useful in a context of scarce resources.

Artificial neural networks improve and simplify intensive care mortality prognostication: a national cohort study of 217,289 first-time intensive care unit admissions

Purpose

We investigated if early intensive care unit (ICU) scoring with the Simplified Acute Physiology Score (SAPS 3) could be improved using artificial neural networks (ANNs).

Methods

All first-time adult intensive care admissions in Sweden during 2009–2017 were included. A test set was set aside for validation. We trained ANNs with two hidden layers with random hyper-parameters and retained the best ANN, determined using cross-validation. The ANNs were constructed using the same parameters as in the SAPS 3 model. The performance was assessed with the area under the receiver operating characteristic curve (AUC) and Brier score.

Results

A total of 217,289 admissions were included. The developed ANN (AUC 0.89 and Brier score 0.096) was found to be superior (p <10⁻¹⁵ for AUC and p <10⁻⁵ for Brier score) in early prediction of 30-day mortality for intensive care patients when compared with SAPS 3 (AUC 0.85 and Brier score 0.109). In addition, a simple, eight-parameter ANN model was found to perform just as well as SAPS 3, but with better calibration (AUC 0.85 and and Brier score 0.106, p <10⁻⁵). Furthermore, the ANN model was superior in correcting mortality for age.

Conclusion

ANNs can outperform the SAPS 3 model for early prediction of 30-day mortality for intensive care patients.

Prediction of good neurological recovery after out-of-hospital cardiac arrest: A machine learning analysis

BACKGROUND

This study aimed to train, validate and compare predictive models that use machine learning analysis for good neurological recovery in OHCA patients.

METHODS

Adult OHCA patients who had a presumed cardiac etiology and a sustained return of spontaneous circulation between 2013 and 2016 were analyzed; 80% of the individuals were analyzed for training and 20% were analyzed for validation. We developed using six machine learning algorithms: logistic regression (LR), extreme gradient boosting (XGB), support vector machine, random forest, elastic net (EN), and neural network. Variables that could be obtained within 24 hours of the emergency department visit were used. The area under the receiver operation curve (AUROC) was calculated to assess the discrimination. Calibration was assessed by the Hosmer-Lemeshow test. Reclassification was assessed by using the continuous net reclassification index (NRI).

RESULTS

A total of 19,860 OHCA patients were included in the analysis. Of the 15,888 patients in the training group, 2228 (14.0%) had a good neurological recovery; of the 3972 patients in the validation group, 577 (14.5%) had a good neurological recovery. The LR, XGB, and EN models showed the highest discrimination powers (AUROC (95% CI)) of 0.949 (0.941-0.957) for all), and all three models were well calibrated (Hosmer-Lemeshow test: p >0.05). The XGB model reclassified patients according to their true risk better than the LR model (NRI: 0.110), but the EN model reclassified patients worse than the LR model (NRI: -1.239).

CONCLUSION

The best performing machine learning algorithm was the XGB and LR algorithm.

Prediction of mortality following pediatric heart transplant using machine learning algorithms

BACKGROUND

Optimizing transplant candidates' priority for donor organs depends on the accurate assessment of post-transplant outcomes. Due to the complexity of transplantation and the wide range of possible serious complications, recipient outcomes are difficult to predict accurately using conventional multivariable regression. Therefore, we evaluated the utility of 3 ML algorithms for predicting mortality after pediatric HTx.

METHODS

We identified patients <18 years of age receiving HTx in 2006-2015 in the UNOS Registry database. Mortality within 1, 3, or 5 years was predicted using classification and regression trees, RFs, and ANN. Each model was trained using cross-validation, then validated in a separate testing set. Model performance was primarily evaluated by the area under the receiver operating characteristic (AUC) curve.

RESULTS

The training set included 2802 patients, whereas 700 were included in the testing set. RF achieved the best fit to the training data with AUCs of 0.74, 0.68, and 0.64 for 1-, 3-, and 5-year mortality, respectively, and performed best in the testing data, with AUCs of 0.72, 0.61, and 0.60, respectively. Nevertheless, sensitivity was poor across models (training: 0.22-0.58; testing: 0.07-0.49).

DISCUSSION

ML algorithms demonstrated fair predictive utility in both training and testing data, but the sensitivity of these algorithms was generally poor. With the registry missing data on many determinants of long-term survival, the ability of ML methods to predict mortality after pediatric HTx may be fundamentally limited.

Population-based dementia prediction model using Korean public health examination data: A cohort study

The early identification and prevention of dementia is important for reducing its worldwide burden and increasing individuals’ quality of life. Although several dementia prediction models have been developed, there remains a need for a practical and precise model targeted to middle-aged and Asian populations. Here, we used national Korean health examination data from adults (331,126 individuals, 40–69 years of age, mean age: 52 years) from 2002–2003 to predict the incidence of dementia after 10 years. We divided the dataset into two cohorts to develop and validate of our prediction model. Cox proportional hazards models were used to construct dementia prediction models for the total group and sex-specific subgroups. Receiver operating characteristics curves, C-statistics, calibration plots, and cumulative hazards were used to validate model performance. Discriminative accuracy as measured by C-statistics was 0.81 in the total group (95% confidence interval (CI) = 0.81 to 0.82), 0.81 in the male subgroup (CI = 0.80 to 0.82), and 0.81 in the female subgroup (CI = 0.80 to 0.82). Significant risk factors for dementia in the total group were age; female sex; underweight; current hypertension; comorbid psychiatric or neurological disorder; past medical history of cardiovascular disease, diabetes mellitus, or hypertension; current smoking; and no exercise. All identified risk factors were statistically significant in the sex-specific subgroups except for low body weight and current hypertension in the female subgroup. These results suggest that public health examination data can be effectively used to predict dementia and facilitate the early identification of dementia within a middle-aged Asian population.