Regression shrinkage methods for clinical prediction models do not guarantee improved performance: Simulation study

Machine learning-assisted decision-support models to better predict patients with calculous pyonephrosis

Background

To develop a machine learning (ML)-assisted model capable of accurately identifying patients with calculous pyonephrosis before making treatment decisions by integrating multiple clinical characteristics.

Methods

We retrospectively collected data from patients with obstructed hydronephrosis who underwent retrograde ureteral stent insertion, percutaneous nephrostomy (PCN), or percutaneous nephrolithotomy (PCNL). The study cohort was divided into training and testing datasets in a 70:30 ratio for further analysis. We developed 5 ML-assisted models from 22 clinical features using logistic regression (LR), LR optimized by least absolute shrinkage and selection operator (Lasso) regularization (Lasso-LR), support vector machine (SVM), extreme gradient boosting (XGBoost), and random forest (RF). The area under the curve (AUC) was applied to determine the model with the highest discrimination. Decision curve analysis (DCA) was used to investigate the clinical net benefit associated with using the predictive models.

Results

A total of 322 patients were included, with 225 patients in the training dataset, and 97 patients in the testing dataset. The XGBoost model showed good discrimination with the AUC, accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of 0.981, 0.991, 0.962, 1.000, 1.000, and 0.989, respectively, followed by SVM [AUC =0.985, 95% confidence interval (CI): 0.970–1.000], Lasso-LR (AUC =0.977, 95% CI: 0.958–0.996), LR (AUC =0.936, 95% CI: 0.905–0.968), and RF (AUC =0.920, 95% CI: 0.870–0.970). Validation of the model showed that SVM yielded the highest AUC (0.977, 95% CI: 0.952–1.000), followed by Lasso-LR (AUC =0.959, 95% CI: 0.921–0.997), XGBoost (AUC =0.958, 95% CI: 0.902–1.000), LR (AUC =0.932, 95% CI: 0.878–0.987), and RF (AUC =0.868, 95% CI: 0.779–0.958) in the testing dataset.

Conclusions

Our ML-based models had good discrimination in predicting patients with obstructed hydronephrosis at high risk of harboring pyonephrosis, and the use of these models may be greatly beneficial to urologists in treatment planning, patient selection, and decision-making.

Application of clinical prediction modeling in pediatric neurosurgery: a case study

There has been an increasing interest in articles reporting on clinical prediction models in pediatric neurosurgery. Clinical prediction models are mathematical equations that combine patient-related risk factors for the estimation of an individual's risk of an outcome. If used sensibly, these evidence-based tools may help pediatric neurosurgeons in medical decision-making processes. Furthermore, they may help to communicate anticipated future events of diseases to children and their parents and facilitate shared decision-making accordingly. A basic understanding of this methodology is incumbent when developing or applying a prediction model. This paper addresses this methodology tailored to pediatric neurosurgery. For illustration, we use original pediatric data from our institution to illustrate this methodology with a case study. The developed model is however not externally validated, and clinical impact has not been assessed; therefore, the model cannot be recommended for clinical use in its current form.

Sample sizes of prediction model studies in prostate cancer were rarely justified and often insufficient

OBJECTIVE

Developing clinical prediction models (CPMs) on data of sufficient sample size is critical to help minimize overfitting. Using prostate cancer as a clinical exemplar, we aimed to investigate to what extent existing CPMs adhere to recent formal sample size criteria, or historic rules of thumb of events per predictor parameter (EPP)≥10.

STUDY DESIGN AND SETTING

A systematic review to identify CPMs related to prostate cancer, which provided enough information to calculate minimum sample size. We compared the reported sample size of each CPM against the traditional 10 EPP rule of thumb and formal sample size criteria.

RESULTS

About 211 CPMs were included. Three of the studies justified the sample size used, mostly using EPP rules of thumb. Overall, 69% of the CPMs were derived on sample sizes that surpassed the traditional EPP≥10 rule of thumb, but only 48% surpassed recent formal sample size criteria. For most CPMs, the required sample size based on formal criteria was higher than the sample sizes to surpass 10 EPP.

CONCLUSION

Few of the CPMs included in this study justified their sample size, with most justifications being based on EPP. This study shows that, in real-world data sets, adhering to the classic EPP rules of thumb is insufficient to adhere to recent formal sample size criteria.

Development and internal validation of prognostic models to predict negative health outcomes in older patients with multimorbidity and polypharmacy in general practice

BACKGROUND

Polypharmacy interventions are resource-intensive and should be targeted to those at risk of negative health outcomes. Our aim was to develop and internally validate prognostic models to predict health-related quality of life (HRQoL) and the combined outcome of falls, hospitalisation, institutionalisation and nursing care needs, in older patients with multimorbidity and polypharmacy in general practices.

METHODS

Design: two independent data sets, one comprising health insurance claims data (n=592 456), the other data from the PRIoritising MUltimedication in Multimorbidity (PRIMUM) cluster randomised controlled trial (n=502). Population: ≥60 years, ≥5 drugs, ≥3 chronic diseases, excluding dementia. Outcomes: combined outcome of falls, hospitalisation, institutionalisation and nursing care needs (after 6, 9 and 24 months) (claims data); and HRQoL (after 6 and 9 months) (trial data). Predictor variables in both data sets: age, sex, morbidity-related variables (disease count), medication-related variables (European Union-Potentially Inappropriate Medication list (EU-PIM list)) and health service utilisation. Predictor variables exclusively in trial data: additional socio-demographics, morbidity-related variables (Cumulative Illness Rating Scale, depression), Medication Appropriateness Index (MAI), lifestyle, functional status and HRQoL (EuroQol EQ-5D-3L). Analysis: mixed regression models, combined with stepwise variable selection, 10-fold cross validation and sensitivity analyses.

RESULTS

Most important predictors of EQ-5D-3L at 6 months in best model (Nagelkerke's R² 0.507) were depressive symptoms (-2.73 (95% CI: -3.56 to -1.91)), MAI (-0.39 (95% CI: -0.7 to -0.08)), baseline EQ-5D-3L (0.55 (95% CI: 0.47 to 0.64)). Models based on claims data and those predicting long-term outcomes based on both data sets produced low R² values. In claims data-based model with highest explanatory power (R²=0.16), previous falls/fall-related injuries, previous hospitalisations, age, number of involved physicians and disease count were most important predictor variables.

CONCLUSIONS

Best trial data-based model predicted HRQoL after 6 months well and included parameters of well-being not found in claims. Performance of claims data-based models and models predicting long-term outcomes was relatively weak. For generalisability, future studies should refit models by considering parameters representing well-being and functional status.

An advanced prediction model for postoperative complications and early implant failure

OBJECTIVES

Risk prediction in implant dentistry presents specific challenges including the dependence of observations from patients with multiple implants and rare outcome events. The aim of this study was to use advanced statistical methods based on penalized regression to assess risk factors in implant dentistry.

MATERIAL AND METHODS

We conducted a retrospective study from January 2016 to November 2018 recording postoperative complications including bleeding, hematoma, local infection, and nerve damage, as well as early implant failure. We further assessed patient- and implant-related risk factors including smoking and diabetes, as well as treatment parameters including types of gaps and surgical procedures. Univariable and multivariable generalized estimating equation (GEE) models were estimated to assess predictor effects, and a prediction model was fitted using L1 penalized estimation (lasso).

RESULTS

In a total of 1,132 patients (mean age: 50.6 ± 16.5 years, 55.4% female) and 2,413 implants, postoperative complications occurred in 71 patients. Sixteen implants were lost prior to loading. Multivariable GEE models showed a higher risk of any complication for diabetes mellitus (p = .006) and bone augmentation (p = .039). The models further revealed a higher risk of local infection for bone augmentation (p = .003), and a higher risk of hematoma formation for diabetes mellitus (p = .007) and edentulous jaws (p = .024). The lasso model did not select any risk factors into the prediction model.

CONCLUSIONS

Using novel methodology well-suited to tackle the specific challenges of risk prediction in implant dentistry, we were able to reliably estimate associations of risk factors with outcomes.