Machine learning algorithms' accuracy in predicting kidney disease progression: a systematic review and meta-analysis

External validation of a 2-year all-cause mortality prediction tool developed using machine learning in patients with stage 4-5 chronic kidney disease

BACKGROUND

Chronic kidney disease (CKD) is associated with increased mortality. Individual mortality prediction could be of interest to improve individual clinical outcomes. Using an independent regional dataset, the aim of the present study was to externally validate the recently published 2-year all-cause mortality prediction tool developed using machine learning.

METHODS

A validation dataset of stage 4 or 5 CKD outpatients was used. External validation performance of the prediction tool at the optimal cutoff-point was assessed by the area under the receiver operating characteristic curve (AUC-ROC), accuracy, sensitivity, and specificity. A survival analysis was then performed using the Kaplan-Meier method.

RESULTS

Data of 527 outpatients with stage 4 or 5 CKD were analyzed. During the 2 years of follow-up, 91 patients died and 436 survived. Compared to the learning dataset, patients in the validation dataset were significantly younger, and the ratio of deceased patients in the validation dataset was significantly lower. The performance of the prediction tool at the optimal cutoff-point was: AUC-ROC = 0.72, accuracy = 63.6%, sensitivity = 72.5%, and specificity = 61.7%. The survival curves of the predicted survived and the predicted deceased groups were significantly different (p < 0.001).

CONCLUSION

The 2-year all-cause mortality prediction tool for patients with stage 4 or 5 CKD showed satisfactory discriminatory capacity with emphasis on sensitivity. The proposed prediction tool appears to be of clinical interest for further development.

Derivation and Validation of a Machine Learning Model for the Prevention of Unplanned Dialysis

Key Points

Background

Approximately half of all patients with advanced CKD who progress to kidney failure initiate dialysis in an unplanned fashion, which is associated with high morbidity, mortality, and health care costs. A novel prediction model designed to identify patients with advanced CKD who are at high risk for developing kidney failure over short time frames (6–12 months) may help reduce the rates of unplanned dialysis and improve the quality of transitions from CKD to kidney failure.

Methods

We performed a retrospective study using machine learning random forest algorithms incorporating routinely collected age and sex data along with time-varying trends in laboratory measurements to derive and validate 6- and 12-month kidney failure risk prediction models in the population with advanced CKD. The models were comprehensively characterized in three independent cohorts in Ontario, Canada—derived in a cohort of 1849 consecutive patients with advanced CKD (mean [SD] age 66 [15] years, eGFR 19 [7] ml/min per 1.73 m2) and validated in two external advanced CKD cohorts (n=1356; age 69 [14] years, eGFR 22 [7] ml/min per 1.73 m2).

Results

Across all cohorts, 55% of patients experienced kidney failure, of whom 35% involved unplanned dialysis. The 6- and 12-month models demonstrated excellent discrimination with area under the receiver operating characteristic curve of 0.88 (95% confidence interval [CI], 0.87 to 0.89) and 0.87 (95% CI, 0.86 to 0.87) along with high probabilistic accuracy with the Brier scores of 0.10 (95% CI, 0.09 to 0.10) and 0.14 (95% CI, 0.13 to 0.14), respectively. The models were also well calibrated and delivered timely alerts on a significant number of patients who ultimately initiated dialysis in an unplanned fashion. Similar results were found upon external validation testing.

Conclusions

These machine learning models using routinely collected patient data accurately predict near-future kidney failure risk among the population with advanced CKD and retrospectively deliver advanced warning on a substantial proportion of unplanned dialysis events. Optimal implementation strategies still need to be elucidated.

ECG-based data-driven solutions for diagnosis and prognosis of cardiovascular diseases: A systematic review

Cardiovascular diseases (CVD) are a leading cause of death globally, and result in significant morbidity and reduced quality of life. The electrocardiogram (ECG) plays a crucial role in CVD diagnosis, prognosis, and prevention; however, different challenges still remain, such as an increasing unmet demand for skilled cardiologists capable of accurately interpreting ECG. This leads to higher workload and potential diagnostic inaccuracies. Data-driven approaches, such as machine learning (ML) and deep learning (DL) have emerged to improve existing computer-assisted solutions and enhance physicians' ECG interpretation of the complex mechanisms underlying CVD. However, many ML and DL models used to detect ECG-based CVD suffer from a lack of explainability, bias, as well as ethical, legal, and societal implications (ELSI). Despite the critical importance of these Trustworthy Artificial Intelligence (AI) aspects, there is a lack of comprehensive literature reviews that examine the current trends in ECG-based solutions for CVD diagnosis or prognosis that use ML and DL models and address the Trustworthy AI requirements. This review aims to bridge this knowledge gap by providing a systematic review to undertake a holistic analysis across multiple dimensions of these data-driven models such as type of CVD addressed, dataset characteristics, data input modalities, ML and DL algorithms (with a focus on DL), and aspects of Trustworthy AI like explainability, bias and ethical considerations. Additionally, within the analyzed dimensions, various challenges are identified. To these, we provide concrete recommendations, equipping other researchers with valuable insights to understand the current state of the field comprehensively.

Interpretable machine learning for predicting chronic kidney disease progression risk

Objective

Chronic kidney disease (CKD) poses a major global health burden. Early CKD risk prediction enables timely interventions, but conventional models have limited accuracy. Machine learning (ML) enhances prediction, but interpretability is needed to support clinical usage with both in diagnostic and decision-making.

Methods

A cohort of 491 patients with clinical data was collected for this study. The dataset was randomly split into an 80% training set and a 20% testing set. To achieve the first objective, we developed four ML algorithms (logistic regression, random forests, neural networks, and eXtreme Gradient Boosting (XGBoost)) to classify patients into two classes-those who progressed to CKD stages 3-5 during follow-up (positive class) and those who did not (negative class). For the classification task, the area under the receiver operating characteristic curve (AUC-ROC) was used to evaluate model performance in discriminating between the two classes. For survival analysis, Cox proportional hazards regression (COX) and random survival forests (RSFs) were employed to predict CKD progression, and the concordance index (C-index) and integrated Brier score were used for model evaluation. Furthermore, variable importance, partial dependence plots, and restrict cubic splines were used to interpret the models' results.

Results

XGBOOST demonstrated the best predictive performance for CKD progression in the classification task, with an AUC-ROC of 0.867 (95% confidence interval (CI): 0.728-0.100), outperforming the other ML algorithms. In survival analysis, RSF showed slightly better discrimination and calibration on the test set compared to COX, indicating better generalization to new data. Variable importance analysis identified estimated glomerular filtration rate, age, and creatinine as the most important predictors for CKD survival analysis. Further analysis revealed non-linear associations between age and CKD progression, suggesting higher risks in patients aged 52-55 and 65-66 years. The association between cholesterol levels and CKD progression was also non-linear, with lower risks observed when cholesterol levels were in the range of 5.8-6.4 mmol/L.

Conclusions

Our study demonstrated the effectiveness of interpretable ML models for predicting CKD progression. The comparison between COX and RSF highlighted the advantages of ML in survival analysis, particularly in handling non-linearity and high-dimensional data. By leveraging interpretable ML for unraveling risk factor relationships, contrasting predictive techniques, and exposing non-linear associations, this study significantly advances CKD risk prediction to enable enhanced clinical decision-making.

Information displays for automated surveillance algorithms of in-hospital patient deterioration: a scoping review

OBJECTIVE

Surveillance algorithms that predict patient decompensation are increasingly integrated with clinical workflows to help identify patients at risk of in-hospital deterioration. This scoping review aimed to identify the design features of the information displays, the types of algorithm that drive the display, and the effect of these displays on process and patient outcomes.

MATERIALS AND METHODS

The scoping review followed Arksey and O'Malley's framework. Five databases were searched with dates between January 1, 2009 and January 26, 2022. Inclusion criteria were: participants-clinicians in inpatient settings; concepts-intervention as deterioration information displays that leveraged automated AI algorithms; comparison as usual care or alternative displays; outcomes as clinical, workflow process, and usability outcomes; and context as simulated or real-world in-hospital settings in any country. Screening, full-text review, and data extraction were reviewed independently by 2 researchers in each step. Display categories were identified inductively through consensus.

RESULTS

Of 14 575 articles, 64 were included in the review, describing 61 unique displays. Forty-one displays were designed for specific deteriorations (eg, sepsis), 24 provided simple alerts (ie, text-based prompts without relevant patient data), 48 leveraged well-accepted score-based algorithms, and 47 included nurses as the target users. Only 1 out of the 10 randomized controlled trials reported a significant effect on the primary outcome.

CONCLUSIONS

Despite significant advancements in surveillance algorithms, most information displays continue to leverage well-understood, well-accepted score-based algorithms. Users' trust, algorithmic transparency, and workflow integration are significant hurdles to adopting new algorithms into effective decision support tools.

Development of a novel score (RENSAFE) to determine probability of acute kidney injury and renal functional decline post surgery: A multicenter analysis

OBJECTIVE

To create and validate 2 models called RENSAFE (RENalSAFEty) to predict postoperative acute kidney injury (AKI) and development of chronic kidney disease (CKD) stage 3b in patients undergoing partial (PN) or radical nephrectomy (RN) for kidney cancer.

METHODS

Primary objective was to develop a predictive model for AKI (reduction >25% of preoperative eGFR) and de novo CKD≥3b (<45 ml/min/1.73m2), through stepwise logistic regression. Secondary outcomes include elucidation of the relationship between AKI and de novo CKD≥3a (<60 ml/min/1.73m2). Accuracy was tested with receiver operator characteristic area under the curve (AUC).

RESULTS

AKI occurred in 452/1,517 patients (29.8%) and CKD≥3b in 116/903 patients (12.8%). Logistic regression demonstrated male sex (OR = 1.3, P = 0.02), ASA score (OR = 1.3, P < 0.01), hypertension (OR = 1.6, P < 0.001), R.E.N.A.L. score (OR = 1.2, P < 0.001), preoperative eGFR<60 (OR = 1.8, P = 0.009), and RN (OR = 10.4, P < 0.0001) as predictors for AKI. Age (OR 1.0, P < 0.001), diabetes mellitus (OR 2.5, P < 0.001), preoperative eGFR <60 (OR 3.6, P < 0.001) and RN (OR 2.2, P < 0.01) were predictors for CKD≥3b. AUC for RENSAFE AKI was 0.80 and 0.76 for CKD≥3b. AKI was predictive for CKD≥3a (OR = 2.2, P < 0.001), but not CKD≥3b (P = 0.1). Using 21% threshold probability for AKI achieved sensitivity: 80.3%, specificity: 61.7% and negative predictive value (NPV): 88.1%. Using 8% cutoff for CKD≥3b achieved sensitivity: 75%, specificity: 65.7%, and NPV: 96%.

CONCLUSION

RENSAFE models utilizing perioperative variables that can predict AKI and CKD may help guide shared decision making. Impact of postsurgical AKI was limited to less severe CKD (eGFR<60 ml/min 71.73m2). Confirmatory studies are requisite.

An integrated machine learning predictive scheme for longitudinal laboratory data to evaluate the factors determining renal function changes in patients with different chronic kidney disease stages

Background and objectives

Chronic kidney disease (CKD) is a global health concern. This study aims to identify key factors associated with renal function changes using the proposed machine learning and important variable selection (ML&IVS) scheme on longitudinal laboratory data. The goal is to predict changes in the estimated glomerular filtration rate (eGFR) in a cohort of patients with CKD stages 3-5.

Design

A retrospective cohort study.

Setting and participants

A total of 710 outpatients who presented with stable nondialysis-dependent CKD stages 3-5 at the Shin-Kong Wu Ho-Su Memorial Hospital Medical Center from 2016 to 2021.

Methods

This study analyzed trimonthly laboratory data including 47 indicators. The proposed scheme used stochastic gradient boosting, multivariate adaptive regression splines, random forest, eXtreme gradient boosting, and light gradient boosting machine algorithms to evaluate the important factors for predicting the results of the fourth eGFR examination, especially in patients with CKD stage 3 and those with CKD stages 4-5, with or without diabetes mellitus (DM).

Main outcome measurement

Subsequent eGFR level after three consecutive laboratory data assessments.

Results

Our ML&IVS scheme demonstrated superior predictive capabilities and identified significant factors contributing to renal function changes in various CKD groups. The latest levels of eGFR, blood urea nitrogen (BUN), proteinuria, sodium, and systolic blood pressure as well as mean levels of eGFR, BUN, proteinuria, and triglyceride were the top 10 significantly important factors for predicting the subsequent eGFR level in patients with CKD stages 3-5. In individuals with DM, the latest levels of BUN and proteinuria, mean levels of phosphate and proteinuria, and variations in diastolic blood pressure levels emerged as important factors for predicting the decline of renal function. In individuals without DM, all phosphate patterns and latest albumin levels were found to be key factors in the advanced CKD group. Moreover, proteinuria was identified as an important factor in the CKD stage 3 group without DM and CKD stages 4-5 group with DM.

Conclusion

The proposed scheme highlighted factors associated with renal function changes in different CKD conditions, offering valuable insights to physicians for raising awareness about renal function changes.

Benefits of Information Technology in Healthcare: Artificial Intelligence, Internet of Things, and Personal Health Records

OBJECTIVES

Systematic evaluations of the benefits of health information technology (HIT) play an essential role in enhancing healthcare quality by improving outcomes. However, there is limited empirical evidence regarding the benefits of IT adoption in healthcare settings. This study aimed to review the benefits of artificial intelligence (AI), the internet of things (IoT), and personal health records (PHR), based on scientific evidence.

METHODS

The literature published in peer-reviewed journals between 2016 and 2022 was searched for systematic reviews and meta-analysis studies using the PubMed, Cochrane, and Embase databases. Manual searches were also performed using the reference lists of systematic reviews and eligible studies from major health informatics journals. The benefits of each HIT were assessed from multiple perspectives across four outcome domains.

RESULTS

Twenty-four systematic review or meta-analysis studies on AI, IoT, and PHR were identified. The benefits of each HIT were assessed and summarized from a multifaceted perspective, focusing on four outcome domains: clinical, psycho-behavioral, managerial, and socioeconomic. The benefits varied depending on the nature of each type of HIT and the diseases to which they were applied.

CONCLUSIONS

Overall, our review indicates that AI and PHR can positively impact clinical outcomes, while IoT holds potential for improving managerial efficiency. Despite ongoing research into the benefits of health IT in line with advances in healthcare, the existing evidence is limited in both volume and scope. The findings of our study can help identify areas for further investigation.