Evaluating prediction model performance

The role of artificial intelligence in the management of liver diseases

Universal neonatal hepatitis B virus (HBV) vaccination and the advent of direct-acting antivirals (DAA) against hepatitis C virus (HCV) have reshaped the epidemiology of chronic liver diseases. However, some aspects of the management of chronic liver diseases remain unresolved. Nucleotide analogs can achieve sustained HBV DNA suppression but rarely lead to a functional cure. Despite the high efficacy of DAAs, successful antiviral therapy does not eliminate the risk of hepatocellular carcinoma (HCC), highlighted the need for cost-effective identification of high-risk populations for HCC surveillance and tailored HCC treatment strategies for these populations. The accessibility of high-throughput genomic data has accelerated the development of precision medicine, and the emergence of artificial intelligence (AI) has led to a new era of precision medicine. AI can learn from complex, non-linear data and identify hidden patterns within real-world datasets. The combination of AI and multi-omics approaches can facilitate disease diagnosis, biomarker discovery, and the prediction of treatment efficacy and prognosis. AI algorithms have been implemented in various aspects, including non-invasive tests, predictive models, image diagnosis, and the interpretation of histopathology findings. AI can support clinicians in decision-making, alleviate clinical burdens, and curtail healthcare expenses. In this review, we introduce the fundamental concepts of machine learning and review the role of AI in the management of chronic liver diseases.

Feasibility of Fall-Risk Detection in Older Adults: Real-World Use of Sensor Data With Machine Learning

PURPOSE

To use machine learning techniques with sensor data to predict fall risk in older adults aging in place.

METHOD

We tested the feasibility of using anomaly detection on a dataset comprising 315 days of continuous unobtrusive sensor data obtained from a single participant to predict fall risk within a 10-day window. Predictions were validated with performance metrics, including accuracy, F1 score, and receiver operating characteristic-area under curve (ROC-AUC), using actual falls documented in the electronic health record.

RESULTS

The model resulted with accuracy = 0.96 (95% confidence interval [CI] [0.94, 0.99]), F1 = 0.78 (95% CI [0.73, 0.83]), and ROC-AUC = 0.89 (95% CI [0.85, 0.93]).

CONCLUSION

The application of anomaly detection on sensor data may provide a timely and valid indication of fall risk in older adults within a 10-day window. Further research and validation are warranted to confirm these findings and expand the scope of application in the domain of older adult care and health care support. [Journal of Gerontological Nursing, 50(10), 7-10.].

An early sepsis prediction model utilizing machine learning and unbalanced data processing in a clinical context

Background

Early and accurate diagnoses of sepsis patients are essential to reduce the mortality. However, the sepsis is still diagnosed in a traditional way in China despite the increasing number of related studies, which may to some extent lead to delays in the treatment.

Methods

The study included 2,385 patients, including 364 with sepsis, collected from the First Affiliated Hospital of Anhui Medical University and partner hospitals from April to July 2022. External validation was conducted using the MIMIC-III database (over 60,000 patients from 2001 to 2012) and the eICU Collaborative Research Database (139,000 patients from 2014 to 2015). Multiple algorithm models, along with the SHapley Additive exPlanations (SHAP) analysis, are applied to explore the main risk factors for the accurate prediction of the sepsis. Multiple Imputations for filling missing data and the Synthetic Minority Oversampling (SMOTE) balancing method for balancing data are used for the data processing.

Result

Eighteen diagnostic features are used in the predictive model for early sepsis. The Random Forest model has the best performance among all the models, with an Area Under the Curve (AUC) of 87% and an F1-score (F1) of 77%. Moreover, the interpretation from the SHAP analysis is generally consistent with the current clinical situation.

Conclusion

The study revealed the relationship between these 18 clinical features and diagnostic outcomes. The results indicate that patients with laboratory values of Systolic Blood Pressure, Albumin, and Heart Rate exceeding certain thresholds are at a high likelihood of developing sepsis.

An artificial intelligence tool to assess the risk of severe mental distress among college students in terms of demographics, eating habits, lifestyles, and sport habits: an externally validated study using machine learning

BACKGROUND

Precisely estimating the probability of mental health challenges among college students is pivotal for facilitating timely intervention and preventative measures. However, to date, no specific artificial intelligence (AI) models have been reported to effectively forecast severe mental distress. This study aimed to develop and validate an advanced AI tool for predicting the likelihood of severe mental distress in college students.

METHODS

A total of 2088 college students from five universities were enrolled in this study. Participants were randomly divided into a training group (80%) and a validation group (20%). Various machine learning models, including logistic regression (LR), extreme gradient boosting machine (eXGBM), decision tree (DT), k-nearest neighbor (KNN), random forest (RF), and support vector machine (SVM), were employed and trained in this study. Model performance was evaluated using 11 metrics, and the highest scoring model was selected. In addition, external validation was conducted on 751 participants from three universities. The AI tool was then deployed as a web-based AI application.

RESULTS

Among the models developed, the eXGBM model achieved the highest area under the curve (AUC) value of 0.932 (95% CI: 0.911-0.949), closely followed by RF with an AUC of 0.927 (95% CI: 0.905-0.943). The eXGBM model demonstrated superior performance in accuracy (0.850), precision (0.824), recall (0.890), specificity (0.810), F1 score (0.856), Brier score (0.103), log loss (0.326), and discrimination slope (0.598). The eXGBM model also received the highest score of 60 based on the evaluation scoring system, while RF achieved a score of 49. The scores of LR, DT, and SVM were only 19, 32, and 36, respectively. External validation yielded an impressive AUC value of 0.918.

CONCLUSIONS

The AI tool demonstrates promising predictive performance for identifying college students at risk of severe mental distress. It has the potential to guide intervention strategies and support early identification and preventive measures.

High-performance grating-like SERS substrate based on machine learning for ultrasensitive detection of Zexie-Baizhu decoction

Rapid, universal and accurate identification of chemical composition changes in multi-component traditional Chinese medicine (TCM) decoction is a necessary condition for elucidating the effectiveness and mechanism of pharmacodynamic substances in TCM. In this paper, SERS technology, combined with grating-like SERS substrate and machine learning method, was used to establish an efficient and sensitive method for the detection of TCM decoction. Firstly, the grating-like substrate prepared by magnetron sputtering technology was served as a reliable SERS sensor for the identification of TCM decoction. The enhancement factor (EF) of 4-ATP probe molecules was as high as 1.90 × 107 and the limit of detection (LOD) was as low as 1 × 10-10 M. Then, SERS technology combined with support vector machine (SVM), decision tree (DT), Naive Bayes (NB) and other machine learning algorithms were used to classify and identify the three TCM decoctions, and the classification accuracy rate was as high as 97.78 %. In summary, it is expected that the proposed method combining SERS and machine learning method will have a high development in the practical application of multi-component analytes in TCM.

Development and validation of machine learning-augmented algorithm for insulin sensitivity assessment in the community and primary care settings: a population-based study in China

Objective

Insulin plays a central role in the regulation of energy and glucose homeostasis, and insulin resistance (IR) is widely considered as the "common soil" of a cluster of cardiometabolic disorders. Assessment of insulin sensitivity is very important in preventing and treating IR-related disease. This study aims to develop and validate machine learning (ML)-augmented algorithms for insulin sensitivity assessment in the community and primary care settings.

Methods

We analyzed the data of 9358 participants over 40 years old who participated in the population-based cohort of the Hubei center of the REACTION study (Risk Evaluation of Cancers in Chinese Diabetic Individuals). Three non-ensemble algorithms and four ensemble algorithms were used to develop the models with 70 non-laboratory variables for the community and 87 (70 non-laboratory and 17 laboratory) variables for the primary care settings to screen the classifier of the state-of-the-art. The models with the best performance were further streamlined using top-ranked 5, 8, 10, 13, 15, and 20 features. Performances of these ML models were evaluated using the area under the receiver operating characteristic curve (AUROC), the area under the precision-recall curve (AUPR), and the Brier score. The Shapley additive explanation (SHAP) analysis was employed to evaluate the importance of features and interpret the models.

Results

The LightGBM models developed for the community (AUROC 0.794, AUPR 0.575, Brier score 0.145) and primary care settings (AUROC 0.867, AUPR 0.705, Brier score 0.119) achieved higher performance than the models constructed by the other six algorithms. The streamlined LightGBM models for the community (AUROC 0.791, AUPR 0.563, Brier score 0.146) and primary care settings (AUROC 0.863, AUPR 0.692, Brier score 0.124) using the 20 top-ranked variables also showed excellent performance. SHAP analysis indicated that the top-ranked features included fasting plasma glucose (FPG), waist circumference (WC), body mass index (BMI), triglycerides (TG), gender, waist-to-height ratio (WHtR), the number of daughters born, resting pulse rate (RPR), etc.

Conclusion

The ML models using the LightGBM algorithm are efficient to predict insulin sensitivity in the community and primary care settings accurately and might potentially become an efficient and practical tool for insulin sensitivity assessment in these settings.