Explaining multivariate molecular diagnostic tests via Shapley values

Prediction of Mycobacterium tuberculosis cell wall permeability using machine learning methods

Tuberculosis (TB) caused by the bacteria Mycobacterium tuberculosis (M. tb), continues to pose a significant worldwide health threat. The advent of drug-resistant strains of the disease highlights the critical need for novel treatments. The unique cell wall of M. tb provides an extra layer of protection for the bacteria and hence only compounds that can penetrate this barrier can reach their targets within the bacterial cell wall. The creation of a reliable machine learning (ML) model to predict the mycobacterial cell wall permeability of small molecules is presented in this work and four ML algorithms, including Random Forest, Support Vector Machines (SVM), k-nearest Neighbour (k-NN) and Logistic Regression were trained on a dataset of 5368 compounds. RDKit and Mordred toolkits were used to calculate features. To determine the most effective model, various performance metrics were used such as accuracy, precision, recall, F1 score, and area under the receiver operating characteristic curve. The best-performing model was further refined with hyperparameter tuning and tenfold cross-validation. The SVM model with filtering outperformed the other machine learning models and demonstrated 80.26% and 81.13% accuracy on the test and validation datasets, respectively. The study also provided insights into the molecular descriptors that play the most important role in predicting the ability of a molecule to pass the M. tb cell wall, which could guide future compound design. The model is available at https://github.com/PGlab-NIPER/MTB_Permeability .

A machine learning-derived risk score to predict left ventricular diastolic dysfunction from clinical cardiovascular magnetic resonance imaging

Introduction

The evaluation of left ventricular diastolic dysfunction (LVDD) by clinical cardiac magnetic resonance (CMR) remains a challenge. We aimed to train and evaluate a machine-learning (ML) algorithm for the assessment of LVDD by clinical CMR variables and to investigate its prognostic value for predicting hospitalized heart failure and all-cause mortality.

Methods

LVDD was characterized by echocardiography following the ASE guidelines. Eight demographic and nineteen common clinical CMR variables including delayed enhancement were used to train Random Forest models with a Bayesian optimizer. The model was evaluated using bootstrap and five-fold cross-validation. Area under the ROC curve (AUC) was utilized to evaluate the model performance. An ML risk score was used to stratify the risk of heart failure hospitalization and all-cause mortality.

Results

A total of 606 consecutive patients underwent CMR and echocardiography within 7 days for cardiovascular disease evaluation. LVDD was present in 303 subjects by echocardiography. The performance of the ML algorithm was good using the CMR variables alone with an AUC of 0.868 (95% CI: 0.811-0.917), which was improved by combining with demographic data yielding an AUC 0.895 (95% CI: 0.845-0.939). The algorithm performed well in an independent validation cohort with AUC 0.810 (0.731-0.874). Subjects with higher ML scores (>0.4121) were associated with increased adjusted hazard ratio for a composite outcome than subjects with lower ML scores (1.72, 95% confidence interval 1.09-2.71).

Discussion

An ML algorithm using variables derived from clinical CMR is effective in identifying patients with LVDD and providing prognostication for adverse clinical outcomes.

Automated Machine Learning and Explainable AI (AutoML-XAI) for Metabolomics: Improving Cancer Diagnostics

Metabolomics generates complex data necessitating advanced computational methods for generating biological insight. While machine learning (ML) is promising, the challenges of selecting the best algorithms and tuning hyperparameters, particularly for nonexperts, remain. Automated machine learning (AutoML) can streamline this process; however, the issue of interpretability could persist. This research introduces a unified pipeline that combines AutoML with explainable AI (XAI) techniques to optimize metabolomics analysis. We tested our approach on two data sets: renal cell carcinoma (RCC) urine metabolomics and ovarian cancer (OC) serum metabolomics. AutoML, using Auto-sklearn, surpassed standalone ML algorithms like SVM and k-Nearest Neighbors in differentiating between RCC and healthy controls, as well as OC patients and those with other gynecological cancers. The effectiveness of Auto-sklearn is highlighted by its AUC scores of 0.97 for RCC and 0.85 for OC, obtained from the unseen test sets. Importantly, on most of the metrics considered, Auto-sklearn demonstrated a better classification performance, leveraging a mix of algorithms and ensemble techniques. Shapley Additive Explanations (SHAP) provided a global ranking of feature importance, identifying dibutylamine and ganglioside GM(d34:1) as the top discriminative metabolites for RCC and OC, respectively. Waterfall plots offered local explanations by illustrating the influence of each metabolite on individual predictions. Dependence plots spotlighted metabolite interactions, such as the connection between hippuric acid and one of its derivatives in RCC, and between GM3(d34:1) and GM3(18:1_16:0) in OC, hinting at potential mechanistic relationships. Through decision plots, a detailed error analysis was conducted, contrasting feature importance for correctly versus incorrectly classified samples. In essence, our pipeline emphasizes the importance of harmonizing AutoML and XAI, facilitating both simplified ML application and improved interpretability in metabolomics data science.

Impossibility theorems for feature attribution

Despite a sea of interpretability methods that can produce plausible explanations, the field has also empirically seen many failure cases of such methods. In light of these results, it remains unclear for practitioners how to use these methods and choose between them in a principled way. In this paper, we show that for moderately rich model classes (easily satisfied by neural networks), any feature attribution method that is complete and linear-for example, Integrated Gradients and Shapley Additive Explanations (SHAP)-can provably fail to improve on random guessing for inferring model behavior. Our results apply to common end-tasks such as characterizing local model behavior, identifying spurious features, and algorithmic recourse. One takeaway from our work is the importance of concretely defining end-tasks: Once such an end-task is defined, a simple and direct approach of repeated model evaluations can outperform many other complex feature attribution methods.

Automated machine learning and explainable AI (AutoML-XAI) for metabolomics: improving cancer diagnostics

Motivation

Metabolomics generates complex data necessitating advanced computational methods for generating biological insight. While machine learning (ML) is promising, the challenges of selecting the best algorithms and tuning hyperparameters, particularly for non-experts, remain. Automated machine learning (AutoML) can streamline this process; however, the issue of interpretability could persist. This research introduces a unified pipeline that combines AutoML with explainable AI (XAI) techniques to optimize metabolomics analysis.

Results

We tested our approach on two datasets: renal cell carcinoma (RCC) urine metabolomics and ovarian cancer (OC) serum metabolomics. AutoML, using auto-sklearn, surpassed standalone ML algorithms such as SVM and random forest in differentiating between RCC and healthy controls, as well as OC patients and those with other gynecological cancers (Non-OC). Auto-sklearn employed a mix of algorithms and ensemble techniques, yielding a superior performance (AUC of 0.97 for RCC and 0.85 for OC). Shapley Additive Explanations (SHAP) provided a global ranking of feature importance, identifying dibutylamine and ganglioside GM(d34:1) as the top discriminative metabolites for RCC and OC, respectively. Waterfall plots offered local explanations by illustrating the influence of each metabolite on individual predictions. Dependence plots spotlighted metabolite interactions, such as the connection between hippuric acid and one of its derivatives in RCC, and between GM3(d34:1) and GM3(18:1_16:0) in OC, hinting at potential mechanistic relationships. Through decision plots, a detailed error analysis was conducted, contrasting feature importance for correctly versus incorrectly classified samples. In essence, our pipeline emphasizes the importance of harmonizing AutoML and XAI, facilitating both simplified ML application and improved interpretability in metabolomics data science.

Availability

https://github.com/obifarin/automl-xai-metabolomics.

Machine learning applications to improve flavor and nutritional content of horticultural crops through breeding and genetics

Over the last decades, significant strides were made in understanding the biochemical factors influencing the nutritional content and flavor profile of fruits and vegetables. Product differentiation in the produce aisle is the natural consequence of increasing consumer power in the food industry. Cotton-candy grapes, specialty tomatoes, and pineapple-flavored white strawberries provide a few examples. Given the increased demand for flavorful varieties, and pressing need to reduce micronutrient malnutrition, we expect breeding to increase its prioritization toward these traits. Reaching this goal will, in part, necessitate knowledge of the genetic architecture controlling these traits, as well as the development of breeding methods that maximize their genetic gain. Can artificial intelligence (AI) help predict flavor preferences, and can such insights be leveraged by breeding programs? In this Perspective, we outline both the opportunities and challenges for the development of more flavorful and nutritious crops, and how AI can support these breeding initiatives.

WindowSHAP: An efficient framework for explaining time-series classifiers based on Shapley values

Unpacking and comprehending how black-box machine learning algorithms (such as deep learning models) make decisions has been a persistent challenge for researchers and end-users. Explaining time-series predictive models is useful for clinical applications with high stakes to understand the behavior of prediction models, e.g., to determine how different variables and time points influence the clinical outcome. However, existing approaches to explain such models are frequently unique to architectures and data where the features do not have a time-varying component. In this paper, we introduce WindowSHAP, a model-agnostic framework for explaining time-series classifiers using Shapley values. We intend for WindowSHAP to mitigate the computational complexity of calculating Shapley values for long time-series data as well as improve the quality of explanations. WindowSHAP is based on partitioning a sequence into time windows. Under this framework, we present three distinct algorithms of Stationary, Sliding and Dynamic WindowSHAP, each evaluated against baseline approaches, KernelSHAP and TimeSHAP, using perturbation and sequence analyses metrics. We applied our framework to clinical time-series data from both a specialized clinical domain (Traumatic Brain Injury - TBI) as well as a broad clinical domain (critical care medicine). The experimental results demonstrate that, based on the two quantitative metrics, our framework is superior at explaining clinical time-series classifiers, while also reducing the complexity of computations. We show that for time-series data with 120 time steps (hours), merging 10 adjacent time points can reduce the CPU time of WindowSHAP by 80 % compared to KernelSHAP. We also show that our Dynamic WindowSHAP algorithm focuses more on the most important time steps and provides more understandable explanations. As a result, WindowSHAP not only accelerates the calculation of Shapley values for time-series data, but also delivers more understandable explanations with higher quality.

Enhancing Classification of liquid chromatography mass spectrometry data with Batch Effect Removal Neural Networks (BERNN)

Liquid Chromatography Mass Spectrometry (LC-MS) is a powerful method for profiling complex biological samples. However, batch effects typically arise from differences in sample processing protocols, experimental conditions and data acquisition techniques, significantlyimpacting the interpretability of results. Correcting batch effects is crucial for the reproducibility of proteomics research, but current methods are not optimal for removal of batch effects without compressing the genuine biological variation under study. We propose a suite of Batch Effect Removal Neural Networks (BERNN) to remove batch effects in large LC-MS experiments, with the goal of maximizing sample classification performance between conditions. More importantly, these models must efficiently generalize in batches not seen during training. Comparison of batch effect correction methods across three diverse datasets demonstrated that BERNN models consistently showed the strongest sample classification performance. However, the model producing the greatest classification improvements did not always perform best in terms of batch effect removal. Finally, we show that overcorrection of batch effects resulted in the loss of some essential biological variability. These findings highlight the importance of balancing batch effect removal while preserving valuable biological diversity in large-scale LC-MS experiments.

Development and Validation of an Insulin Resistance Model for a Population with Chronic Kidney Disease Using a Machine Learning Approach

Background: Chronic kidney disease (CKD) is a complex syndrome without a definitive treatment. For these patients, insulin resistance (IR) is associated with worse renal and patient outcomes. Until now, no predictive model using machine learning (ML) has been reported on IR in CKD patients. Methods: The CKD population studied was based on results from the National Health and Nutrition Examination Survey (NHANES) of the USA from 1999 to 2012. The homeostasis model assessment of IR (HOMA-IR) was used to assess insulin resistance. We began the model building process via the ML algorithm (random forest (RF), eXtreme Gradient Boosting (XGboost), logistic regression algorithms, and deep neural learning (DNN)). We compared different receiver operating characteristic (ROC) curves from different algorithms. Finally, we used SHAP values (SHapley Additive exPlanations) to explain how the different ML models worked. Results: In this study population, 71,916 participants were enrolled. Finally, we analyzed 1,229 of these participants. Their data were segregated into the IR group (HOMA IR > 3, n = 572) or non-IR group (HOMR IR ≤ 3, n = 657). In the validation group, RF had a higher accuracy (0.77), specificity (0.81), PPV (0.77), and NPV (0.77). In the test group, XGboost had a higher AUC of ROC (0.78). In addition, XGBoost also had a higher accuracy (0.7) and NPV (0.71). RF had a higher accuracy (0.7), specificity (0.78), and PPV (0.7). In the RF algorithm, the body mass index had a much larger impact on IR (0.1654), followed by triglyceride (0.0117), the daily calorie intake (0.0602), blood HDL value (0.0587), and age (0.0446). As for the SHAP value, in the RF algorithm, almost all features were well separated to show a positive or negative association with IR. Conclusion: This was the first study using ML to predict IR in patients with CKD. Our results showed that the RF algorithm had the best AUC of ROC and the best SHAP value differentiation. This was also the first study that included both macronutrients and micronutrients. We concluded that ML algorithms, particularly RF, can help determine risk factors and predict IR in patients with CKD.

Ageing and cancer: a research gap to fill

The complex mechanisms of ageing biology are increasingly understood. Interventions to reduce or delay ageing‐associated diseases are emerging. Cancer is one of the diseases promoted by tissue ageing. A clockwise mutational signature is identified in many tumours. Ageing might be a modifiable cancer risk factor. To reduce the incidence of ageing‐related cancer and to detect the disease at earlier stages, we need to understand better the links between ageing and tumours. When a cancer is established, geriatric assessment and measures of biological age might help to generate evidence‐based therapeutic recommendations. In this approach, patients and caregivers would include the respective weight to give to the quality of life and survival in the therapeutic choices. The increasing burden of cancer in older patients requires new generations of researchers and geriatric oncologists to be trained, to properly address disease complexity in a multidisciplinary manner, and to reduce health inequities in this population of patients. In this review, we propose a series of research challenges to tackle in the next few years to better prevent, detect and treat cancer in older patients while preserving their quality of life.

Real-world performance of blood-based proteomic profiling in first-line immunotherapy treatment in advanced stage non-small cell lung cancer

Purpose

Immune checkpoint inhibition (ICI) therapy has improved patient outcomes in advanced non-small cell lung cancer (NSCLC), but better biomarkers are needed. A clinically validated, blood-based proteomic test, or host immune classifier (HIC), was assessed for its ability to predict ICI therapy outcomes in this real-world, prospectively designed, observational study.

Materials and methods

The prospectively designed, observational registry study INSIGHT (Clinical Effectiveness Assessment of VeriStrat® Testing and Validation of Immunotherapy Tests in NSCLC Subjects) (NCT03289780) includes 35 US sites having enrolled over 3570 NSCLC patients at any stage and line of therapy. After enrolment and prior to therapy initiation, all patients are tested and designated HIC-Hot (HIC-H) or HIC-Cold (HIC-C). A prespecified interim analysis was performed after 1-year follow-up with the first 2000 enrolled patients. We report the overall survival (OS) of patients with advanced stage (IIIB and IV) NSCLC treated in the first-line (ICI-containing therapies n=284; all first-line therapies n=877), by treatment type and in HIC-defined subgroups.

Results

OS for HIC-H patients was longer than OS for HIC-C patients across treatment regimens, including ICI. For patients treated with all ICI regimens, median OS was not reached (95% CI 15.4 to undefined months) for HIC-H (n=196) vs 5.0 months (95% CI 2.9 to 6.4) for HIC-C patients (n=88); HR=0.38 (95% CI 0.27 to 0.53), p<0.0001. For ICI monotherapy, OS was 16.8 vs 2.8 months (HR=0.36 (95% CI 0.22 to 0.58), p<0.0001) and for ICI with chemotherapy OS was unreached vs 6.4 months (HR=0.41 (95% CI 0.26 to 0.67), p=0.0003). HIC results were independent of programmed death ligand 1 (PD-L1). In a subgroup with PD-L1 ≥50% and performance status 0–1, HIC stratified survival significantly for ICI monotherapy but not ICI with chemotherapy.

Conclusion

Blood-based HIC proteomic testing provides clinically meaningful information for immunotherapy treatment decision in NSCLC independent of PD-L1. The data suggest that HIC-C patients should not be treated with ICI alone regardless of their PD-L1 expression.