A novel approach to generate robust classification models to predict developmental toxicity from imbalanced datasets

Where developmental toxicity meets explainable artificial intelligence: state-of-the-art and perspectives

INTRODUCTION

The application of Artificial Intelligence (AI) to predictive toxicology is rapidly increasing, particularly aiming to develop non-testing methods that effectively address ethical concerns and reduce economic costs. In this context, Developmental Toxicity (Dev Tox) stands as a key human health endpoint, especially significant for safeguarding maternal and child well-being.

AREAS COVERED

This review outlines the existing methods employed in Dev Tox predictions and underscores the benefits of utilizing New Approach Methodologies (NAMs), specifically focusing on eXplainable Artificial Intelligence (XAI), which proves highly efficient in constructing reliable and transparent models aligned with recommendations from international regulatory bodies.

EXPERT OPINION

The limited availability of high-quality data and the absence of dependable Dev Tox methodologies render XAI an appealing avenue for systematically developing interpretable and transparent models, which hold immense potential for both scientific evaluations and regulatory decision-making.

A QSAR study for predicting malformation in zebrafish embryo

BACKGROUND

Developmental toxicity tests are extremely expensive, require a large number of animals, and are time-consuming. It is necessary to develop a new approach to simplify the analysis of developmental endpoints. One of these endpoints is malformation, and one group of ongoing methods for simplifying is in silico models. In this study, we aim to develop a quantitative structure-activity relationship (QSAR) model and identify the best algorithm for predicting malformations, as well as the most important and effective physicochemical properties associated with malformation.

METHODS

The dataset was extracted from a reliable database called COMPTOX. Physicochemical properties (descriptors) were calculated using Mordred and RDKit chemoinformatics software. The data were cleaned, preprocessed, and then split into training and testing sets. Machine learning algorithms, such as gradient boosting model (GBM) and logistic regression (LR), as well as deep learning models, including multilayer perceptron (MLP) and neural networks (NNs) trained with train set data and different sets of descriptors. The models were then validated with test set and various statistical parameters, such as Matthew's correlation coefficient (MCC) and balanced accuracy (BAC) score, were used to compare the models.

RESULTS

A set of descriptors containing with 78% AUC was identified as the best set of descriptors. Gradient boosting was determined to be the best algorithm with 78% predictive power.

CONCLUSIONS

The descriptors that were the most effective for developing models directly impact the mechanism of malformation, and GBM is the best model due to its MCC and BAC.

TISBE: A Public Web Platform for the Consensus-Based Explainable Prediction of Developmental Toxicity

Despite being extremely relevant for the protection of prenatal and neonatal health, the developmental toxicity (Dev Tox) is a highly complex endpoint whose molecular rationale is still largely unknown. The lack of availability of high-quality data as well as robust nontesting methods makes its understanding even more difficult. Thus, the application of new explainable alternative methods is of utmost importance, with Dev Tox being one of the most animal-intensive research themes of regulatory toxicology. Descending from TIRESIA (Toxicology Intelligence and Regulatory Evaluations for Scientific and Industry Applications), the present work describes TISBE (TIRESIA Improved on Structure-Based Explainability), a new public web platform implementing four fundamental advancements for in silico analyses: a three times larger dataset, a transparent XAI (explainable artificial intelligence) framework employing a fragment-based fingerprint coding, a novel consensus classifier based on five independent machine learning models, and a new applicability domain (AD) method based on a double top-down approach for better estimating the prediction reliability. The training set (TS) includes as many as 1008 chemicals annotated with experimental toxicity values. Based on a 5-fold cross-validation, a median value of 0.410 for the Matthews correlation coefficient was calculated; TISBE was very effective, with a median value of sensitivity and specificity equal to 0.984 and 0.274, respectively. TISBE was applied on two external pools made of 1484 bioactive compounds and 85 pediatric drugs taken from ChEMBL (Chemical European Molecular Biology Laboratory) and TEDDY (Task-Force in Europe for Drug Development in the Young) repositories, respectively. Notably, TISBE gives users the option to clearly spot the molecular fragments responsible for the toxicity or the safety of a given chemical query and is available for free at https://prometheus.farmacia.uniba.it/tisbe.

Making sense of chemical space network shows signs of criticality

Chemical space modelling has great importance in unveiling and visualising latent information, which is critical in predictive toxicology related to drug discovery process. While the use of traditional molecular descriptors and fingerprints may suffer from the so-called curse of dimensionality, complex networks are devoid of the typical drawbacks of coordinate-based representations. Herein, we use chemical space networks (CSNs) to analyse the case of the developmental toxicity (Dev Tox), which remains a challenging endpoint for the difficulty of gathering enough reliable data despite very important for the protection of the maternal and child health. Our study proved that the Dev Tox CSN has a complex non-random organisation and can thus provide a wealth of meaningful information also for predictive purposes. At a phase transition, chemical similarities highlight well-established toxicophores, such as aryl derivatives, mostly neurotoxic hydantoins, barbiturates and amino alcohols, steroids, and volatile organic compounds ether-like chemicals, which are strongly suspected of the Dev Tox onset and can thus be employed as effective alerts for prioritising chemicals before testing.

Review of machine learning and deep learning models for toxicity prediction

The ever-increasing number of chemicals has raised public concerns due to their adverse effects on human health and the environment. To protect public health and the environment, it is critical to assess the toxicity of these chemicals. Traditional in vitro and in vivo toxicity assays are complicated, costly, and time-consuming and may face ethical issues. These constraints raise the need for alternative methods for assessing the toxicity of chemicals. Recently, due to the advancement of machine learning algorithms and the increase in computational power, many toxicity prediction models have been developed using various machine learning and deep learning algorithms such as support vector machine, random forest, k-nearest neighbors, ensemble learning, and deep neural network. This review summarizes the machine learning- and deep learning-based toxicity prediction models developed in recent years. Support vector machine and random forest are the most popular machine learning algorithms, and hepatotoxicity, cardiotoxicity, and carcinogenicity are the frequently modeled toxicity endpoints in predictive toxicology. It is known that datasets impact model performance. The quality of datasets used in the development of toxicity prediction models using machine learning and deep learning is vital to the performance of the developed models. The different toxicity assignments for the same chemicals among different datasets of the same type of toxicity have been observed, indicating benchmarking datasets is needed for developing reliable toxicity prediction models using machine learning and deep learning algorithms. This review provides insights into current machine learning models in predictive toxicology, which are expected to promote the development and application of toxicity prediction models in the future.

TIRESIA: An eXplainable Artificial Intelligence Platform for Predicting Developmental Toxicity

Herein, a robust and reproducible eXplainable Artificial Intelligence (XAI) approach is presented, which allows prediction of developmental toxicity, a challenging human-health endpoint in toxicology. The application of XAI as an alternative method is of the utmost importance with developmental toxicity being one of the most animal-intensive areas of regulatory toxicology. In this work, the established CAESAR (Computer Assisted Evaluation of industrial chemical Substances According to Regulations) training set made of 234 chemicals for model learning is employed. Two test sets, including as a whole 585 chemicals, were instead used for validation and generalization purposes. The proposed framework favorably compares with the state-of-the-art approaches in terms of accuracy, sensitivity, and specificity, thus resulting in a reliable support system for developmental toxicity ensuring informativeness, uncertainty estimation, generalization, and transparency. Based on the eXtreme Gradient Boosting (XGB) algorithm, our predictive model provides easy interpretative keys based on specific molecular descriptors and structural alerts enabling one to distinguish toxic and nontoxic chemicals. Inspired by the Organisation for Economic Co-operation and Development (OECD) principles for the validation of Quantitative Structure-Activity Relationships (QSARs) for regulatory purposes, the results are summarized in a standard report in portable document format, enclosing also details concerned with a density-based model applicability domain and SHAP (SHapley Additive exPlanations) explainability, the latter particularly useful to better understand the effective roles played by molecular features. Notably, our model has been implemented in TIRESIA (Toxicology Intelligence and Regulatory Evaluations for Scientific and Industry Applications), a free of charge web platform available at http://tiresia.uniba.it.

Effect of Dataset Size and Train/Test Split Ratios in QSAR/QSPR Multiclass Classification

Applied datasets can vary from a few hundred to thousands of samples in typical quantitative structure-activity/property (QSAR/QSPR) relationships and classification. However, the size of the datasets and the train/test split ratios can greatly affect the outcome of the models, and thus the classification performance itself. We compared several combinations of dataset sizes and split ratios with five different machine learning algorithms to find the differences or similarities and to select the best parameter settings in nonbinary (multiclass) classification. It is also known that the models are ranked differently according to the performance merit(s) used. Here, 25 performance parameters were calculated for each model, then factorial ANOVA was applied to compare the results. The results clearly show the differences not just between the applied machine learning algorithms but also between the dataset sizes and to a lesser extent the train/test split ratios. The XGBoost algorithm could outperform the others, even in multiclass modeling. The performance parameters reacted differently to the change of the sample set size; some of them were much more sensitive to this factor than the others. Moreover, significant differences could be detected between train/test split ratios as well, exerting a great effect on the test validation of our models.

Predicting the reproductive toxicity of chemicals using ensemble learning methods and molecular fingerprints

Reproductive toxicity endpoints are a significant safety concern in the assessment of the adverse effects of chemicals in drug discovery. Computational models that can accurately predict a chemical's toxic potential are increasingly pursued to replace traditional animal experiments. Thus, ensemble learning models were built to predict the reproductive toxicity of compounds. Our ensemble models were developed using support vector machine, random forest, and extreme gradient boosting methods and 9 molecular fingerprints calculated for a dataset containing 1823 chemicals. The best prediction performance was achieved by the Ensemble-Top12 model, with an accuracy (ACC) of 86.33 %, a sensitivity (SEN) of 82.02 %, a specificity (SPE) of 90.19 %, and an area under the receiver operating characteristic curve (AUC) of 0.937 in 5-fold cross-validation and ACC, SEN, SPE, and AUC values of 84.38 %, 86.90 %, 90.67 %, and 0.920, respectively, in external validation. We also defined the applicability domain (AD) of the ensemble model by calculating the Tanimoto distance of the training set. Compared with models in existing literature, our ensemble model achieves relatively high ACC, SPE and AUC values. We also identified several fingerprint features related to chemical reproductive toxicity. Considering the performance of model, we recommend using the Ensemble-Top12 model to predict reproductive toxicity in early drug development.

In silico prediction of drug-induced developmental toxicity by using machine learning approaches

Some drugs and xenobiotics have the potential to disturb homeostasis, normal growth, differentiation, development or behavior during prenatal development or postnatally until puberty. Assessment of the developmental toxicity is one of the important safety considerations incorporated by international regulatory agencies. In this investigation, seven machine learning methods, including naïve Bayes, support vector machine, recursive partitioning, k-nearest neighbor, C4.5 decision tree, random forest and Adaboost, were used to build binary classification models for developmental toxicity. Among these models, the naïve Bayes classifier represented the best predictive performance and stability, which gave 91.11% overall prediction accuracy, 91.50% balanced accuracy and 0.818 MCC for the training set, and generated 83.93% concordance, 81.85% balanced accuracy and 0.627 MCC for the test set. The application domains were analyzed, and only one chemical in the test set was identified as outside the application domain. In addition, 10 important molecular descriptors related to developmental toxicity were selected by the genetic algorithm, which may contribute to explanation of the mechanisms of developmental toxicants. The best naïve Bayes classification model should be employed as alternative method for qualitative prediction of chemical-induced developmental toxicity in early stages of drug development.