Interpreting Neural Network Models for Toxicity Prediction by Extracting Learned Chemical Features

Neural network models have become a popular machine-learning technique for the toxicity prediction of chemicals. However, due to their complex structure, it is difficult to understand predictions made by these models which limits confidence. Current techniques to tackle this problem such as SHAP or integrated gradients provide insights by attributing importance to the input features of individual compounds. While these methods have produced promising results in some cases, they do not shed light on how representations of compounds are transformed in hidden layers, which constitute how neural networks learn. We present a novel technique to interpret neural networks which identifies chemical substructures in training data found to be responsible for the activation of hidden neurons. For individual test compounds, the importance of hidden neurons is determined, and the associated substructures are leveraged to explain the model prediction. Using structural alerts for mutagenicity from the Derek Nexus expert system as ground truth, we demonstrate the validity of the approach and show that model explanations are competitive with and complementary to explanations obtained from an established feature attribution method.

Prediction of molecular-specific mutagenic alerts and related mechanisms of chemicals by a convolutional neural network (CNN) model based on SMILES split

Structural alerts (SAs) are essential to identify chemicals for toxicity evaluation and health risk assessment. We constructed a novel SMILES split-based deep learning model (SSDL) that was trained and verified with 5850 chemicals from the ISSSTY database and 384 external test chemicals from published papers. The training accuracy was above 0.90 and the evaluation metrics (precision, recall and F1-score) all reached 0.78 or above on both internal and external test chemicals. In this model, the molecular-specific fragment importance of chemicals was first quantified independently. Then, the SA identification method based on the importance of these fragments was statistically analyzed and verified with the ISSSTY test and external test chemicals containing one of 28 typical SAs, and most of the performances were better than that of expert rules. Furthermore, a mutagenicity mechanism prediction method was developed using 237 chemicals with four known mutagenic mechanisms based on molecular similarity calibrated by the SSDL method and fragment importance, which significantly improved accuracy in three mechanisms and had comparable accuracy in the other one compared to traditional methods. Overall, the SSDL model quantifying fragment toxicity within molecules would be a novel potentially powerful tool in the determination and visualization of molecular-specific SAs and the prediction of mutagenicity mechanisms for environmental or industrial compounds and drugs.

Evaluation of QSAR models for predicting mutagenicity: outcome of the Second Ames/QSAR international challenge project

Quantitative structure-activity relationship (QSAR) models are powerful in silico tools for predicting the mutagenicity of unstable compounds, impurities and metabolites that are difficult to examine using the Ames test. Ideally, Ames/QSAR models for regulatory use should demonstrate high sensitivity, low false-negative rate and wide coverage of chemical space. To promote superior model development, the Division of Genetics and Mutagenesis, National Institute of Health Sciences, Japan (DGM/NIHS), conducted the Second Ames/QSAR International Challenge Project (2020-2022) as a successor to the First Project (2014-2017), with 21 teams from 11 countries participating. The DGM/NIHS provided a curated training dataset of approximately 12,000 chemicals and a trial dataset of approximately 1,600 chemicals, and each participating team predicted the Ames mutagenicity of each trial chemical using various Ames/QSAR models. The DGM/NIHS then provided the Ames test results for trial chemicals to assist in model improvement. Although overall model performance on the Second Project was not superior to that on the First, models from the eight teams participating in both projects achieved higher sensitivity than models from teams participating in only the Second Project. Thus, these evaluations have facilitated the development of QSAR models.

Multitask Deep Neural Networks for Ames Mutagenicity Prediction

The Ames mutagenicity test constitutes the most frequently used assay to estimate the mutagenic potential of drug candidates. While this test employs experimental results using various strains of Salmonella typhimurium, the vast majority of the published in silico models for predicting mutagenicity do not take into account the test results of the individual experiments conducted for each strain. Instead, such QSAR models are generally trained employing overall labels (i.e., mutagenic and nonmutagenic). Recently, neural-based models combined with multitask learning strategies have yielded interesting results in different domains, given their capabilities to model multitarget functions. In this scenario, we propose a novel neural-based QSAR model to predict mutagenicity that leverages experimental results from different strains involved in the Ames test by means of a multitask learning approach. To the best of our knowledge, the modeling strategy hereby proposed has not been applied to model Ames mutagenicity previously. The results yielded by our model surpass those obtained by single-task modeling strategies, such as models that predict the overall Ames label or ensemble models built from individual strains. For reproducibility and accessibility purposes, all source code and datasets used in our experiments are publicly available.

Analysis of the benefits of imputation models over traditional QSAR models for toxicity prediction

Recently, imputation techniques have been adapted to predict activity values among sparse bioactivity matrices, showing improvements in predictive performance over traditional QSAR models. These models are able to use experimental activity values for auxiliary assays when predicting the activity of a test compound on a specific assay. In this study, we tested three different multi-task imputation techniques on three classification-based toxicity datasets: two of small scale (12 assays each) and one large scale with 417 assays. Moreover, we analyzed in detail the improvements shown by the imputation models. We found that test compounds that were dissimilar to training compounds, as well as test compounds with a large number of experimental values for other assays, showed the largest improvements. We also investigated the impact of sparsity on the improvements seen as well as the relatedness of the assays being considered. Our results show that even a small amount of additional information can provide imputation methods with a strong boost in predictive performance over traditional single task and multi-task predictive models.

Prioritization of Mycotoxins Based on Their Genotoxic Potential with an In Silico-In Vitro Strategy

Humans are widely exposed to a great variety of mycotoxins and their mixtures. Therefore, it is important to design strategies that allow prioritizing mycotoxins based on their toxic potential in a time and cost-effective manner. A strategy combining in silico tools (Phase 1), including an expert knowledge-based (DEREK Nexus®, Lhasa Limited, Leeds, UK) and a statistical-based platform (VEGA QSAR©, Mario Negri Institute, Milan, Italy), followed by the in vitro SOS/umu test (Phase 2), was applied to a set of 12 mycotoxins clustered according to their structure into three groups. Phase 1 allowed us to clearly classify group 1 (aflatoxin and sterigmatocystin) as mutagenic and group 3 (ochratoxin A, zearalenone and fumonisin B1) as non-mutagenic. For group 2 (trichothecenes), contradictory conclusions were obtained between the two in silico tools, being out of the applicability domain of many models. Phase 2 confirmed the results obtained in the previous phase for groups 1 and 3. It also provided extra information regarding the role of metabolic activation in aflatoxin B1 and sterigmatocystin mutagenicity. Regarding group 2, equivocal results were obtained in few experiments; however, the group was finally classified as non-mutagenic. The strategy used correlated with the published Ames tests, which detect point mutations. Few alerts for chromosome aberrations could be detected. The SOS/umu test appeared as a good screening test for mutagenicity that can be used in the absence and presence of metabolic activation and independently of Phase 1, although the in silico-in vitro combination gave more information for decision making.

Toward regulatory acceptance and improving the prediction confidence of in silico approaches: a case study of genotoxicity

Introduction: Genotoxicity is an imperative component of the human health safety assessment of chemicals. Its secure forecast is of the utmost importance for all health prevention strategies and regulations.Areas covered: We surveyed several types of alternative, animal-free approaches ((quantitative) structure-activity relationship (Q)SAR, read-across, Adverse Outcome Pathway, Integrated Approaches to Testing and Assessment) for genotoxicity prediction within the needs of regulatory frameworks, putting special emphasis on data quality and uncertainties issues.Expert opinion: (Q)SAR models and read-across approaches for in vitro bacterial mutagenicity have sufficient reliability for use in prioritization processes, and as support in regulatory decisions in combination with other types of evidence. (Q)SARs and read-across methodologies for other genotoxicity endpoints need further improvements and should be applied with caution. It appears that there is still large room for improvement of genotoxicity prediction methods. Availability of well-curated high-quality databases, covering a broader chemical space, is one of the most important needs. Integration of in silico predictions with expert knowledge, weight-of-evidence-based assessment, and mechanistic understanding of genotoxicity pathways are other key points to be addressed for the generation of more accurate and trustable results.

PySmash: Python package and individual executable program for representative substructure generation and application

BACKGROUND

Substructure screening is widely applied to evaluate the molecular potency and ADMET properties of compounds in drug discovery pipelines, and it can also be used to interpret QSAR models for the design of new compounds with desirable physicochemical and biological properties. With the continuous accumulation of more experimental data, data-driven computational systems which can derive representative substructures from large chemical libraries attract more attention. Therefore, the development of an integrated and convenient tool to generate and implement representative substructures is urgently needed.

RESULTS

In this study, PySmash, a user-friendly and powerful tool to generate different types of representative substructures, was developed. The current version of PySmash provides both a Python package and an individual executable program, which achieves ease of operation and pipeline integration. Three types of substructure generation algorithms, including circular, path-based and functional group-based algorithms, are provided. Users can conveniently customize their own requirements for substructure size, accuracy and coverage, statistical significance and parallel computation during execution. Besides, PySmash provides the function for external data screening.

CONCLUSION

PySmash, a user-friendly and integrated tool for the automatic generation and implementation of representative substructures, is presented. Three screening examples, including toxicophore derivation, privileged motif detection and the integration of substructures with machine learning (ML) models, are provided to illustrate the utility of PySmash in safety profile evaluation, therapeutic activity exploration and molecular optimization, respectively. Its executable program and Python package are available at https://github.com/kotori-y/pySmash.

Towards quantitative read across: Prediction of Ames mutagenicity in a large database

In silico chemical safety assessment can support the evaluation of hazard and risk following potential exposure to a substance, thus stimulating an increased interest for the use of Structure-Activity based approaches by regulatory authorities, particularly QSAR and Read Across. Whereas the longer history of QSAR led to recognize the crucial requirements for predictivity, there are still challenges faced by adopting Read Across to a larger extent in a regulatory setting, namely standardization and objective criteria. In previous research, suitable conditions for applying Read Across to the prediction of the Ames mutagenicity of metabolites and degradation products of pesticides were established: a standardized similarity criterion based simultaneously on basic molecular properties and Structural Similarity was successfully applied to a number of case studies. Here the investigation is extended to a large database of curated Ames mutagenicity results. For around 2,000 chemicals for which the similarity criterion was applicable, the predictivity of Read Across was high: specificity 0.72, sensitivity 0.90, accuracy 0.85. This compares favourably with the Ames test intra-assay variability, and with the predictivity of QSAR models. The need for standardization and rigorous validation of Read Across is emphasized.

In Silico Toxicology Data Resources to Support Read-Across and (Q)SAR

A plethora of databases exist online that can assist in in silico chemical or drug safety assessment. However, a systematic review and grouping of databases, based on purpose and information content, consolidated in a single source, has been lacking. To resolve this issue, this review provides a comprehensive listing of the key in silico data resources relevant to: chemical identity and properties, drug action, toxicology (including nano-material toxicity), exposure, omics, pathways, Absorption, Distribution, Metabolism and Elimination (ADME) properties, clinical trials, pharmacovigilance, patents-related databases, biological (genes, enzymes, proteins, other macromolecules etc.) databases, protein-protein interactions (PPIs), environmental exposure related, and finally databases relating to animal alternatives in support of 3Rs policies. More than nine hundred databases were identified and reviewed against criteria relating to accessibility, data coverage, interoperability or application programming interface (API), appropriate identifiers, types of in vitro, in vivo,-clinical or other data recorded and suitability for modelling, read-across, or similarity searching. This review also specifically addresses the need for solutions for mapping and integration of databases into a common platform for better translatability of preclinical data to clinical data.

(Q)SAR Methods for Predicting Genotoxicity and Carcinogenicity: Scientific Rationale and Regulatory Frameworks

Knowledge of the genotoxicity and carcinogenicity potential of chemical substances is one of the key scientific elements able to better protect human health. Genotoxicity assessment is also considered as prescreening of carcinogenicity. The assessment of both endpoints is a fundamental component of national and international legislations, for all types of substances, and has stimulated the development of alternative, nontesting methods. Over the recent decades, much attention has been given to the use and further development of structure-activity relationships-based approaches, to be used in isolation or in combination with in vitro assays for predictive purposes. In this chapter, we briefly introduce the rationale for the main (Q)SAR approaches, and detail the most important regulatory initiatives and frameworks. It appears that the existence and needs of regulatory frameworks stimulate the development of better predictive tools; in turn, this allows the regulators to fine-tune their requirements for an improved defense of human health.

Identifying unknown by-products in drinking water using comprehensive two-dimensional gas chromatography-quadrupole mass spectrometry and in silico toxicity assessment

Improvements in extraction and detection technologies have increased our abilities to identify new disinfection by-products (DBPs) over the last 40 years. However, most previous studies combined DBP identification and measurement efforts with toxicology to address concerns on a few expected DBPs, making it difficult to better define the health risk from the individual DBPs. In this study, a nontargeted screening method involving comprehensive two-dimensional gas chromatography-quadrupole mass spectrometry (GC × GC-qMS) combined with OECD QSAR Toolbox Ver. 3.2 was developed for identifying and prioritizing of volatile and semi-volatile DBPs in drinking water. The method was successfully applied to analyze DBPs formed during chlorination, chloramination or ozonation of the raw water. Over 500 compounds were tentatively identified in each sample, showing the superior performance of this analytical technique. A total of 170 volatile and semi-volatile DBPs representing fourteen chemical classes were then identified, according to the criteria that the DBP was presented in the duplicate treated samples. The genotoxicity and carcinogenicity of the DBPs were evaluated using Toolbox, and 58 DBPs were found to be actual or potential genotoxicants. The accuracy of the compound identification was determined by comparing 47 identified compounds with commercially available standards. About 90% (41 of the 47) of the compounds that were automatically identified using the library were correct. The results show that GC×GC-qMS coupled with a quantitative structure-activity relationship model is a powerful and fast nontargeted screening technique for compounds. The method and results provide us a new idea for identification and prioritization of DBPs.

Effect-based assessment of persistent organic pollutant and pesticide dumpsite using mammalian CALUX reporter cell lines

A combined chemical and biological analysis of samples from a major obsolete pesticide and persistent organic pollutant (POP) dumpsite in Northern Tajikistan was carried out. The chemical analytical screening focused on a range of prioritized compounds and compounds known to be present locally. Since chemical analytics does not allow measurements of hazards in complex mixtures, we tested the use of a novel effect-based approach using a panel of quantitative high-throughput CALUX reporter assays measuring distinct biological effects relevant in hazard assessment. Assays were included for assessing effects related to estrogen, androgen, and progestin signaling, aryl hydrocarbon receptor-mediated signaling, AP1 signaling, genotoxicity, oxidative stress, chemical hypoxia, and ER stress. With this panel of assays, we first quantified the biological activities of the individual chemicals measured in chemical analytics. Next, we calculated the expected sum activity by these chemicals in the samples of the pesticide dump site and compared the results with the measured CALUX bioactivity of the total extracts of these samples. The results showed that particularly endocrine disruption-related effects were common among the samples. This was consistent with the toxicological profiles of the individual chemicals that dominated these samples. However, large discrepancies between chemical and biological analysis were found in a sample from a burn place present in this site, with biological activities that could not be explained by chemical analysis. This is likely to be caused by toxic combustion products or by spills of compounds that were not targeted in the chemical analysis.

The eTOX data-sharing project to advance in silico drug-induced toxicity prediction

The high-quality in vivo preclinical safety data produced by the pharmaceutical industry during drug development, which follows numerous strict guidelines, are mostly not available in the public domain. These safety data are sometimes published as a condensed summary for the few compounds that reach the market, but the majority of studies are never made public and are often difficult to access in an automated way, even sometimes within the owning company itself. It is evident from many academic and industrial examples, that useful data mining and model development requires large and representative data sets and careful curation of the collected data. In 2010, under the auspices of the Innovative Medicines Initiative, the eTOX project started with the objective of extracting and sharing preclinical study data from paper or pdf archives of toxicology departments of the 13 participating pharmaceutical companies and using such data for establishing a detailed, well-curated database, which could then serve as source for read-across approaches (early assessment of the potential toxicity of a drug candidate by comparison of similar structure and/or effects) and training of predictive models. The paper describes the efforts undertaken to allow effective data sharing intellectual property (IP) protection and set up of adequate controlled vocabularies) and to establish the database (currently with over 4000 studies contributed by the pharma companies corresponding to more than 1400 compounds). In addition, the status of predictive models building and some specific features of the eTOX predictive system (eTOXsys) are presented as decision support knowledge-based tools for drug development process at an early stage.

IARC classes 1 and 2 carcinogens are successfully identified by an alternative strategy that detects DNA-reactivity and cell transformation ability of chemicals

For decades, traditional toxicology has been the ultimate source of information on the carcinogenic potential of chemicals; however with increasing demand on regulation of chemicals and decreasing resources for testing, opportunities to accept "alternative" approaches have dramatically expanded. The need for tools able to identify carcinogens in shorter times and at a lower cost in terms of animal lives and money is still an open issue, and the present strategies and regulations for carcinogenicity pre-screening do not adequately protect human health. In previous papers, we have proposed an integrated in vitro/in silico strategy that detects DNA-reactivity and tissue disorganization/disruption by chemicals, and we have shown that the combination of Salmonella and Structural Alerts for the DNA-reactive carcinogens, and in vitro cell transformation assays for nongenotoxic carcinogens permits the identification of a very large proportion (up to 95%) of rodent carcinogens, while having a considerable specificity with the rodent noncarcinogens. In the present paper we expand the previous investigation and show that this alternative strategy identifies correctly IARC Classes 1 and 2 carcinogens. If implemented, this alternative strategy can contribute to improve the protection of human health while decreasing the use of animals.

Evaluation of the toxicity forecasting capability of EPA's ToxCast Phase I data: can ToxCast in vitro assays predict carcinogenicity?

Long-term rodent bioassays have played a central role in protecting human health from carcinogens; for ethical and practical reasons their use is decreasing whereas genotoxicity testing has taken a pivotal role. However, this strategy--as presently implemented--is not sensitive enough to detect all genotoxic carcinogens, and cannot detect nongenotoxic carcinogens. Among the alternative approaches under study there is the ToxCast/Tox21 project. Following a previous study from our laboratory, here we present a new, more extensive analysis of ToxCast Phase I results, indicating that at the present state-of-art this approach is not able to predict the carcinogenicity of chemicals. Possible reasons for this mediocre performance are discussed, and opinions on ways to tune up the project in the next phases are presented.