1
|
Wedlake AJ, Allen TEH, Goodman JM, Gutsell S, Kukic P, Russell PJ. Confidence in Inactive and Active Predictions from Structural Alerts. Chem Res Toxicol 2020; 33:3010-3022. [PMID: 33295767 DOI: 10.1021/acs.chemrestox.0c00332] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
Abstract
Having a measure of confidence in computational predictions of biological activity from in silico tools is vital when making predictions for new chemicals, for example, in chemical risk assessment. Where predictions of biological activity are used as an indicator of a potential hazard, false-negative predictions are the most concerning prediction; however, assigning confidence in inactive predictions is particularly challenging. How can one confidently identify the absence of activating features? In this study, we present methods for assigning confidence to both active and inactive predictions from structural alerts for protein-binding molecular initiating events (MIEs). Structural alerts were derived through an iterative statistical method. Confidence in the activity predictions is assigned by measuring the Tanimoto similarity between Morgan fingerprints of chemicals in the test set to relevant chemicals in the training set, and suitable cutoff values have been defined to give different confidence categories. To avoid a potential compound series bias in the test set and hence overestimate the performance of the method, we measured the biological activity of 27 compounds with 24 proteins, which gave us an additional 648 experimental measurements; many of the measurements are currently nonexistent in the literature and databases. This data set was complemented with newly measured biological activities published in ChEMBL25 and formed a combined independent validation data set. Applying the confidence categories to the computational predictions for the new data leads to the identification of chemicals for which one should be confident of either an inactive or active prediction, allowing model predictions to be used responsibly.
Collapse
Affiliation(s)
- Andrew J Wedlake
- Centre for Molecular Informatics, Department of Chemistry, University of Cambridge, Lensfield Road, Cambridge CB2 1EW, United Kingdom
| | - Timothy E H Allen
- Centre for Molecular Informatics, Department of Chemistry, University of Cambridge, Lensfield Road, Cambridge CB2 1EW, United Kingdom.,MRC Toxicology Unit, University of Cambridge, Gleeson Building, Tennis Court Road, Cambridge CB2 1QR, United Kingdom
| | - Jonathan M Goodman
- Centre for Molecular Informatics, Department of Chemistry, University of Cambridge, Lensfield Road, Cambridge CB2 1EW, United Kingdom
| | - Steve Gutsell
- Unilever Safety and Environmental Assurance Centre, Colworth Science Park, Sharnbrook, Bedfordshire MK44 1LQ, United Kingdom
| | - Predrag Kukic
- Unilever Safety and Environmental Assurance Centre, Colworth Science Park, Sharnbrook, Bedfordshire MK44 1LQ, United Kingdom
| | - Paul J Russell
- Unilever Safety and Environmental Assurance Centre, Colworth Science Park, Sharnbrook, Bedfordshire MK44 1LQ, United Kingdom
| |
Collapse
|
2
|
Allen TEH, Wedlake AJ, Gelžinytė E, Gong C, Goodman JM, Gutsell S, Russell PJ. Neural network activation similarity: a new measure to assist decision making in chemical toxicology. Chem Sci 2020; 11:7335-7348. [PMID: 34123016 PMCID: PMC8159362 DOI: 10.1039/d0sc01637c] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/19/2020] [Accepted: 06/23/2020] [Indexed: 12/03/2022] Open
Abstract
Deep learning neural networks, constructed for the prediction of chemical binding at 79 pharmacologically important human biological targets, show extremely high performance on test data (accuracy 92.2 ± 4.2%, MCC 0.814 ± 0.093 and ROC-AUC 0.96 ± 0.04). A new molecular similarity measure, Neural Network Activation Similarity, has been developed, based on signal propagation through the network. This is complementary to standard Tanimoto similarity, and the combined use increases confidence in the computer's prediction of activity for new chemicals by providing a greater understanding of the underlying justification. The in silico prediction of these human molecular initiating events is central to the future of chemical safety risk assessment and improves the efficiency of safety decision making.
Collapse
Affiliation(s)
- Timothy E H Allen
- MRC Toxicology Unit, University of Cambridge Hodgkin Building, Lancaster Road Leicester LE1 7HB UK
- Centre for Molecular Informatics, Department of Chemistry, University of Cambridge Lensfield Road Cambridge CB2 1EW UK
| | - Andrew J Wedlake
- Centre for Molecular Informatics, Department of Chemistry, University of Cambridge Lensfield Road Cambridge CB2 1EW UK
| | - Elena Gelžinytė
- Centre for Molecular Informatics, Department of Chemistry, University of Cambridge Lensfield Road Cambridge CB2 1EW UK
| | - Charles Gong
- Centre for Molecular Informatics, Department of Chemistry, University of Cambridge Lensfield Road Cambridge CB2 1EW UK
| | - Jonathan M Goodman
- Centre for Molecular Informatics, Department of Chemistry, University of Cambridge Lensfield Road Cambridge CB2 1EW UK
| | - Steve Gutsell
- Unilever Safety and Environmental Assurance Centre, Colworth Science Park Sharnbrook Bedfordshire MK44 1LQ UK
| | - Paul J Russell
- Unilever Safety and Environmental Assurance Centre, Colworth Science Park Sharnbrook Bedfordshire MK44 1LQ UK
| |
Collapse
|
3
|
Wedlake AJ, Folia M, Piechota S, Allen TEH, Goodman JM, Gutsell S, Russell PJ. Structural Alerts and Random Forest Models in a Consensus Approach for Receptor Binding Molecular Initiating Events. Chem Res Toxicol 2020; 33:388-401. [PMID: 31850746 DOI: 10.1021/acs.chemrestox.9b00325] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/22/2023]
Abstract
A molecular initiating event (MIE) is the gateway to an adverse outcome pathway (AOP), a sequence of events ending in an adverse effect. In silico predictions of MIEs are a vital tool in a modern, mechanism-focused approach to chemical risk assessment. For 90 biological targets representing important human MIEs, structural alert-based models have been constructed with an automated procedure that uses Bayesian statistics to iteratively select substructures. These models give impressive average performance statistics (an average of 92% correct predictions across targets), significantly improving on previous models. Random Forest models have been constructed from physicochemical features for the same targets, giving similarly impressive performance statistics (93% correct predictions). A key difference between the models is interpretation of predictions-the structural alert models are transparent and easy to interpret, while Random Forest models can only identify the most important physicochemical features for making predictions. The two complementary models have been combined in a consensus model, improving performance compared to each individual model (94% correct predictions) and increasing confidence in predictions. Variation in model performance has been explained by calculating a modelability index (MODI), using Tanimoto coefficient between Morgan fingerprints to identify nearest neighbor chemicals. This work is an important step toward building confidence in the use of in silico tools for assessment of toxicity.
Collapse
Affiliation(s)
- Andrew J Wedlake
- Centre for Molecular Informatics, Department of Chemistry , University of Cambridge , Lensfield Road , Cambridge , CB2 1EW , United Kingdom
| | - Maria Folia
- Unilever Safety and Environmental Assurance Centre , Colworth Science Park , Sharnbrook , Bedfordshire , MK44 1LQ , United Kingdom
| | - Sam Piechota
- Unilever Safety and Environmental Assurance Centre , Colworth Science Park , Sharnbrook , Bedfordshire , MK44 1LQ , United Kingdom
| | - Timothy E H Allen
- Centre for Molecular Informatics, Department of Chemistry , University of Cambridge , Lensfield Road , Cambridge , CB2 1EW , United Kingdom.,MRC Toxicology Unit , University of Cambridge , Lancaster Road , Leicester LE19HN , United Kingdom
| | - Jonathan M Goodman
- Centre for Molecular Informatics, Department of Chemistry , University of Cambridge , Lensfield Road , Cambridge , CB2 1EW , United Kingdom
| | - Steve Gutsell
- Unilever Safety and Environmental Assurance Centre , Colworth Science Park , Sharnbrook , Bedfordshire , MK44 1LQ , United Kingdom
| | - Paul J Russell
- Unilever Safety and Environmental Assurance Centre , Colworth Science Park , Sharnbrook , Bedfordshire , MK44 1LQ , United Kingdom
| |
Collapse
|
4
|
Wilm A, Kühnl J, Kirchmair J. Computational approaches for skin sensitization prediction. Crit Rev Toxicol 2018; 48:738-760. [DOI: 10.1080/10408444.2018.1528207] [Citation(s) in RCA: 20] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/12/2022]
Affiliation(s)
- Anke Wilm
- Center for Bioinformatics, Universität Hamburg, Hamburg, Germany
- HITeC e.V, Hamburg, Germany
| | - Jochen Kühnl
- Front End Innovation, Beiersdorf AG, Hamburg, Germany
| | - Johannes Kirchmair
- Center for Bioinformatics, Universität Hamburg, Hamburg, Germany
- Department of Chemistry, University of Bergen, Bergen, Norway
- Computational Biology Unit (CBU), University of Bergen, Bergen, Norway
| |
Collapse
|
5
|
Fitzpatrick JM, Roberts DW, Patlewicz G. An evaluation of selected (Q)SARs/expert systems for predicting skin sensitisation potential. SAR AND QSAR IN ENVIRONMENTAL RESEARCH 2018; 29:439-468. [PMID: 29676182 PMCID: PMC6077848 DOI: 10.1080/1062936x.2018.1455223] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/05/2018] [Accepted: 03/17/2018] [Indexed: 06/08/2023]
Abstract
Predictive testing to characterise substances for their skin sensitisation potential has historically been based on animal models such as the Local Lymph Node Assay (LLNA) and the Guinea Pig Maximisation Test (GPMT). In recent years, EU regulations, have provided a strong incentive to develop non-animal alternatives, such as expert systems software. Here we selected three different types of expert systems: VEGA (statistical), Derek Nexus (knowledge-based) and TIMES-SS (hybrid), and evaluated their performance using two large sets of animal data: one set of 1249 substances from eChemportal and a second set of 515 substances from NICEATM. A model was considered successful at predicting skin sensitisation potential if it had at least the same balanced accuracy as the LLNA and the GPMT had in predicting the other outcomes, which ranged from 79% to 86%. We found that the highest balanced accuracy of any of the expert systems evaluated was 65% when making global predictions. For substances within the domain of TIMES-SS, however, balanced accuracies for the two datasets were found to be 79% and 82%. In those cases where a chemical was within the TIMES-SS domain, the TIMES-SS skin sensitisation hazard prediction had the same confidence as the result from LLNA or GPMT.
Collapse
Affiliation(s)
- Jeremy M Fitzpatrick
- National Center for Computational Toxicology (NCCT), US Environmental Protection Agency (US EPA), 109 T W Alexander Dr, Research Triangle Park (RTP), NC 27711, USA
| | - David W Roberts
- School of Pharmacy, Liverpool John Moores University, Byrom Street, Liverpool, UK
| | - Grace Patlewicz
- National Center for Computational Toxicology (NCCT), US Environmental Protection Agency (US EPA), 109 T W Alexander Dr, Research Triangle Park (RTP), NC 27711, USA
| |
Collapse
|
6
|
Abstract
The use of computational toxicology methods within drug discovery began in the early 2000s with applications such as predicting bacterial mutagenicity and hERG inhibition. The field has been continuously expanding ever since and the tasks at hand have become more complex. These approaches are now strategically integrated into the risk assessment process, as a complement to in vitro and in vivo methods. Today, computational toxicology can be used in every phase of drug discovery and development, from profiling large libraries early on, to predicting off-target effects in the mid-discovery phase, to assessing potential mutagenic impurities in development and degradants as part of life-cycle management. This chapter provides an overview of the field and describes the application of computational toxicology throughout the entire discovery and development process.
Collapse
Affiliation(s)
- Catrin Hasselgren
- PureInfo Discovery Inc., Albuquerque, NM, USA.
- Leadscope Inc., Columbus, OH, USA.
| | | |
Collapse
|
7
|
Coish P, Brooks BW, Gallagher EP, Mills M, Kavanagh TJ, Simcox N, Lasker GA, Botta D, Schmuck SC, Voutchkova-Kostal A, Kostal J, Mullins ML, Nesmith SM, Mellor KE, Corrales J, Kristofco LA, Saari GN, Steele B, Shen LQ, Melnikov F, Zimmerman JB, Anastas PT. The Molecular Design Research Network. Toxicol Sci 2017; 161:241-248. [DOI: 10.1093/toxsci/kfx175] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/18/2023] Open
|
8
|
Integrated Computational Solution for Predicting Skin Sensitization Potential of Molecules. PLoS One 2016; 11:e0155419. [PMID: 27271321 PMCID: PMC4896476 DOI: 10.1371/journal.pone.0155419] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/21/2015] [Accepted: 04/28/2016] [Indexed: 11/29/2022] Open
Abstract
Introduction Skin sensitization forms a major toxicological endpoint for dermatology and cosmetic products. Recent ban on animal testing for cosmetics demands for alternative methods. We developed an integrated computational solution (SkinSense) that offers a robust solution and addresses the limitations of existing computational tools i.e. high false positive rate and/or limited coverage. Results The key components of our solution include: QSAR models selected from a combinatorial set, similarity information and literature-derived sub-structure patterns of known skin protein reactive groups. Its prediction performance on a challenge set of molecules showed accuracy = 75.32%, CCR = 74.36%, sensitivity = 70.00% and specificity = 78.72%, which is better than several existing tools including VEGA (accuracy = 45.00% and CCR = 54.17% with ‘High’ reliability scoring), DEREK (accuracy = 72.73% and CCR = 71.44%) and TOPKAT (accuracy = 60.00% and CCR = 61.67%). Although, TIMES-SS showed higher predictive power (accuracy = 90.00% and CCR = 92.86%), the coverage was very low (only 10 out of 77 molecules were predicted reliably). Conclusions Owing to improved prediction performance and coverage, our solution can serve as a useful expert system towards Integrated Approaches to Testing and Assessment for skin sensitization. It would be invaluable to cosmetic/ dermatology industry for pre-screening their molecules, and reducing time, cost and animal testing.
Collapse
|
9
|
Furuhama A, Hasunuma K, Hayashi TI, Tatarazako N. Predicting algal growth inhibition toxicity: three-step strategy using structural and physicochemical properties. SAR AND QSAR IN ENVIRONMENTAL RESEARCH 2016; 27:343-362. [PMID: 27171903 DOI: 10.1080/1062936x.2016.1174151] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/17/2016] [Accepted: 03/31/2016] [Indexed: 06/05/2023]
Abstract
We propose a three-step strategy that uses structural and physicochemical properties of chemicals to predict their 72 h algal growth inhibition toxicities against Pseudokirchneriella subcapitata. In Step 1, using a log D-based criterion and structural alerts, we produced an interspecies QSAR between algal and acute daphnid toxicities for initial screening of chemicals. In Step 2, we categorized chemicals according to the Verhaar scheme for aquatic toxicity, and we developed QSARs for toxicities of Class 1 (non-polar narcotic) and Class 2 (polar narcotic) chemicals by means of simple regression with a hydrophobicity descriptor and multiple regression with a hydrophobicity descriptor and a quantum chemical descriptor. Using the algal toxicities of the Class 1 chemicals, we proposed a baseline QSAR for calculating their excess toxicities. In Step 3, we used structural profiles to predict toxicity either quantitatively or qualitatively and to assign chemicals to the following categories: Pesticide, Reactive, Toxic, Toxic low and Uncategorized. Although this three-step strategy cannot be used to estimate the algal toxicities of all chemicals, it is useful for chemicals within its domain. The strategy is also applicable as a component of Integrated Approaches to Testing and Assessment.
Collapse
Affiliation(s)
- A Furuhama
- a Centre for Health and Environmental Risk Research , National Institute for Environmental Studies , Tsukuba , Japan
| | - K Hasunuma
- a Centre for Health and Environmental Risk Research , National Institute for Environmental Studies , Tsukuba , Japan
| | - T I Hayashi
- a Centre for Health and Environmental Risk Research , National Institute for Environmental Studies , Tsukuba , Japan
| | - N Tatarazako
- a Centre for Health and Environmental Risk Research , National Institute for Environmental Studies , Tsukuba , Japan
| |
Collapse
|
10
|
Dearden JC, Hewitt M, Roberts DW, Enoch SJ, Rowe PH, Przybylak KR, Vaughan-Williams GD, Smith ML, Pillai GG, Katritzky AR. Mechanism-Based QSAR Modeling of Skin Sensitization. Chem Res Toxicol 2015; 28:1975-86. [PMID: 26382665 DOI: 10.1021/acs.chemrestox.5b00197] [Citation(s) in RCA: 29] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
Abstract
Many chemicals can induce skin sensitization, and there is a pressing need for non-animal methods to give a quantitative indication of potency. Using two large published data sets of skin sensitizers, we have allocated each sensitizing chemical to one of 10 mechanistic categories and then developed good QSAR models for the seven categories that have a sufficient number of chemicals to allow modeling. Both internal and external validation checks showed that each model had good predictivity.
Collapse
Affiliation(s)
- J C Dearden
- School of Pharmacy & Biomolecular Sciences, Liverpool John Moores University , Byrom Street, Liverpool L3 3AF, United Kingdom
| | - M Hewitt
- School of Pharmacy, University of Wolverhampton , Wulfruna Street, Wolverhampton WV1 1LY, United Kingdom
| | - D W Roberts
- School of Pharmacy & Biomolecular Sciences, Liverpool John Moores University , Byrom Street, Liverpool L3 3AF, United Kingdom
| | - S J Enoch
- School of Pharmacy & Biomolecular Sciences, Liverpool John Moores University , Byrom Street, Liverpool L3 3AF, United Kingdom
| | - P H Rowe
- School of Pharmacy & Biomolecular Sciences, Liverpool John Moores University , Byrom Street, Liverpool L3 3AF, United Kingdom
| | - K R Przybylak
- School of Pharmacy & Biomolecular Sciences, Liverpool John Moores University , Byrom Street, Liverpool L3 3AF, United Kingdom
| | - G D Vaughan-Williams
- School of Pharmacy & Biomolecular Sciences, Liverpool John Moores University , Byrom Street, Liverpool L3 3AF, United Kingdom
| | - M L Smith
- School of Pharmacy & Biomolecular Sciences, Liverpool John Moores University , Byrom Street, Liverpool L3 3AF, United Kingdom
| | - G G Pillai
- Department of Chemistry, University of Florida , Gainsville, Florida 32611-7200, United States.,Institute of Chemistry, University of Tartu , 50411 Tartu, Estonia
| | - A R Katritzky
- Department of Chemistry, University of Florida , Gainsville, Florida 32611-7200, United States
| |
Collapse
|
11
|
Tsujita-Inoue K, Atobe T, Hirota M, Ashikaga T, Kouzuki H. In silico risk assessment for skin sensitization using artificial neural network analysis. J Toxicol Sci 2015; 40:193-209. [DOI: 10.2131/jts.40.193] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/02/2022]
|
12
|
Gutsell S, Russell P. The role of chemistry in developing understanding of adverse outcome pathways and their application in risk assessment. Toxicol Res (Camb) 2013. [DOI: 10.1039/c3tx50024a] [Citation(s) in RCA: 26] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/03/2023] Open
|
13
|
Thyssen JP, Giménez-Arnau E, Lepoittevin JP, Menné T, Boman A, Schnuch A. The critical review of methodologies and approaches to assess the inherent skin sensitization potential (skin allergies) of chemicals Part I. Contact Dermatitis 2012; 66 Suppl 1:11-24. [DOI: 10.1111/j.1600-0536.2011.02004_2.x] [Citation(s) in RCA: 22] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
|
14
|
Furuhama A, Aoki Y, Shiraishi H. Consideration of reactivity to acute fish toxicity of α,β-unsaturated carbonyl ketones and aldehydes. SAR AND QSAR IN ENVIRONMENTAL RESEARCH 2012; 23:169-184. [PMID: 22150015 DOI: 10.1080/1062936x.2011.636381] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/31/2023]
Abstract
To understand the key factor for fish toxicity of 11 α,β-unsaturated carbonyl aldehydes and ketones, we used quantum chemical calculations to investigate their Michael reactions with methanethiol or glutathione. We used two reaction schemes, with and without an explicit water molecule (Scheme-1wat and Scheme-0wat, respectively), to account for the effects of a catalytic water molecule on the reaction pathway. We determined the energies of the reactants, transition states (TS), and products, as well as the activation energies of the reactions. The acute fish toxicities of nine of the carbonyl compounds were evaluated to correlate with their hydrophobicities; no correlation was observed for acrolein and crotonaldehyde. The most toxic compound, acrolein, had the lowest activation energy. The activation energy of the reaction could be estimated with Scheme-1wat but not with Scheme-0wat. The complexity of the reaction pathways of the compounds was reflected in the difficulty of the TS structure searches when Scheme-1wat was used with the polarizable continuum model. The theoretical estimations of activation energies of α,β-unsaturated carbonyl compounds with catalytic molecules or groups including hydrogen-bond networks may complement traditional tools for predicting the acute aquatic toxicities of compounds that cannot be easily obtained experimentally.
Collapse
Affiliation(s)
- A Furuhama
- Center for Environmental Risk Research, National Institute for Environmental Studies (NIES), Tsukuba, Japan.
| | | | | |
Collapse
|
15
|
Lu J, Zheng M, Wang Y, Shen Q, Luo X, Jiang H, Chen K. Fragment-based prediction of skin sensitization using recursive partitioning. J Comput Aided Mol Des 2011; 25:885-93. [DOI: 10.1007/s10822-011-9472-7] [Citation(s) in RCA: 9] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/13/2011] [Accepted: 09/02/2011] [Indexed: 11/25/2022]
|
16
|
Sharma NS, Jindal R, Mitra B, Lee S, Li L, Maguire TJ, Schloss R, Yarmush ML. Perspectives on Non-Animal Alternatives for Assessing Sensitization Potential in Allergic Contact Dermatitis. Cell Mol Bioeng 2011; 5:52-72. [PMID: 24741377 DOI: 10.1007/s12195-011-0189-4] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/09/2023] Open
Abstract
Skin sensitization remains a major environmental and occupational health hazard. Animal models have been used as the gold standard method of choice for estimating chemical sensitization potential. However, a growing international drive and consensus for minimizing animal usage have prompted the development of in vitro methods to assess chemical sensitivity. In this paper, we examine existing approaches including in silico models, cell and tissue based assays for distinguishing between sensitizers and irritants. The in silico approaches that have been discussed include Quantitative Structure Activity Relationships (QSAR) and QSAR based expert models that correlate chemical molecular structure with biological activity and mechanism based read-across models that incorporate compound electrophilicity. The cell and tissue based assays rely on an assortment of mono and co-culture cell systems in conjunction with 3D skin models. Given the complexity of allergen induced immune responses, and the limited ability of existing systems to capture the entire gamut of cellular and molecular events associated with these responses, we also introduce a microfabricated platform that can capture all the key steps involved in allergic contact sensitivity. Finally, we describe the development of an integrated testing strategy comprised of two or three tier systems for evaluating sensitization potential of chemicals.
Collapse
Affiliation(s)
- Nripen S Sharma
- Department of Biomedical Engineering, Rutgers University, 599 Taylor Road, 231, Piscataway, NJ 08854, USA
| | - Rohit Jindal
- Department of Biomedical Engineering, Rutgers University, 599 Taylor Road, 231, Piscataway, NJ 08854, USA
| | - Bhaskar Mitra
- Department of Biomedical Engineering, Rutgers University, 599 Taylor Road, 231, Piscataway, NJ 08854, USA
| | - Serom Lee
- Department of Biomedical Engineering, Rutgers University, 599 Taylor Road, 231, Piscataway, NJ 08854, USA
| | - Lulu Li
- Department of Biomedical Engineering, Rutgers University, 599 Taylor Road, 231, Piscataway, NJ 08854, USA
| | - Tim J Maguire
- Department of Biomedical Engineering, Rutgers University, 599 Taylor Road, 231, Piscataway, NJ 08854, USA
| | - Rene Schloss
- Department of Biomedical Engineering, Rutgers University, 599 Taylor Road, 231, Piscataway, NJ 08854, USA
| | - Martin L Yarmush
- Department of Biomedical Engineering, Rutgers University, 599 Taylor Road, 231, Piscataway, NJ 08854, USA ; Center for Engineering in Medicine, Massachusetts General Hospital, Boston, MA 02114, USA
| |
Collapse
|
17
|
Furuhama A, Hasunuma K, Aoki Y, Yoshioka Y, Shiraishi H. Application of chemical reaction mechanistic domains to an ecotoxicity QSAR model, the KAshinhou Tool for Ecotoxicity (KATE). SAR AND QSAR IN ENVIRONMENTAL RESEARCH 2011; 22:505-523. [PMID: 21604231 DOI: 10.1080/1062936x.2011.569944] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/30/2023]
Abstract
The validity of chemical reaction mechanistic domains defined by skin sensitisation in the Quantitative Structure-Activity Relationship (QSAR) ecotoxicity system, KAshinhou Tools for Ecotoxicity (KATE), March 2009 version, has been assessed and an external validation of the current KATE system carried out. In the case of the fish end-point, the group of chemicals with substructures reactive to skin sensitisation always exhibited higher root mean square errors (RMSEs) than chemicals without reactive substructures under identical C- or log P-judgements in KATE. However, in the case of the Daphnia end-point this was not so, and the group of chemicals with reactive substructures did not always have higher RMSEs: the Schiff base mechanism did not function as a high error detector. In addition to the RMSE findings, the presence of outliers suggested that the KATE classification rules needs to be reconsidered, particularly for the amine group. Examination of the dependency of the organism on the toxic action of chemicals in fish and Daphnia revealed that some of the reactive substructures could be applied to the improvement of the KATE system. It was concluded that the reaction mechanistic domains of toxic action for skin sensitisation could provide useful complementary information in predicting acute aquatic ecotoxicity, especially at the fish end-point.
Collapse
Affiliation(s)
- A Furuhama
- Research Center for Environmental Risk, National Institute for Environmental Studies (NIES), Tsukuba, Japan
| | | | | | | | | |
Collapse
|
18
|
Schwöbel JAH, Koleva YK, Enoch SJ, Bajot F, Hewitt M, Madden JC, Roberts DW, Schultz TW, Cronin MTD. Measurement and Estimation of Electrophilic Reactivity for Predictive Toxicology. Chem Rev 2011; 111:2562-96. [DOI: 10.1021/cr100098n] [Citation(s) in RCA: 149] [Impact Index Per Article: 11.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/15/2022]
Affiliation(s)
- Johannes A. H. Schwöbel
- School of Pharmacy and Chemistry, Liverpool John Moores University, Byrom Street, Liverpool L3 3AF, England
| | - Yana K. Koleva
- School of Pharmacy and Chemistry, Liverpool John Moores University, Byrom Street, Liverpool L3 3AF, England
| | - Steven J. Enoch
- School of Pharmacy and Chemistry, Liverpool John Moores University, Byrom Street, Liverpool L3 3AF, England
| | - Fania Bajot
- School of Pharmacy and Chemistry, Liverpool John Moores University, Byrom Street, Liverpool L3 3AF, England
| | - Mark Hewitt
- School of Pharmacy and Chemistry, Liverpool John Moores University, Byrom Street, Liverpool L3 3AF, England
| | - Judith C. Madden
- School of Pharmacy and Chemistry, Liverpool John Moores University, Byrom Street, Liverpool L3 3AF, England
| | - David W. Roberts
- School of Pharmacy and Chemistry, Liverpool John Moores University, Byrom Street, Liverpool L3 3AF, England
| | - Terry W. Schultz
- College of Veterinary Medicine, Department of Comparative Medicine, The University of Tennessee, 2407 River Drive, Knoxville, Tennessee 37996-4543, United States
| | - Mark T. D. Cronin
- School of Pharmacy and Chemistry, Liverpool John Moores University, Byrom Street, Liverpool L3 3AF, England
| |
Collapse
|
19
|
Price K, Krishnan K. An integrated QSAR-PBPK modelling approach for predicting the inhalation toxicokinetics of mixtures of volatile organic chemicals in the rat. SAR AND QSAR IN ENVIRONMENTAL RESEARCH 2011; 22:107-128. [PMID: 21391144 DOI: 10.1080/1062936x.2010.548350] [Citation(s) in RCA: 19] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/30/2023]
Abstract
The objective of this study was to predict the inhalation toxicokinetics of chemicals in mixtures using an integrated QSAR-PBPK modelling approach. The approach involved: (1) the determination of partition coefficients as well as V(max) and K(m) based solely on chemical structure for 53 volatile organic compounds, according to the group contribution approach; and (2) using the QSAR-driven coefficients as input in interaction-based PBPK models in the rat to predict the pharmacokinetics of chemicals in mixtures of up to 10 components (benzene, toluene, m-xylene, o-xylene, p-xylene, ethylbenzene, dichloromethane, trichloroethylene, tetrachloroethylene, and styrene). QSAR-estimated values of V(max) varied compared with experimental results by a factor of three for 43 out of 53 studied volatile organic compounds (VOCs). K(m) values were within a factor of three compared with experimental values for 43 out of 53 VOCs. Cross-validation performed as a ratio of predicted residual sum of squares and sum of squares of the response value indicates a value of 0.108 for V(max) and 0.208 for K(m). The integration of QSARs for partition coefficients, V(max) and K(m), as well as setting the K(m) equal to K(i) (metabolic inhibition constant) within the mixture PBPK model allowed to generate simulations of the inhalation pharmacokinetics of benzene, toluene, m-xylene, o-xylene, p-xylene, ethylbenzene, dichloromethane, trichloroethylene, tetrachloroethylene and styrene in various mixtures. Overall, the present study indicates the potential usefulness of the QSAR-PBPK modelling approach to provide first-cut evaluations of the kinetics of chemicals in mixtures of increasing complexity, on the basis of chemical structure.
Collapse
Affiliation(s)
- K Price
- Departement de sante environnementale et sante au travail, Faculte de medecine, Universite de Montreal, PQ, Canada
| | | |
Collapse
|
20
|
Patlewicz G, Chen MW, Bellin CA. Non-testing approaches under REACH--help or hindrance? Perspectives from a practitioner within industry. SAR AND QSAR IN ENVIRONMENTAL RESEARCH 2011; 22:67-88. [PMID: 21391142 DOI: 10.1080/1062936x.2010.528448] [Citation(s) in RCA: 10] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/30/2023]
Abstract
Legislation such as REACH strongly advocates the use of alternative approaches including in vitro, (Q)SARs, and chemical categories as a means to satisfy the information requirements for risk assessment. One of the most promising alternative approaches is that of chemical categories, where the underlying hypothesis is that the compounds within the category are similar and therefore should have similar biological activities. The challenge lies in characterizing the chemicals, understanding the mode/mechanism of action for the activity of interest and deriving a way of relating these together to form inferences about the likely activity outcomes. (Q)SARs are underpinned by the same hypothesis but are packaged in a more formalized manner. Since the publication of the White Paper for REACH, there have been a number of efforts aimed at developing tools, approaches and techniques for (Q)SARs and read-across for regulatory purposes. While technical guidance is available, there still remains little practical guidance about how these approaches can or should be applied in either the evaluation of existing (Q)SARs or in the formation of robust categories. Here we provide a perspective of how some of these approaches have been utilized to address our in-house REACH requirements.
Collapse
Affiliation(s)
- G Patlewicz
- DuPont Haskell Global Centers for Health and Environmental Sciences, Newark, USA.
| | | | | |
Collapse
|
21
|
Roberts DW, Natsch A. High throughput kinetic profiling approach for covalent binding to peptides: application to skin sensitization potency of Michael acceptor electrophiles. Chem Res Toxicol 2010; 22:592-603. [PMID: 19206519 DOI: 10.1021/tx800431x] [Citation(s) in RCA: 105] [Impact Index Per Article: 7.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
Abstract
Research aimed at nonanimal approaches to provide the relevant information needed for the effective assessment of skin sensitization, for both hazard characterization and risk assessment purposes, is currently an area of high activity, stimulated by regulatory initiatives related to chemicals used in consumer products. The ability of a chemical to react covalently with protein or peptide nucleophiles in the skin is recognized as the key determinant in determining sensitization potency, and initiatives to develop peptide reactivity assays to replace animal testing have been undertaken recently. This paper describes a high throughput kinetic profiling (HTKP) approach, developed as an extension of a published standard assay, with the aim of providing a quantitatively robust end point in the form of a kinetic profile from which reactivity to a model peptide can be quantified in the form of second order rate constants. The approach allows solubility issues to be identified and overcome; these are frequently encountered, but can often go undetected, in aqueous reactivity assays with organic compounds of interest in the skin sensitization context. Using rate constants determined by the HTKP approach we have obtained a quantitative mechanistic model for the Michael acceptor reaction mechanistic domain, relating the sensitization potency in the murine local lymph node assay to the rate constant. The observation that the correlation is not improved by incorporation of a hydrophobicity term has implications regarding the nature and location of the skin nucleophile whose reaction leads to sensitization by Michael acceptor electrophiles.
Collapse
Affiliation(s)
- David W Roberts
- School of Pharmacy and Chemistry, Liverpool John Moores University, Byrom Street, Liverpool, L33AF England.
| | | |
Collapse
|
22
|
Cronin MTD. Characterisation, Evaluation and Possible Validation of In Silico Models for Toxicity: Determining if a Prediction is Valid. IN SILICO TOXICOLOGY 2010. [DOI: 10.1039/9781849732093-00275] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/29/2023]
Abstract
This chapter describes the process whereby a (Q)SAR may be described, evaluated and, where possible, validated. The emphasis here is not to develop models, but to characterise them according to the guidance supplied by the Organisation for Economic Co-operation and Development (OECD) and the European Chemicals Agency (EChA). The backbone to this process are the OECD Principles for the Validation of (Q)SARs. Three case studies illustrating how to approach the OECD Principles are supplied.
Collapse
Affiliation(s)
- M. T. D. Cronin
- School of Pharmacy and Chemistry, Liverpool John Moores University Byrom Street Liverpool L3 3AF UK
| |
Collapse
|
23
|
Cronin MTD. Finding the Data to Develop and Evaluate (Q)SARs and Populate Categories for Toxicity Prediction. IN SILICO TOXICOLOGY 2010. [DOI: 10.1039/9781849732093-00031] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/29/2023]
Abstract
This chapter describes the sources of data for in silico modelling. It is assumed that the modeller will not normally have the facilities to experimentally determine toxicological data, thus they must rely on existing data. Data can be obtained from in-house sources (e.g. for industry) or from publicly available databases and the scientific literature. For the publicly available data, the sources of toxicologically information and the relevant advantages and disadvantages are defined. The sources include “well-established” datasets and the use of literature searching, through to the use of databases and more global (meta) data portals which call on a number of databases. To use the data collected efficiently, the modeller must define the required endpoint, allow the nature of the data to drive the modelling approach and control the quality of the data and implications for that on in silico models.
Collapse
Affiliation(s)
- M. T. D. Cronin
- School of Pharmacy and Chemistry, Liverpool John Moores University Byrom Street Liverpool L3 3AF UK
| |
Collapse
|
24
|
Mekenyan O, Patlewicz G, Dimitrova G, Kuseva C, Todorov M, Stoeva S, Kotov S, Donner EM. Use of Genotoxicity Information in the Development of Integrated Testing Strategies (ITS) for Skin Sensitization. Chem Res Toxicol 2010; 23:1519-40. [DOI: 10.1021/tx100161j] [Citation(s) in RCA: 19] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
Affiliation(s)
- Ovanes Mekenyan
- Laboratory of Mathematical Chemistry, “Prof. As. Zlatarov” University, Bourgas, Bulgaria, and DuPont Haskell Global Centers for Health and Environmental Sciences, 1090 Elkton Road, Newark, Delaware 19711
| | - Grace Patlewicz
- Laboratory of Mathematical Chemistry, “Prof. As. Zlatarov” University, Bourgas, Bulgaria, and DuPont Haskell Global Centers for Health and Environmental Sciences, 1090 Elkton Road, Newark, Delaware 19711
| | - Gergana Dimitrova
- Laboratory of Mathematical Chemistry, “Prof. As. Zlatarov” University, Bourgas, Bulgaria, and DuPont Haskell Global Centers for Health and Environmental Sciences, 1090 Elkton Road, Newark, Delaware 19711
| | - Chanita Kuseva
- Laboratory of Mathematical Chemistry, “Prof. As. Zlatarov” University, Bourgas, Bulgaria, and DuPont Haskell Global Centers for Health and Environmental Sciences, 1090 Elkton Road, Newark, Delaware 19711
| | - Milen Todorov
- Laboratory of Mathematical Chemistry, “Prof. As. Zlatarov” University, Bourgas, Bulgaria, and DuPont Haskell Global Centers for Health and Environmental Sciences, 1090 Elkton Road, Newark, Delaware 19711
| | - Stoyanka Stoeva
- Laboratory of Mathematical Chemistry, “Prof. As. Zlatarov” University, Bourgas, Bulgaria, and DuPont Haskell Global Centers for Health and Environmental Sciences, 1090 Elkton Road, Newark, Delaware 19711
| | - Stefan Kotov
- Laboratory of Mathematical Chemistry, “Prof. As. Zlatarov” University, Bourgas, Bulgaria, and DuPont Haskell Global Centers for Health and Environmental Sciences, 1090 Elkton Road, Newark, Delaware 19711
| | - E Maria Donner
- Laboratory of Mathematical Chemistry, “Prof. As. Zlatarov” University, Bourgas, Bulgaria, and DuPont Haskell Global Centers for Health and Environmental Sciences, 1090 Elkton Road, Newark, Delaware 19711
| |
Collapse
|
25
|
Enoch SJ, Roberts DW, Cronin MTD. Mechanistic Category Formation for the Prediction of Respiratory Sensitization. Chem Res Toxicol 2010; 23:1547-55. [DOI: 10.1021/tx100218h] [Citation(s) in RCA: 48] [Impact Index Per Article: 3.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/01/2023]
Affiliation(s)
- S. J. Enoch
- School of Pharmacy and Chemistry, Liverpool John Moores University, Byrom Street, Liverpool, L3 3AF, England
| | - D. W. Roberts
- School of Pharmacy and Chemistry, Liverpool John Moores University, Byrom Street, Liverpool, L3 3AF, England
| | - M. T. D. Cronin
- School of Pharmacy and Chemistry, Liverpool John Moores University, Byrom Street, Liverpool, L3 3AF, England
| |
Collapse
|
26
|
Worth AP. The Role of Qsar Methodology in the Regulatory Assessment of Chemicals. CHALLENGES AND ADVANCES IN COMPUTATIONAL CHEMISTRY AND PHYSICS 2010. [DOI: 10.1007/978-1-4020-9783-6_13] [Citation(s) in RCA: 21] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/25/2023]
|
27
|
|
28
|
Carrera GVSM, Gupta S, Aires-de-Sousa J. Machine learning of chemical reactivity from databases of organic reactions. J Comput Aided Mol Des 2009; 23:419-29. [PMID: 19468693 DOI: 10.1007/s10822-009-9275-2] [Citation(s) in RCA: 15] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/27/2008] [Accepted: 04/18/2009] [Indexed: 10/20/2022]
Abstract
Databases of chemical reactions contain knowledge about the reactivity of specific reagents. Although information is in general only explicitly available for compounds reported to react, it is possible to derive information about substructures that do not react in the reported reactions. Both types of information (positive and negative) can be used to train machine learning techniques to predict if a compound reacts or not with a specific reagent. The whole process was implemented with two databases of reactions, one involving BuNH2 as the reagent, and the other NaCNBH3. Negative information was derived using MOLMAP molecular descriptors, and classification models were developed with Random Forests also based on MOLMAP descriptors. MOLMAP descriptors were based exclusively on calculated physicochemical features of molecules. Correct predictions were achieved for approximately 90% of independent test sets. While NaCNBH3 is a selective reducing reagent widely used in organic synthesis, BuNH2 is a nucleophile that mimics the reactivity of the lysine side chain (involved in an initiating step of the mechanism leading to skin sensitization).
Collapse
Affiliation(s)
- Gonçalo V S M Carrera
- REQUIMTE, CQFB, Departamento de Química, Faculdade de Ciências e Tecnologia, Universidade Nova de Lisboa, 2829-516 Caparica, Portugal
| | | | | |
Collapse
|
29
|
Golla S, Madihally S, Robinson RL, Gasem KA. Quantitative structure–property relationship modeling of skin sensitization: A quantitative prediction. Toxicol In Vitro 2009; 23:454-65. [DOI: 10.1016/j.tiv.2008.12.025] [Citation(s) in RCA: 21] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/05/2008] [Revised: 12/03/2008] [Accepted: 12/17/2008] [Indexed: 10/21/2022]
|
30
|
Putz MV, Putz AM, Lazea M, Ienciu L, Chiriac A. Quantum-SAR extension of the spectral-SAR algorithm: application to polyphenolic anticancer bioactivity. Int J Mol Sci 2009; 10:1193-1214. [PMID: 19399244 PMCID: PMC2672025 DOI: 10.3390/ijms10031193] [Citation(s) in RCA: 23] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/18/2009] [Revised: 03/09/2009] [Accepted: 03/11/2009] [Indexed: 11/30/2022] Open
Abstract
Aiming to assess the role of individual molecular structures in the molecular mechanism of ligand-receptor interaction correlation analysis, the recent Spectral-SAR approach is employed to introduce the Quantum-SAR (QuaSAR) “wave” and “conversion factor” in terms of difference between inter-endpoint inter-molecular activities for a given set of compounds; this may account for inter-conversion (metabolization) of molecular (concentration) effects while indicating the structural (quantum) based influential/detrimental role on bio-/eco- effect in a causal manner rather than by simple inspection of measured values; the introduced QuaSAR method is then illustrated for a study of the activity of a series of flavonoids on breast cancer resistance protein.
Collapse
Affiliation(s)
- Mihai V. Putz
- Laboratory of Computational and Structural Physical Chemistry, Chemistry Department, West University of Timişoara, Pestalozzi Street No.16, Timişoara, RO-300115, Romania; E-Mails:
(M.P.);
(M.L.);
(A.C.)
- “Nicolas Georgescu-Roegen” Forming and Researching Center, 4th, Oituz Str., Timişoara, RO- 300086, Romania
- Author to whom correspondence should be addressed; E-Mail:
; Tel. +40-0256-592-633; Fax: +40-0256-592-620
| | - Ana-Maria Putz
- Laboratory of Computational and Structural Physical Chemistry, Chemistry Department, West University of Timişoara, Pestalozzi Street No.16, Timişoara, RO-300115, Romania; E-Mails:
(M.P.);
(M.L.);
(A.C.)
- Laboratory of Inorganic Chemistry, Timişoara Institute of Chemistry of Romanian Academy, Av. Mihai Viteazul, No.24, Timişoara RO-300223, Romania
| | - Marius Lazea
- Laboratory of Computational and Structural Physical Chemistry, Chemistry Department, West University of Timişoara, Pestalozzi Street No.16, Timişoara, RO-300115, Romania; E-Mails:
(M.P.);
(M.L.);
(A.C.)
| | - Luciana Ienciu
- Whatman, Part of GE Healthcare, Inc, 200 Park Avenue Suite 210, Florham Park, NJ 07932-1026, USA; E-Mail:
| | - Adrian Chiriac
- Laboratory of Computational and Structural Physical Chemistry, Chemistry Department, West University of Timişoara, Pestalozzi Street No.16, Timişoara, RO-300115, Romania; E-Mails:
(M.P.);
(M.L.);
(A.C.)
- “Nicolas Georgescu-Roegen” Forming and Researching Center, 4th, Oituz Str., Timişoara, RO- 300086, Romania
| |
Collapse
|
31
|
Roberts DW, Patlewicz GY. Nonanimal Alternatives for Skin Sensitization: Letter to the Editor. Toxicol Sci 2008; 106:572-4; author reply 575. [DOI: 10.1093/toxsci/kfn181] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022] Open
|
32
|
Bassan A, Worth A. The Integrated Use of Models for the Properties and Effects of Chemicals by means of a Structured Workflow. ACTA ACUST UNITED AC 2008. [DOI: 10.1002/qsar.200710119] [Citation(s) in RCA: 26] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
|