1
|
Bahl A, Halappanavar S, Wohlleben W, Nymark P, Kohonen P, Wallin H, Vogel U, Haase A. Bioinformatics and machine learning to support nanomaterial grouping. Nanotoxicology 2024; 18:373-400. [PMID: 38949108 DOI: 10.1080/17435390.2024.2368005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/28/2023] [Revised: 05/22/2024] [Accepted: 06/11/2024] [Indexed: 07/02/2024]
Abstract
Nanomaterials (NMs) offer plenty of novel functionalities. Moreover, their physicochemical properties can be fine-tuned to meet the needs of specific applications, leading to virtually unlimited numbers of NM variants. Hence, efficient hazard and risk assessment strategies building on New Approach Methodologies (NAMs) become indispensable. Indeed, the design, the development and implementation of NAMs has been a major topic in a substantial number of research projects. One of the promising strategies that can help to deal with the high number of NMs variants is grouping and read-across. Based on demonstrated structural and physicochemical similarity, NMs can be grouped and assessed together. Within an established NM group, read-across may be performed to fill in data gaps for data-poor variants using existing data for NMs within the group. Establishing a group requires a sound justification, usually based on a grouping hypothesis that links specific physicochemical properties to well-defined hazard endpoints. However, for NMs these interrelationships are only beginning to be understood. The aim of this review is to demonstrate the power of bioinformatics with a specific focus on Machine Learning (ML) approaches to unravel the NM Modes-of-Action (MoA) and identify the properties that are relevant to specific hazards, in support of grouping strategies. This review emphasizes the following messages: 1) ML supports identification of the most relevant properties contributing to specific hazards; 2) ML supports analysis of large omics datasets and identification of MoA patterns in support of hypothesis formulation in grouping approaches; 3) omics approaches are useful for shifting away from consideration of single endpoints towards a more mechanistic understanding across multiple endpoints gained from one experiment; and 4) approaches from other fields of Artificial Intelligence (AI) like Natural Language Processing or image analysis may support automated extraction and interlinkage of information related to NM toxicity. Here, existing ML models for predicting NM toxicity and for analyzing omics data in support of NM grouping are reviewed. Various challenges related to building robust models in the field of nanotoxicology exist and are also discussed.
Collapse
Affiliation(s)
- Aileen Bahl
- Department of Chemical and Product Safety, German Federal Institute for Risk Assessment (BfR), Berlin, Germany
- Department of Biological Safety, German Federal Institute for Risk Assessment (BfR), Berlin, Germany
- Freie Universität Berlin, Institute of Pharmacy, Berlin, Germany
| | - Sabina Halappanavar
- Environmental Health Science and Research Bureau, Health Canada, Ottawa, Canada
| | - Wendel Wohlleben
- BASF SE, Department Analytical and Material Science and Department Experimental Toxicology and Ecology, Ludwigshafen, Germany
| | - Penny Nymark
- Institute of Environmental Medicine, Karolinska Institutet, Stockholm, Sweden
| | - Pekka Kohonen
- Institute of Environmental Medicine, Karolinska Institutet, Stockholm, Sweden
| | - Håkan Wallin
- Department of Chemical and Biological Risk Factors, National Institute of Occupational Health, Oslo, Norway
- Department of Public Health, Copenhagen University, Copenhagen, Denmark
| | - Ulla Vogel
- National Research Centre for the Working Environment, Copenhagen, Denmark
| | - Andrea Haase
- Department of Chemical and Product Safety, German Federal Institute for Risk Assessment (BfR), Berlin, Germany
- Freie Universität Berlin, Institute of Pharmacy, Berlin, Germany
| |
Collapse
|
2
|
Wohlleben W, Mehling A, Landsiedel R. Lessons Learned from the Grouping of Chemicals to Assess Risks to Human Health. Angew Chem Int Ed Engl 2023; 62:e202210651. [PMID: 36254879 DOI: 10.1002/anie.202210651] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2022] [Revised: 10/15/2022] [Accepted: 10/17/2022] [Indexed: 11/05/2022]
Abstract
In analogy to the periodic system that groups elements by their similarity in structure and chemical properties, the hazard of chemicals can be assessed in groups having similar structures and similar toxicological properties. Here we review case studies of chemical grouping strategies that supported the assessment of hazard, exposure, and risk to human health. By the EU-REACH and the US-TSCA New Chemicals Program, structural similarity is commonly used as the basis for grouping, but that criterion is not always adequate and sufficient. Based on the lessons learned, we derive ten principles for grouping, including: transparency of the purpose, criteria, and boundaries of the group; adequacy of methods used to justify the group; and inclusion or exclusion of substances in the group by toxicological properties. These principles apply to initial grouping to prioritize further actions as well as to definitive grouping to generate data for risk assessment. Both can expedite effective risk management.
Collapse
Affiliation(s)
- Wendel Wohlleben
- Department of Analytical and Material Science, BASF SE, 67056, Ludwigshafen am Rhein, Germany
- Department of Experimental Toxicology and Ecology, BASF SE, 67056, Ludwigshafen am Rhein, Germany
| | - Annette Mehling
- Dept. of Advanced Formulation and Performance Technology, BASF Personal Care and Nutrition GmbH, 40589, Duesseldorf, Germany
| | - Robert Landsiedel
- Department of Experimental Toxicology and Ecology, BASF SE, 67056, Ludwigshafen am Rhein, Germany
- Free University of Berlin, Biology, Chemistry and Pharmacy-Pharmacology and Toxicology, 14195, Berlin, Germany
| |
Collapse
|
3
|
Guseva Canu I, Plys E, Velarde Crézé C, Fito C, Hopf NB, Progiou A, Riganti C, Sauvain JJ, Squillacioti G, Suarez G, Wild P, Bergamaschi E. A harmonized protocol for an international multicenter prospective study of nanotechnology workers: the NanoExplore cohort. Nanotoxicology 2023; 17:1-19. [PMID: 36927342 DOI: 10.1080/17435390.2023.2180220] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/18/2023]
Abstract
Nanotechnology applications are fast-growing in many industrial fields. Consequently, health effects of engineered nanomaterials (ENMs) should be investigated. Within the EU-Life project NanoExplore, we developed a harmonized protocol of an international multicenter prospective cohort study of workers in ENM-producing companies. This article describes the development of the protocol, sample size calculation, data collection and management procedures and discusses its relevance with respect to research needs. Within this protocol, workers' ENM exposure will be assessed over four consecutive working days during the initial recruitment campaign and the subsequent follow-up campaigns. Biomonitoring using noninvasive sampling of exhaled breath condensate (EBC), exhaled air, and urine will be collected before and after 4-day exposure monitoring. Both exposure and effect biomarkers, will be quantified along with pulmonary function tests and diagnosed diseases reported using a standardized epidemiological questionnaire available in four languages. Until now, this protocol was implemented at seven companies in Switzerland, Spain and Italy. The protocol is well standardized, though sufficiently flexible to include company-specific conditions and occupational hygiene measures. The recruitment, to date, of 140 participants and collection of all data and samples, enabled us launching the first international cohort of nanotechnology workers. All companies dealing with ENMs could join the NanoExplore Consortium, apply this harmonized protocol and enter in the cohort, concieved as an open cohort. Its protocol meets all requirements of a hypotheses-driven prospective study, which will assess and reassess effects of ENM exposure on workers' health by updating the follow-up of the cohort. New hypothesis could be also considered.
Collapse
Affiliation(s)
- Irina Guseva Canu
- Department of Occupational and Environmental Health, Center for Primary Care and Public Health (Unisanté), University of Lausanne, Lausanne, Switzerland
| | - Ekaterina Plys
- Department of Occupational and Environmental Health, Center for Primary Care and Public Health (Unisanté), University of Lausanne, Lausanne, Switzerland
| | - Camille Velarde Crézé
- Department of Occupational and Environmental Health, Center for Primary Care and Public Health (Unisanté), University of Lausanne, Lausanne, Switzerland
| | - Carlos Fito
- Institutotecnológico del embalaje, transporte y logística (ITENE), Paterna, Spain
| | - Nancy B Hopf
- Department of Occupational and Environmental Health, Center for Primary Care and Public Health (Unisanté), University of Lausanne, Lausanne, Switzerland
| | | | - Chiara Riganti
- Department of Oncology, University of Torino, Torino, Italy
| | - Jean-Jacques Sauvain
- Department of Occupational and Environmental Health, Center for Primary Care and Public Health (Unisanté), University of Lausanne, Lausanne, Switzerland
| | - Giulia Squillacioti
- Department of Public Health and Pediatrics, University of Torino, Torino, Italy
| | - Guillaume Suarez
- Department of Occupational and Environmental Health, Center for Primary Care and Public Health (Unisanté), University of Lausanne, Lausanne, Switzerland
| | - Pascal Wild
- Department of Occupational and Environmental Health, Center for Primary Care and Public Health (Unisanté), University of Lausanne, Lausanne, Switzerland
| | - Enrico Bergamaschi
- Department of Public Health and Pediatrics, University of Torino, Torino, Italy
| |
Collapse
|
4
|
McLean P, Mueller W, Gosens I, Cassee FR, Rothen-Rutishauser B, Boyles M, Tran L. Establishing relationships between particle-induced in vitro and in vivo inflammation endpoints to better extrapolate between in vitro markers and in vivo fibrosis. Part Fibre Toxicol 2023; 20:5. [PMID: 36759844 PMCID: PMC9909881 DOI: 10.1186/s12989-023-00516-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/05/2022] [Accepted: 02/01/2023] [Indexed: 02/11/2023] Open
Abstract
BACKGROUND Toxicity assessment for regulatory purposes is starting to move away from traditional in vivo methods and towards new approach methodologies (NAM) such as high-throughput in vitro models and computational tools. For materials with limited hazard information, utilising quantitative Adverse Outcome Pathways (AOPs) in a testing strategy involving NAM can produce information relevant for risk assessment. The aim of this work was to determine the feasibility of linking in vitro endpoints to in vivo events, and moreover to key events associated with the onset of a chosen adverse outcome to aid in the development of NAM testing strategies. To do this, we focussed on the adverse outcome pathway (AOP) relating to the onset of pulmonary fibrosis. RESULTS We extracted in vivo and in vitro dose-response information for particles known to induce this pulmonary fibrosis (crystalline silica, specifically α-quartz). To test the in vivo-in vitro extrapolation (IVIVE) determined for crystalline silica, cerium dioxide nanoparticles (nano-CeO2) were used as a case study allowing us to evaluate our findings with a less studied substance. The IVIVE methodology outlined in this paper is formed of five steps, which can be more generally summarised into two categories (i) aligning the in vivo and in vitro dosimetry, (ii) comparing the dose-response curves and derivation of conversion factors. CONCLUSION Our analysis shows promising results with regards to correlation of in vitro cytokine secretion to in vivo acute pulmonary inflammation assessed by polymorphonuclear leukocyte influx, most notable is the potential of using IL-6 and IL-1β cytokine secretion from simple in vitro submerged models as a screening tool to assess the likelihood of lung inflammation at an early stage in product development, hence allowing a more targeted investigation using either a smaller, more targeted in vivo study or in the future a more complex in vitro protocol. This paper also highlights the strengths and limitations as well as the current difficulties in performing IVIVE assessment and suggestions for overcoming these issues.
Collapse
Affiliation(s)
- Polly McLean
- Institute of Occupational Medicine (IOM), Edinburgh, UK.
| | - William Mueller
- grid.410343.10000 0001 2224 0230Institute of Occupational Medicine (IOM), Edinburgh, UK
| | - Ilse Gosens
- grid.31147.300000 0001 2208 0118National Institute for Public Health and the Environment – RIVM, Bilthoven, The Netherlands
| | - Flemming R. Cassee
- grid.31147.300000 0001 2208 0118National Institute for Public Health and the Environment – RIVM, Bilthoven, The Netherlands ,grid.5477.10000000120346234Institute for Risk Assessment Sciences, Utrecht University, Utrecht, The Netherlands
| | - Barbara Rothen-Rutishauser
- grid.8534.a0000 0004 0478 1713Adolphe Merkle Institute, University of Fribourg, Chemin des Verdiers 4, 1700 Fribourg, Switzerland
| | - Matthew Boyles
- grid.410343.10000 0001 2224 0230Institute of Occupational Medicine (IOM), Edinburgh, UK
| | - Lang Tran
- grid.410343.10000 0001 2224 0230Institute of Occupational Medicine (IOM), Edinburgh, UK
| |
Collapse
|
5
|
Tsiliki G, Ag Seleci D, Zabeo A, Basei G, Hristozov D, Jeliazkova N, Boyles M, Murphy F, Peijnenburg W, Wohlleben W, Stone V. Bayesian based similarity assessment of nanomaterials to inform grouping. NANOIMPACT 2022; 25:100389. [PMID: 35559895 DOI: 10.1016/j.impact.2022.100389] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/29/2021] [Revised: 12/29/2021] [Accepted: 02/02/2022] [Indexed: 06/15/2023]
Abstract
Nanoforms can be manufactured in plenty of variants by differing their physicochemical properties and toxicokinetic behaviour which can affect their hazard potential. To avoid testing of each single nanomaterial and nanoform variation and subsequently save resources, grouping and read-across strategies are used to estimate groups of substances, based on carefully selected evidence, that could potentially have similar human health and environmental hazard impact. A novel computational similarity method is presented aiming to compare dose-response curves and identify sets of similar nanoforms. The suggested method estimates the statistical model that best fits the data by leveraging pairwise Bayes Factor analysis to compare pairs of curves and evaluate whether each of the nanoforms is sufficiently similar to all other nanoforms. Pairwise comparisons to benchmark materials are used to define threshold similarity values and set the criteria for identifying groups of nanoforms with comparatively similar toxicity. Applications to use case data are shown to demonstrate that the method can support grouping hypotheses linked to a certain hazard endpoint and route of exposure.
Collapse
Affiliation(s)
- Georgia Tsiliki
- Institute for the Management of Information Systems, Athena Research Center, Marousi, Greece.
| | - Didem Ag Seleci
- Advanced Materials Research, Dept. of Material Physics and Analytics and Dept. of Experimental Toxicology and Ecology, BASF SE, Ludwigshafen, Germany
| | | | | | | | | | - Matthew Boyles
- Institute of Occupational Medicine, Edinburgh, United Kingdom
| | - Fiona Murphy
- NanoSafety Group, Heriot-Watt University, Edinburgh, United Kingdom
| | - Willie Peijnenburg
- National Institute of Public Health and the Environment (RIVM), Center for Safety of Substances and Products, Bilthoven, the Netherlands; Leiden University, Institute of Environmental Sciences (CML), Leiden, the Netherlands
| | - Wendel Wohlleben
- Advanced Materials Research, Dept. of Material Physics and Analytics and Dept. of Experimental Toxicology and Ecology, BASF SE, Ludwigshafen, Germany
| | - Vicki Stone
- NanoSafety Group, Heriot-Watt University, Edinburgh, United Kingdom
| |
Collapse
|
6
|
Jeliazkova N, Bleeker E, Cross R, Haase A, Janer G, Peijnenburg W, Pink M, Rauscher H, Svendsen C, Tsiliki G, Zabeo A, Hristozov D, Stone V, Wohlleben W. How can we justify grouping of nanoforms for hazard assessment? Concepts and tools to quantify similarity. NANOIMPACT 2022; 25:100366. [PMID: 35559874 DOI: 10.1016/j.impact.2021.100366] [Citation(s) in RCA: 16] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/09/2021] [Revised: 10/15/2021] [Accepted: 11/12/2021] [Indexed: 06/15/2023]
Abstract
The risk of each nanoform (NF) of the same substance cannot be assumed to be the same, as they may vary in their physicochemical characteristics, exposure and hazard. However, neither can we justify a need for more animal testing and resources to test every NF individually. To reduce the need to test all NFs, (regulatory) information requirements may be fulfilled by grouping approaches. For such grouping to be acceptable, it is important to demonstrate similarities in physicochemical properties, toxicokinetic behaviour, and (eco)toxicological behaviour. The GRACIOUS Framework supports the grouping of NFs, by identifying suitable grouping hypotheses that describe the key similarities between different NFs. The Framework then supports the user to gather the evidence required to test these hypotheses and to subsequently assess the similarity of the NFs within the proposed group. The evidence needed to support a hypothesis is gathered by an Integrated Approach to Testing and Assessment (IATA), designed as decision trees constructed of decision nodes. Each decision node asks the questions and provides the methods needed to obtain the most relevant information. This White paper outlines existing and novel methods to assess similarity of the data generated for each decision node, either via a pairwise analysis conducted property-by-property, or by assessing multiple decision nodes simultaneously via a multidimensional analysis. For the pairwise comparison conducted property-by-property we included in this White paper: The x-fold, Bayesian and Arsinh-OWA distance algorithms performed comparably in the scoring of similarity between NF pairs. The Euclidean distance was also useful, but only with proper data transformation. The x-fold method does not standardize data, and thus produces skewed histograms, but has the advantage that it can be implemented without programming knowhow. A range of multidimensional evaluations, using for example dendrogram clustering approaches, were also investigated. Multidimensional distance metrics were demonstrated to be difficult to use in a regulatory context, but from a scientific perspective were found to offer unexpected insights into the overall similarity of very different materials. In conclusion, for regulatory purposes, a property-by-property evaluation of the data matrix is recommended to substantiate grouping, while the multidimensional approaches are considered to be tools of discovery rather than regulatory methods.
Collapse
Affiliation(s)
| | - Eric Bleeker
- National Institute for Public Health and the Environment (RIVM), Bilthoven, the Netherlands
| | - Richard Cross
- UKRI Centre for Ecology and Hydrology, MacLean Building, Benson Lane, Wallingford OX10 8BB, UK
| | - Andrea Haase
- German Federal Institute for Risk Assessment (BfR), Department of Chemical and Product Safety, Berlin, Germany
| | - Gemma Janer
- LEITAT Technological Center, Barcelona, Spain
| | - Willie Peijnenburg
- National Institute for Public Health and the Environment (RIVM), Bilthoven, the Netherlands; Institute of Environmental Sciences (CML), Leiden University, Leiden, the Netherlands
| | - Mario Pink
- German Federal Institute for Risk Assessment (BfR), Department of Chemical and Product Safety, Berlin, Germany
| | - Hubert Rauscher
- European Commission, Joint Research Centre (JRC), Ispra, Italy
| | - Claus Svendsen
- UKRI Centre for Ecology and Hydrology, MacLean Building, Benson Lane, Wallingford OX10 8BB, UK
| | - Georgia Tsiliki
- Athena-Research and Innovation Center in Information, Communication and Knowledge Technologies, Marousi, Greece
| | | | | | - Vicki Stone
- NanoSafety Research Group, Heriot-Watt University, Riccarton, Edinburgh EH14 4AS, UK
| | - Wendel Wohlleben
- BASF SE, Dept. Material Physics and Dept Experimental Toxicology & Ecology, Ludwigshafen, Germany.
| |
Collapse
|
7
|
Basei G, Zabeo A, Rasmussen K, Tsiliki G, Hristozov D. A Weight of Evidence approach to classify nanomaterials according to the EU Classification, Labelling and Packaging Regulation criteria. NANOIMPACT 2021; 24:100359. [PMID: 35559818 DOI: 10.1016/j.impact.2021.100359] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/14/2021] [Revised: 09/13/2021] [Accepted: 09/30/2021] [Indexed: 06/15/2023]
Abstract
In the context of the European Union (EU) Horizon 2020 GRACIOUS project (Grouping, Read-Across, Characterisation and classification framework for regulatory risk assessment of manufactured nanomaterials and Safer design of nano-enabled products), we proposed a quantitative Weight of Evidence (WoE) approach for hazard classification of nanomaterials (NMs). This approach is based on the requirements of the European Regulation on Classification, Labelling and Packaging of Substances and Mixtures (the CLP regulation), which implements the United Nations' Globally Harmonized System of Classification and Labelling of Chemicals (UN GHS) in the European Union. The goal of this WoE methodology is to facilitate classification of NMs according to CLP criteria, following the decision trees defined in ECHA's CLP regulatory guidance. In the WoE, results from heterogeneous studies are weighted according to data quality and completeness criteria, integrated, and then evaluated by expert judgment to obtain a hazard classification, resulting in a coherent and justifiable methodology. Moreover, the probabilistic nature of the proposed approach enables highlighting the uncertainty in the analysis. The proposed methodology involves the following stages: (1) collection of data for different NMs related to the endpoint of interest: each study related to each NM is referred as a Line of Evidence (LoE); (2) computation of weighted scores for each LoE: each LoE is weighted by a score calculated based on data quality and completeness criteria defined in the GRACIOUS project; (3) comparison and integration of the weighed LoEs for each NM: A Monte Carlo resampling approach is adopted to quantitatively and probabilistically integrate the weighted evidence; and (4) assignment of each NM to a hazard class: according to the results, each NM is assigned to one of the classes defined by the CLP regulation. Furthermore, to facilitate the integration and the classification of the weighted LoEs, an online R tool was developed. Finally, the approach was tested against an endpoint relevant to CLP (Aquatic Toxicity) using data retrieved from the eNanoMapper database, results obtained were consistent to results in REACH registration dossiers and in recent literature.
Collapse
|
8
|
Keller JG, Persson M, Müller P, Ma-Hock L, Werle K, Arts J, Landsiedel R, Wohlleben W. Variation in dissolution behavior among different nanoforms and its implication for grouping approaches in inhalation toxicity. NANOIMPACT 2021; 23:100341. [PMID: 35559842 DOI: 10.1016/j.impact.2021.100341] [Citation(s) in RCA: 20] [Impact Index Per Article: 6.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/12/2021] [Revised: 06/17/2021] [Accepted: 07/06/2021] [Indexed: 06/15/2023]
Abstract
Different nanoforms (NF) of the same substance each need to be registered under REACH, but similarities in physiological interaction -among them biodissolution- can justify read-across within a group of NFs, thereby reducing the need to perform animal studies. Here we focused on the endpoint of inhalation toxicity and explored how differences in physical parameters of 17 NFs of silica, and organic and inorganic pigments impact dissolution rates, half-times, and transformation under both pH 7.4 lung lining conditions and pH 4.5 lysosomal conditions. We benchmarked our observations against well-known TiO2, BaSO4 and ZnO nanomaterials, representing very slow, partial and quick dissolution respectively. By automated image evaluation, structural transformations were observed for dissolution rates in the order of 0.1 to 10 ng/cm2/h, but did not provide additional decision criteria on the similarity of NFs. Dissolution half-times spanned nearly five orders of magnitude, mostly dictated by the substance and simulant fluid, but modulated up to ten-fold by the subtle differences between NFs. Physiological time scales and benchmark materials help to frame the biologically relevant range, proposed as 1 h to 1 y. NFs of ZnO, Ag, SiO2, BaSO4 were in this range. We proposed numerical rules of pairwise similarity within a group, of which the worst case NF would be further assessed by in vivo inhalation studies. These rules divided the colloidal silica NFs into two separate candidate groups, one with Al-doping, one without. Shape or silane surface treatment were less important. The dissolution halftimes of many organic and inorganic pigment NFs were longer than the biologically relevant range, such that dissolution behavior is not an obstacle for their groupings.
Collapse
Affiliation(s)
- Johannes G Keller
- BASF SE, Dept. Experimental Toxicology and Ecology, Dept. Material Physics, 67056 Ludwigshafen, Germany.
| | - Michael Persson
- Nouryon Pulp and Performance Chemicals AB, S-445 80 Bohus, Sweden.
| | - Philipp Müller
- BASF SE, Dept. Experimental Toxicology and Ecology, Dept. Material Physics, 67056 Ludwigshafen, Germany.
| | - Lan Ma-Hock
- BASF SE, Dept. Experimental Toxicology and Ecology, Dept. Material Physics, 67056 Ludwigshafen, Germany.
| | - Kai Werle
- BASF SE, Dept. Experimental Toxicology and Ecology, Dept. Material Physics, 67056 Ludwigshafen, Germany.
| | - Josje Arts
- Nouryon Pulp and Performance Chemicals AB, S-445 80 Bohus, Sweden.
| | - Robert Landsiedel
- BASF SE, Dept. Experimental Toxicology and Ecology, Dept. Material Physics, 67056 Ludwigshafen, Germany.
| | - Wendel Wohlleben
- BASF SE, Dept. Experimental Toxicology and Ecology, Dept. Material Physics, 67056 Ludwigshafen, Germany.
| |
Collapse
|
9
|
Boots TE, Kogel AM, Drew NM, Kuempel ED. Utilizing literature-based rodent toxicology data to derive potency estimates for quantitative risk assessment. Nanotoxicology 2021; 15:740-760. [PMID: 34087078 DOI: 10.1080/17435390.2021.1918278] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
Abstract
Evaluating the potential occupational health risk of engineered nanomaterials is an ongoing need. The objective of this meta-analysis, which consisted of 36 studies containing 86 materials, was to assess the availability of published in vivo rodent pulmonary toxicity data for a variety of nanoscale and microscale materials and to derive potency estimates via benchmark dose modeling. Additionally, the potency estimates based on particle mass lung dose associated with acute pulmonary inflammation were used to group materials based on toxicity. The commonalities among the physicochemical properties of the materials in each group were also explored. This exploration found that a material's potency tended to be associated primarily with the material class based on chemical composition and form (e.g. carbon nanotubes, TiO2, ZnO) rather than with particular physicochemical properties. Limitations in the data available precluded a more extensive analysis of these associations. Issues such as data reporting and appropriate experimental design for use in quantitative risk assessment are the main reasons publications were excluded from these analyses and are discussed.
Collapse
Affiliation(s)
- Theresa E Boots
- Health Effect Laboratory Division (HELD), BioAnalytics Branch (BB), National Institute for Occupational Safety and Health (NIOSH), Morgantown, WV, USA
| | - Alyssa M Kogel
- Formerly Oak Ridge Associated Universities/Oak Ridge Institute for Science and Education, at NIOSH, Oak Ridge, TN, USA
| | - Nathan M Drew
- Division of Science Integration (DSI), Emerging Technologies Branch (ETB), NIOSH, Cincinnati, OH, USA
| | - Eileen D Kuempel
- Division of Science Integration (DSI), Emerging Technologies Branch (ETB), NIOSH, Cincinnati, OH, USA
| |
Collapse
|
10
|
Janer G, Landsiedel R, Wohlleben W. Rationale and decision rules behind the ECETOC NanoApp to support registration of sets of similar nanoforms within REACH. Nanotoxicology 2020; 15:145-166. [PMID: 33320695 DOI: 10.1080/17435390.2020.1842933] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/14/2022]
Abstract
New registration requirements for nanomaterials under REACH consider the possibility to form 'sets of similar nanoforms' for a joined human health and environmental hazard, exposure and risk assessment. We developed a tool to create and justify sets of similar nanoforms and to ensure that each of the nanoforms is sufficiently similar to all other nanoforms. The decision logic is following the ECHA guidance in a transparent and evidence-based manner. For each two nanoforms the properties under consideration are compared and corresponding thresholds for maximal differences are proposed. In tier1, similarity is assessed based on intrinsic properties that mostly correspond to those required for nanoform identification under REACH: composition, impurities/additives, size, crystallinity, shape and surface treatment. Moreover, potential differences in the agglomeration/aggregation state resulting from different production processes are considered. If nanoforms were not sufficiently similar based on tier1 criteria, additional data from functional assays are required in tier2. In rare cases, additional short-term in vivo rodent data could be required in a third tier. Data required by tier 2 are triggered by the intrinsic properties in the first tier that did not match the similarity criteria. Most often this will be data on dissolution and surface reactivity followed by in vitro toxicity, dispersion stability, dustiness. Out of several nanoforms given by the user, the tool concludes which nanoforms could be justified to be in the same set and which nanoforms are outside. It defines the boundaries of sets of similar nanoforms and generates a justification for the REACH registration.
Collapse
Affiliation(s)
- Gemma Janer
- Leitat Technological Center, Barcelona, Spain
| | - Robert Landsiedel
- Department of Experimental Toxicology and Ecology, BASF SE, Ludwigshafen am Rhein, Germany
| | - Wendel Wohlleben
- Department of Material Physics and Analytics, BASF SE, Ludwigshafen am Rhein, Germany
| |
Collapse
|
11
|
Halappanavar S, Ede JD, Mahapatra I, Krug HF, Kuempel ED, Lynch I, Vandebriel RJ, Shatkin JA. A methodology for developing key events to advance nanomaterial-relevant adverse outcome pathways to inform risk assessment. Nanotoxicology 2020; 15:289-310. [PMID: 33317378 DOI: 10.1080/17435390.2020.1851419] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/18/2022]
Abstract
Significant advances have been made in the development of Adverse Outcome Pathways (AOPs) over the last decade, mainly focused on the toxicity mechanisms of chemicals. These AOPs, although relevant to manufactured nanomaterials (MNs), do not currently capture the reported roles of size-associated properties of MNs on toxicity. Moreover, some AOs of relevance to airborne exposures to MNs such as lung inflammation and fibrosis shown in animal studies may not be targeted in routine regulatory decision making. The primary objective of the present study was to establish an approach to advance the development of AOPs of relevance to MNs using existing, publicly available, nanotoxicology literature. A systematic methodology was created for curating, organizing and applying the available literature for identifying key events (KEs). Using a case study approach, the study applied the available literature to build the biological plausibility for 'tissue injury', a KE of regulatory relevance to MNs. The results of the analysis reveal the various endpoints, assays and specific biological markers used for assessing and reporting tissue injury. The study elaborates on the limitations and opportunities of the current nanotoxicology literature and provides recommendations for the future reporting of nanotoxicology results that will expedite not only the development of AOPs for MNs but also aid in application of existing data for decision making.
Collapse
Affiliation(s)
- Sabina Halappanavar
- Environmental Health Science and Research Bureau, Health Canada, Ottawa, Canada
| | | | - Indrani Mahapatra
- School of Geography, Earth and Environmental Sciences, University of Birmingham, Birmingham, UK
| | - Harald F Krug
- Retired International Research Cooperation Manager, Empa - Swiss Federal Laboratories for Science and Materials Technology, St. Gallen, Switzerland.,NanoCASE GmbH, Engelburg, Switzerland
| | - Eileen D Kuempel
- National Institute for Occupational Safety and Health, Nanotechnology Research Center, Cincinnati, OH, USA
| | - Iseult Lynch
- School of Geography, Earth and Environmental Sciences, University of Birmingham, Birmingham, UK
| | - Rob J Vandebriel
- National Institute for Public Health and the Environment (RIVM), Bilthoven, Netherlands
| | | |
Collapse
|
12
|
Fraser K, Kodali V, Yanamala N, Birch ME, Cena L, Casuccio G, Bunker K, Lersch TL, Evans DE, Stefaniak A, Hammer MA, Kashon ML, Boots T, Eye T, Hubczak J, Friend SA, Dahm M, Schubauer-Berigan MK, Siegrist K, Lowry D, Bauer AK, Sargent LM, Erdely A. Physicochemical characterization and genotoxicity of the broad class of carbon nanotubes and nanofibers used or produced in U.S. facilities. Part Fibre Toxicol 2020; 17:62. [PMID: 33287860 PMCID: PMC7720492 DOI: 10.1186/s12989-020-00392-w] [Citation(s) in RCA: 30] [Impact Index Per Article: 7.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2020] [Accepted: 11/18/2020] [Indexed: 12/23/2022] Open
Abstract
BACKGROUND Carbon nanotubes and nanofibers (CNT/F) have known toxicity but simultaneous comparative studies of the broad material class, especially those with a larger diameter, with computational analyses linking toxicity to their fundamental material characteristics was lacking. It was unclear if all CNT/F confer similar toxicity, in particular, genotoxicity. Nine CNT/F (MW #1-7 and CNF #1-2), commonly found in exposure assessment studies of U.S. facilities, were evaluated with reported diameters ranging from 6 to 150 nm. All materials were extensively characterized to include distributions of physical dimensions and prevalence of bundled agglomerates. Human bronchial epithelial cells were exposed to the nine CNT/F (0-24 μg/ml) to determine cell viability, inflammation, cellular oxidative stress, micronuclei formation, and DNA double-strand breakage. Computational modeling was used to understand various permutations of physicochemical characteristics and toxicity outcomes. RESULTS Analyses of the CNT/F physicochemical characteristics illustrate that using detailed distributions of physical dimensions provided a more consistent grouping of CNT/F compared to using particle dimension means alone. In fact, analysis of binning of nominal tube physical dimensions alone produced a similar grouping as all characterization parameters together. All materials induced epithelial cell toxicity and micronuclei formation within the dose range tested. Cellular oxidative stress, DNA double strand breaks, and micronuclei formation consistently clustered together and with larger physical CNT/F dimensions and agglomerate characteristics but were distinct from inflammatory protein changes. Larger nominal tube diameters, greater lengths, and bundled agglomerate characteristics were associated with greater severity of effect. The portion of tubes with greater nominal length and larger diameters within a sample was not the majority in number, meaning a smaller percentage of tubes with these characteristics was sufficient to increase toxicity. Many of the traditional physicochemical characteristics including surface area, density, impurities, and dustiness did not cluster with the toxicity outcomes. CONCLUSION Distributions of physical dimensions provided more consistent grouping of CNT/F with respect to toxicity outcomes compared to means only. All CNT/F induced some level of genotoxicity in human epithelial cells. The severity of toxicity was dependent on the sample containing a proportion of tubes with greater nominal lengths and diameters.
Collapse
Affiliation(s)
- Kelly Fraser
- Health Effect Laboratory Division, National Institute for Occupational Safety and Health, 1095 Willowdale Rd, MS-2015, Morgantown, WV 26505-2888 USA
- West Virginia University, Morgantown, WV USA
| | - Vamsi Kodali
- Health Effect Laboratory Division, National Institute for Occupational Safety and Health, 1095 Willowdale Rd, MS-2015, Morgantown, WV 26505-2888 USA
- West Virginia University, Morgantown, WV USA
| | - Naveena Yanamala
- Health Effect Laboratory Division, National Institute for Occupational Safety and Health, 1095 Willowdale Rd, MS-2015, Morgantown, WV 26505-2888 USA
- West Virginia University, Morgantown, WV USA
| | - M. Eileen Birch
- Health Effects Laboratory Division, National Institute for Occupational Safety and Health, Cincinnati, OH USA
| | | | | | | | | | - Douglas E. Evans
- Health Effects Laboratory Division, National Institute for Occupational Safety and Health, Cincinnati, OH USA
| | - Aleksandr Stefaniak
- Repiratory Health Division, National Institute for Occupational Safety and Health, Morgantown, WV USA
| | - Mary Ann Hammer
- Health Effect Laboratory Division, National Institute for Occupational Safety and Health, 1095 Willowdale Rd, MS-2015, Morgantown, WV 26505-2888 USA
| | - Michael L. Kashon
- Health Effect Laboratory Division, National Institute for Occupational Safety and Health, 1095 Willowdale Rd, MS-2015, Morgantown, WV 26505-2888 USA
| | - Theresa Boots
- Health Effect Laboratory Division, National Institute for Occupational Safety and Health, 1095 Willowdale Rd, MS-2015, Morgantown, WV 26505-2888 USA
| | - Tracy Eye
- Health Effect Laboratory Division, National Institute for Occupational Safety and Health, 1095 Willowdale Rd, MS-2015, Morgantown, WV 26505-2888 USA
| | - John Hubczak
- Health Effect Laboratory Division, National Institute for Occupational Safety and Health, 1095 Willowdale Rd, MS-2015, Morgantown, WV 26505-2888 USA
- West Virginia University, Morgantown, WV USA
| | - Sherri A. Friend
- Health Effect Laboratory Division, National Institute for Occupational Safety and Health, 1095 Willowdale Rd, MS-2015, Morgantown, WV 26505-2888 USA
| | - Matthew Dahm
- Division of Field Studies Evaluation, National Institute for Occupational Safety and Health, Cincinnati, OH USA
| | - Mary K. Schubauer-Berigan
- Division of Field Studies Evaluation, National Institute for Occupational Safety and Health, Cincinnati, OH USA
- International Agency for Research on Cancer, Lyon, France
| | - Katelyn Siegrist
- Health Effect Laboratory Division, National Institute for Occupational Safety and Health, 1095 Willowdale Rd, MS-2015, Morgantown, WV 26505-2888 USA
| | - David Lowry
- Health Effect Laboratory Division, National Institute for Occupational Safety and Health, 1095 Willowdale Rd, MS-2015, Morgantown, WV 26505-2888 USA
| | - Alison K. Bauer
- Department of Environmental and Occupational Health, University of Colorado Anschutz Medical Campus, Aurora, CO USA
| | - Linda M. Sargent
- Health Effect Laboratory Division, National Institute for Occupational Safety and Health, 1095 Willowdale Rd, MS-2015, Morgantown, WV 26505-2888 USA
| | - Aaron Erdely
- Health Effect Laboratory Division, National Institute for Occupational Safety and Health, 1095 Willowdale Rd, MS-2015, Morgantown, WV 26505-2888 USA
- West Virginia University, Morgantown, WV USA
| |
Collapse
|
13
|
Keller JG, Graham UM, Koltermann-Jülly J, Gelein R, Ma-Hock L, Landsiedel R, Wiemann M, Oberdörster G, Elder A, Wohlleben W. Predicting dissolution and transformation of inhaled nanoparticles in the lung using abiotic flow cells: The case of barium sulfate. Sci Rep 2020; 10:458. [PMID: 31949204 PMCID: PMC6965653 DOI: 10.1038/s41598-019-56872-3] [Citation(s) in RCA: 24] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2019] [Accepted: 12/17/2019] [Indexed: 02/04/2023] Open
Abstract
Barium sulfate (BaSO4) was considered to be poorly-soluble and of low toxicity, but BaSO4 NM-220 showed a surprisingly short retention after intratracheal instillation in rat lungs, and incorporation of Ba within the bones. Here we show that static abiotic dissolution cannot rationalize this result, whereas two dynamic abiotic dissolution systems (one flow-through and one flow-by) indicated 50% dissolution after 5 to 6 days at non-saturating conditions regardless of flow orientation, which is close to the in vivo half-time of 9.6 days. Non-equilibrium conditions were thus essential to simulate in vivo biodissolution. Instead of shrinking from 32 nm to 23 nm (to match the mass loss to ions), TEM scans of particles retrieved from flow-cells showed an increase to 40 nm. Such transformation suggested either material transport through interfacial contact or Ostwald ripening at super-saturating conditions and was also observed in vivo inside macrophages by high-resolution TEM following 12 months inhalation exposure. The abiotic flow cells thus adequately predicted the overall pulmonary biopersistence of the particles that was mediated by non-equilibrium dissolution and recrystallization. The present methodology for dissolution and transformation fills a high priority gap in nanomaterial hazard assessment and is proposed for the implementation of grouping and read-across by dissolution rates.
Collapse
Affiliation(s)
- Johannes G Keller
- Department Experimental Toxicology and Ecology and Department Material Physics, BASF SE, 67056, Ludwigshafen, Germany.,Institute of Pharmacy, Faculty of Biology, Chemistry & Pharmacy, Freie Universität Berlin, 14195, Berlin, Germany
| | - Uschi M Graham
- National Institute of Occupational Safety and Health, Cincinnati, Ohio, 45226, USA
| | - Johanna Koltermann-Jülly
- Department Experimental Toxicology and Ecology and Department Material Physics, BASF SE, 67056, Ludwigshafen, Germany.,Biopharmaceutics and Pharmaceutical Technology, Saarland University, 66123, Saarbrücken, Germany
| | - Robert Gelein
- University of Rochester Medical Center, Rochester, New York, USA
| | - Lan Ma-Hock
- Department Experimental Toxicology and Ecology and Department Material Physics, BASF SE, 67056, Ludwigshafen, Germany
| | - Robert Landsiedel
- Department Experimental Toxicology and Ecology and Department Material Physics, BASF SE, 67056, Ludwigshafen, Germany
| | - Martin Wiemann
- IBE R&D Institute for Lung Health gGmbH, Mendelstr. 11, 48149, Münster, Germany
| | | | - Alison Elder
- University of Rochester Medical Center, Rochester, New York, USA.
| | - Wendel Wohlleben
- Department Experimental Toxicology and Ecology and Department Material Physics, BASF SE, 67056, Ludwigshafen, Germany.
| |
Collapse
|
14
|
Wohlleben W, Hellack B, Nickel C, Herrchen M, Hund-Rinke K, Kettler K, Riebeling C, Haase A, Funk B, Kühnel D, Göhler D, Stintz M, Schumacher C, Wiemann M, Keller J, Landsiedel R, Broßell D, Pitzko S, Kuhlbusch TAJ. The nanoGRAVUR framework to group (nano)materials for their occupational, consumer, environmental risks based on a harmonized set of material properties, applied to 34 case studies. NANOSCALE 2019; 11:17637-17654. [PMID: 31539006 DOI: 10.1039/c9nr03306h] [Citation(s) in RCA: 29] [Impact Index Per Article: 5.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
The project nanoGRAVUR (BMBF, 2015-2018) developed a framework for grouping of nanomaterials. Different groups may result for each of the three distinct perspectives of occupational, consumer and environmental safety. The properties, methods and descriptors are harmonised between the three perspectives and are based on: Tier 1 intrinsic physico-chemical properties (what they are) or GHS classification of the non-nano-form (human tox, ecotox, physical hazards); Tier 2 extrinsic physico-chemical properties, release from nano-enabled products, in vitro assays with cells (where they go; what they do); Tier 3 case-specific tests, potentially in vivo studies to substantiate the similarity within groups or application-specific exposure testing. Amongst all properties, dissolution and transformation are least modulated by different nanoforms within one substance, whereas dustiness, dispersion stability, abiotic and especially in vitro surface reactivity vary more often between different nanoforms. The methods developed or selected by nanoGRAVUR fill several gaps highlighted in the ProSafe reviews, and are useful to implement (i) the concept of nanoforms of the European Chemicals Agency (ECHA) and (ii) the concept of discrete forms of the United States Environmental Protection Agency (EPA). One cannot assess the significance of a dissimilarity, if the dynamic range of that property is unknown. Benchmark materials span dynamic ranges that enable us to establish bands, often with order-of-magnitude ranges. In 34 case studies we observed high biological similarity within each substance when we compared different (nano)forms of SiO2, BaSO4, kaolin, CeO2, ZnO, organic pigments, especially when we compared forms that are all untreated on the surface. In contrast, different Fe2O3 or TiO2 (nano)forms differ more significantly. The same nanoforms were also integrated in nano-enabled products (NEPs) for automotive coatings, clinker-reduced cements, cosmetic sunscreen, and lightweight polymers.
Collapse
Affiliation(s)
- Wendel Wohlleben
- BASF SE, Dept. of Material Physics and Dept. of Experimental Toxicology and Ecology, 67056 Ludwigshafen, Germany
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
Collapse
|
15
|
Stueckle TA, Roberts JR. Perspective on Current Alternatives in Nanotoxicology Research. ACTA ACUST UNITED AC 2019. [DOI: 10.1089/aivt.2019.29020.jrr] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/06/2023]
Affiliation(s)
- Todd A. Stueckle
- Health Effects Laboratory Division, National Institute for Occupational Safety and Health, Morgantown, West Virginia
| | - Jenny R. Roberts
- Health Effects Laboratory Division, National Institute for Occupational Safety and Health, Morgantown, West Virginia
| |
Collapse
|
16
|
Lamon L, Aschberger K, Asturiol D, Richarz A, Worth A. Grouping of nanomaterials to read-across hazard endpoints: a review. Nanotoxicology 2018; 13:100-118. [DOI: 10.1080/17435390.2018.1506060] [Citation(s) in RCA: 37] [Impact Index Per Article: 6.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/07/2023]
Affiliation(s)
- L. Lamon
- European Commission, Joint Research Centre, Ispra, Italy
| | - K. Aschberger
- European Commission, Joint Research Centre, Ispra, Italy
| | - D. Asturiol
- European Commission, Joint Research Centre, Ispra, Italy
| | - A. Richarz
- European Commission, Joint Research Centre, Ispra, Italy
| | - A. Worth
- European Commission, Joint Research Centre, Ispra, Italy
| |
Collapse
|
17
|
Schulte PA, Kuempel ED, Drew NM. Characterizing risk assessments for the development of occupational exposure limits for engineered nanomaterials. Regul Toxicol Pharmacol 2018; 95:207-219. [PMID: 29574195 PMCID: PMC6075708 DOI: 10.1016/j.yrtph.2018.03.018] [Citation(s) in RCA: 29] [Impact Index Per Article: 4.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2017] [Revised: 03/05/2018] [Accepted: 03/20/2018] [Indexed: 12/16/2022]
Abstract
The commercialization of engineered nanomaterials (ENMs) began in the early 2000's. Since then the number of commercial products and the number of workers potentially exposed to ENMs is growing, as is the need to evaluate and manage the potential health risks. Occupational exposure limits (OELs) have been developed for some of the first generation of ENMs. These OELs have been based on risk assessments that progressed from qualitative to quantitative as nanotoxicology data became available. In this paper, that progression is characterized. It traces OEL development through the qualitative approach of general groups of ENMs based primarily on read-across with other materials to quantitative risk assessments for nanoscale particles including titanium dioxide, carbon nanotubes and nanofibers, silver nanoparticles, and cellulose nanocrystals. These represent prototypic approaches to risk assessment and OEL development for ENMs. Such substance-by-substance efforts are not practical given the insufficient data for many ENMs that are currently being used or potentially entering commerce. Consequently, categorical approaches are emerging to group and rank ENMs by hazard and potential health risk. The strengths and limitations of these approaches are described, and future derivations and research needs are discussed. Critical needs in moving forward with understanding the health effects of the numerous EMNs include more standardized and accessible quantitative data on the toxicity and physicochemical properties of ENMs.
Collapse
Affiliation(s)
- P A Schulte
- National Institute for Occupational Safety and Health, Centers for Disease Control and Prevention, United States.
| | - E D Kuempel
- National Institute for Occupational Safety and Health, Centers for Disease Control and Prevention, United States
| | - N M Drew
- National Institute for Occupational Safety and Health, Centers for Disease Control and Prevention, United States
| |
Collapse
|