Accounting Artifacts in High-Throughput Toxicity Assays

From viability to cell death: Claims with insufficient evidence in high-impact cell culture studies

Background

Reliability of preclinical research is of critical concern. Prior studies have demonstrated the low reproducibility of research results and recommend implementing higher standards to improve overall quality and robustness of research. One understudied aspect of this quality issue is the harmony between the research hypotheses and the experimental design in published work.

Methods and findings

In this study we focused on highly cited cell culture studies and investigated whether commonly asserted cell culture claims such as viability, cytotoxicity, proliferation rate, cell death and apoptosis are backed with sufficient experimental evidence or not. We created an open access database containing 280 claims asserted by 103 different high-impact articles as well as the results of this study. Our findings revealed that only 64% of all claims were sufficiently supported by evidence and there were concerning misinterpretations such as considering the results of tetrazolium salt reduction assays as indicators of cell death or apoptosis.

Conclusions

Our analysis revealed a discordance between experimental findings and the way they were presented and discussed in the manuscripts. To improve quality of pre-clinical research, we require clear nomenclature by which different cell culture claims are distinctively categorized; materials and methods sections to be written more meticulously; and cell culture methods to be selected and utilized more carefully. In this paper we recommend a nomenclature for selected cell culture claims as well as a methodology for collecting evidence to support those claims.

Benchmarking Data Sets from PubChem BioAssay Data: Current Scenario and Room for Improvement

Developing realistic data sets for evaluating virtual screening methods is a task that has been tackled by the cheminformatics community for many years. Numerous artificially constructed data collections were developed, such as DUD, DUD-E, or DEKOIS. However, they all suffer from multiple drawbacks, one of which is the absence of experimental results confirming the impotence of presumably inactive molecules, leading to possible false negatives in the ligand sets. In light of this problem, the PubChem BioAssay database, an open-access repository providing the bioactivity information of compounds that were already tested on a biological target, is now a recommended source for data set construction. Nevertheless, there exist several issues with the use of such data that need to be properly addressed. In this article, an overview of benchmarking data collections built upon experimental PubChem BioAssay input is provided, along with a thorough discussion of noteworthy issues that one must consider during the design of new ligand sets from this database. The points raised in this review are expected to guide future developments in this regard, in hopes of offering better evaluation tools for novel in silico screening procedures.

Identifying Compounds with Genotoxicity Potential Using Tox21 High-Throughput Screening Assays

Genotoxicity is a critical component of a comprehensive toxicological profile. The Tox21 Program used five quantitative high-throughput screening (qHTS) assays measuring some aspect of DNA damage/repair to provide information on the genotoxic potential of over 10 000 compounds. Included were assays detecting activation of p53, increases in the DNA repair protein ATAD5, phosphorylation of H2AX, and enhanced cytotoxicity in DT40 cells deficient in DNA-repair proteins REV3 or KU70/RAD54. Each assay measures a distinct component of the DNA damage response signaling network; >70% of active compounds were detected in only one of the five assays. When qHTS results were compared with results from three standard genotoxicity assays (bacterial mutation, in vitro chromosomal aberration, and in vivo micronucleus), a maximum of 40% of known, direct-acting genotoxicants were active in one or more of the qHTS genotoxicity assays, indicating low sensitivity. This suggests that these qHTS assays cannot in their current form be used to replace traditional genotoxicity assays. However, despite the low sensitivity, ranking chemicals by potency of response in the qHTS assays revealed an enrichment for genotoxicants up to 12-fold compared with random selection, when allowing a 1% false positive rate. This finding indicates these qHTS assays can be used to prioritize chemicals for further investigation, allowing resources to focus on compounds most likely to induce genotoxic effects. To refine this prioritization process, models for predicting the genotoxicity potential of chemicals that were active in Tox21 genotoxicity assays were constructed using all Tox21 assay data, yielding a prediction accuracy up to 0.83. Data from qHTS assays related to stress-response pathway signaling (including genotoxicity) were the most informative for model construction. By using the results from qHTS genotoxicity assays, predictions from models based on qHTS data, and predictions from commercial bacterial mutagenicity QSAR models, we prioritized Tox21 chemicals for genotoxicity characterization.

Application of Benchmark Concentration (BMC) Analysis on Zebrafish Data: A New Perspective for Quantifying Toxicity in Alternative Animal Models

Over the past decade, the zebrafish is increasingly being used as a model to screen for chemical-mediated toxicities including developmental toxicity (DT) and neurotoxicity (NT). One of the major challenges is lack of harmonization in data analysis approaches, thereby posing difficulty in comparing findings across laboratories. To address this, we sought to establish a unified data analysis strategy for both DT and NT data, by adopting the benchmark concentration (BMC) analysis. There are two critical aspects in the BMC analysis: having a toxicity endpoint amenable for BMC and selecting a proper benchmark response (BMR) for the endpoint. For the former, in addition to the typical endpoints in NT assay (eg, hyper/hypo- response quantified by distance moved), we also used endpoints that assess the differences in movement patterns between chemical-treated embryos and control embryos. For the latter, we standardized the selection of BMR, which is analogous to minimum activity threshold, based on intrinsic response variations in the endpoint. When comparing our BMC results with a traditionally used LOAEL method (lowest-observed-adverse-effect level), we found high active compound concordance (100% for DT vs 74% for NT); generally, the BMC was more sensitive than LOAEL (no. of BMC more sensitive/no. of concordant active compounds, 43/50 for DT vs 16/26 for NT). Using the BMC with standardized toxicity endpoints and an appropriate BMR, we may now have a unified data-analysis approach to comparing results across different zebrafish datasets, for a better understanding of strengths and challenges when using the zebrafish as a screening tool.

Identifying Attributes That Influence In Vitro-to-In Vivo Concordance by Comparing In Vitro Tox21 Bioactivity Versus In Vivo DrugMatrix Transcriptomic Responses Across 130 Chemicals

Recent efforts aimed at integrating in vitro high-throughput screening (HTS) data into chemical toxicity assessments are necessitating increased understanding of concordance between chemical-induced responses observed in vitro versus in vivo. This investigation set out to (1) measure concordance between in vitro HTS data and transcriptomic responses observed in vivo, focusing on the liver, and (2) identify attributes that can influence concordance. Signal response profiles from 130 substances were compared between in vitro data produced through Tox21 and liver transcriptomic data through DrugMatrix, collected from rats exposed to a chemical for ≤5 days. A global in vitro-to-in vivo comparative analysis based on pathway-level responses resulted in an overall average percent agreement of 79%, ranging on a per-chemical basis between 41% and 100%. Whereas concordance amongst inactive chemicals was high (89%), concordance amongst chemicals showing in vitro activity was only 13%, suggesting that follow-up in vivo and/or orthogonal in vitro assays would improve interpretations of in vitro activity. Attributes identified to influence concordance included experimental design attributes (eg, cell type), target pathways, and physicochemical properties (eg, logP). The attribute that most consistently increased concordance was dose applicability, evaluated by filtering for experimental doses administered to rats that were within 10-fold of those related to likely bioactivity, derived using Tox21 data and high-throughput toxicokinetic modeling. Together, findings suggest that in vitro screening approaches to predict in vivo toxicity are viable particularly when certain attributes are considered, including whether activity versus inactivity is observed, experimental design, chemical properties, and dose applicability.

Nonionic Ethoxylated Surfactants Induce Adipogenesis in 3T3-L1 Cells

Recent studies have demonstrated that a number of environmental contaminants can act as metabolic disruptors and modulate metabolic function both in vitro and in vivo. 3T3-L1 mouse preadipocytes are commonly utilized to assess perturbations to adipogenesis, providing insight into environmental contaminants that may impact in vivo metabolic health. This study sought to assess whether various alkylphenol ethoxylates and alcohol ethoxylates (APEOs and AEOs, respectively), ubiquitous contaminants used in common household products, could disrupt metabolic health. 3T3-L1 cells were exposed to increasing concentrations of individual ethoxylated surfactants and base hydrophobes, and assessed for triglyceride accumulation (relative to a rosiglitazone-induced maximum response) and preadipocyte proliferation (relative to a differentiated vehicle control). We report herein that nonionic APEOs and AEOs promoted triglyceride accumulation and/or preadipocyte proliferation in 3T3-L1 cells at concentrations from 0.1 to 10 μM. Activity appeared to be an effect of the polyethoxylate chain length, as the alkylphenol/alcohol hydrophobes exhibited minimal or no adipogenic activity. In addition, nonylphenol ethoxylates (NPEO) of various ethoxylate chain lengths exhibited biphasic adipogenic activity, with increasing triglyceride accumulation and preadipocyte proliferation from NPEO (0, average ethoxylate number) through NPEO (4), and then decreasing activities from NPEO (4) through NPEO (20). Our results suggest potential metabolic impacts of these compounds at environmentally relevant concentrations, demonstrating a need to further assess molecular mechanisms and better characterize environmental concentrations of the specific AEOs and APEOs that are inducing the greatest degree of adipogenic activity herein.

An Intuitive Approach for Predicting Potential Human Health Risk with the Tox21 10k Library

In vitro-in vivo extrapolation (IVIVE) analyses translating high-throughput screening (HTS) data to human relevance have been limited. This study represents the first report applying IVIVE approaches and exposure comparisons using the entirety of the Tox21 federal collaboration chemical screening data, incorporating assay response efficacy and quality of concentration-response fits, and providing quantitative anchoring to first address the likelihood of human in vivo interactions with Tox21 compounds. This likelihood was assessed using a maximum blood concentration to in vitro response ratio approach (C_max/AC₅₀), analogous to decision-making methods for clinical drug-drug interactions. Fraction unbound in plasma (f_up) and intrinsic hepatic clearance (CL_int) parameters were estimated in silico and incorporated in a three-compartment toxicokinetic (TK) model to first predict C_max for in vivo corroboration using therapeutic scenarios. Toward lower exposure scenarios, 36 compounds of 3925 unique chemicals with curated activity in the HTS data using high-quality dose-response model fits and ≥40% efficacy gave "possible" human in vivo interaction likelihoods lower than median human exposures predicted in the United States Environmental Protection Agency's ExpoCast program. A publicly available web application has been designed to provide all Tox21-ToxCast dose-likelihood predictions. Overall, this approach provides an intuitive framework to relate in vitro toxicology data rapidly and quantitatively to exposures using either in vitro or in silico derived TK parameters and can be thought of as an important step toward estimating plausible biological interactions in a high-throughput risk-assessment framework.