Enhancing credibility of chemical safety studies: emerging consensus on key assessment criteria

Toxicity testing and endocrine disrupting chemicals

Regulatory agencies around the world depend on standardized testing approaches to evaluate environmental chemicals for endocrine disrupting properties. The US Environmental Protection Agency (EPA) has developed a two-tiered testing approach within its Endocrine Disruptor Screening Program (EDSP). The eleven Tier 1 and three Tier 2 EDSP assays can be used to identify chemicals that act as agonists or antagonists of estrogen receptor, androgen receptor, or thyroid hormone receptor, or chemicals that interfere with steroidogenesis. Additional assays have been developed in the context of Tox21, and others have been validated by the OECD. In spite of the availability of validated toxicity tests, problems have been identified with the approaches and methods used to identify endocrine disrupting chemicals (EDCs). This chapter will provide an overview of several of these issues including: (1) The way an EDC is defined by an agency impacts whether a specific test can be used to determine if a chemical is an EDC. This is especially important when considering which assays examine outcomes that are considered "adverse effects." (2) Some assumptions about the validated studies used to identify EDCs may not be true (e.g., their reproducibility has been questioned). (3) Many of the validated assays are less sensitive than other methods that have not yet been validated. Ultimately, these and other problems contribute to the current landscape, where testing approaches have failed to protect the public from known EDCs. The chapter concludes with a review of approaches that have been taken to improve current guideline studies.

Facilitation of risk assessment with evidence-based methods - A framework for use of systematic mapping and systematic reviews in determining hazard, developing toxicity values, and characterizing uncertainty

Systematic review tools and approaches developed for clinical medicine are often difficult to apply "off the shelf" in order to meet the needs of chemical risk assessments. To address such, we propose an approach that can be used by practitioners for using evidence-based methods to facilitate the risk assessment process. The framework builds on and combines efforts conducted to date by a number of agencies and researchers; the novelty is in combining these efforts with a practical understanding of risk assessment, and translating such into a 'step-by-step' guide. The approach relies on three key components: problem formulation, systematic evidence mapping, and systematic review, applied using a stepwise approach. Unique to this framework is the consideration of exposure in selecting, prioritizing, and evaluating data (e.g., dose-relevance, routes of exposure, etc.). Using the proposed step-by-step process, critical appraisal of individual studies (e.g., formal and structured assessment of both relevance and reliability) and integration efforts are considered in context of specified risk assessment objectives (e.g., mode of action, dose-response) as well as chemical-specific considerations. The resulting framework provides a logical approach of how evidence-based methods can be used to facilitate risk assessment, and elevates the use of systematic methods beyond hazard identification to directly facilitating transparent and objective selection of candidate studies and/or datasets used to quantitatively characterize risk, and to better use the underlying process to inform the approaches used to develop toxicity values.

Safety of Safety Evaluation of Pesticides: developmental neurotoxicity of chlorpyrifos and chlorpyrifos-methyl

Authorization of pesticides for market release requires toxicity testing on animals, typically performed by test laboratories on contract with the pesticide producer. The latter provides the results and summary to the regulatory authorities. For the commonly used pesticide chlorpyrifos, an industry-funded toxicity study concludes that no selective effects on neurodevelopment occur even at high exposures. In contrast, the evidence from independent studies points to adverse effects of current exposures on cognitive development in children. We reviewed the industry-funded developmental neurotoxicity test data on chlorpyrifos and the related substance chlorpyrifos-methyl. We noted treatment-related changes in a brain dimension measure for chlorpyrifos at all dose levels tested, although not been reported in the original test summary. We further found issues which inappropriately decrease the ability of the studies to reveal true effects, including a dosage regimen that resulted in too low exposure of the nursing pups for chlorpyrifos and possibly for chlorpyrifos-methyl, and a failure to detect any neurobehavioral effects of lead nitrate used as positive control in the chlorpyrifos study. Our observations thus suggest that conclusions in test reports submitted by the producer may be misleading. This discrepancy affects the ability of regulatory authorities to perform a valid and safe evaluation of these pesticides. The difference between raw data and conclusions in the test reports indicates a potential existence of bias that would require regulatory attention and possible resolution.

FREE INVENTORY PLATFORM MANAGES CHEMICAL RISKS, ADDRESSES CHEMICAL ACCOUNTABILITY, AND MEASURES COST-EFFECTIVENESS

To develop best practices for laboratory safety and for chemical product and supplies management accountability, the freely-available online platform, Quartzy, was integrated within an interdisciplinary science department at a small Mid-Atlantic liberal-arts college. This was done to ensure the accuracy of purchase records, the appropriate use of storage and handling protocols, and for a continually updated chemical inventory system. Quartzy also facilitated the digital tracking and dispersal of the College's hazardous waste inventory. Since the implementation of the Quartzy platform, the science department achieved significant cost-savings during the procurement of laboratory supplies and equipment, and it developed a sense of ownership towards the common goal of lowering the College's environmental impact as it relates to its managing of laboratory-generated hazardous wastes.

CRED: Criteria for reporting and evaluating ecotoxicity data

Predicted-no-effect concentrations (PNECs) and environmental quality standards (EQSs) are derived in a large number of legal frameworks worldwide. When deriving these safe concentrations, it is necessary to evaluate the reliability and relevance of ecotoxicity studies. Such evaluation is often subject to expert judgment, which may introduce bias and decrease consistency when risk assessors evaluate the same study. The Criteria for Reporting and Evaluating Ecotoxicity Data (CRED) project attempts to address this problem. It aims to improve the reproducibility, transparency, and consistency of reliability and relevance evaluations of aquatic ecotoxicity studies among regulatory frameworks, countries, institutes, and individual assessors. In the present study, the CRED evaluation method is presented. It includes a set of 20 reliability and 13 relevance criteria, accompanied by extensive guidance. Risk assessors who participated in the CRED ring test evaluated the CRED evaluation method to be more accurate, applicable, consistent, and transparent than the often-used Klimisch method. The CRED evaluation method is accompanied by reporting recommendations for aquatic ecotoxicity studies, with 50 specific criteria divided into 6 categories: general information, test design, test substance, test organism, exposure conditions, and statistical design and biological response. An ecotoxicity study in which all important information is reported is more likely to be considered for regulatory use, and proper reporting may also help in the peer-review process.

A path forward in the debate over health impacts of endocrine disrupting chemicals

Several recent publications reflect debate on the issue of "endocrine disrupting chemicals" (EDCs), indicating that two seemingly mutually exclusive perspectives are being articulated separately and independently. Considering this, a group of scientists with expertise in basic science, medicine and risk assessment reviewed the various aspects of the debate to identify the most significant areas of dispute and to propose a path forward. We identified four areas of debate. The first is about the definitions for terms such as "endocrine disrupting chemical", "adverse effects", and "endocrine system". The second is focused on elements of hormone action including "potency", "endpoints", "timing", "dose" and "thresholds". The third addresses the information needed to establish sufficient evidence of harm. Finally, the fourth focuses on the need to develop and the characteristics of transparent, systematic methods to review the EDC literature. Herein we identify areas of general consensus and propose resolutions for these four areas that would allow the field to move beyond the current and, in our opinion, ineffective debate.

Instruments for assessing risk of bias and other methodological criteria of published animal studies: a systematic review

BACKGROUND

Results from animal toxicology studies are critical to evaluating the potential harm from exposure to environmental chemicals or the safety of drugs prior to human testing. However, there is significant debate about how to evaluate the methodology and potential biases of the animal studies. There is no agreed-upon approach, and a systematic evaluation of current best practices is lacking.

OBJECTIVE

We performed a systematic review to identify and evaluate instruments for assessing the risk of bias and/or other methodological criteria of animal studies.

METHOD

We searched Medline (January 1966-November 2011) to identify all relevant articles. We extracted data on risk of bias criteria (e.g., randomization, blinding, allocation concealment) and other study design features included in each assessment instrument.

DISCUSSION

Thirty distinct instruments were identified, with the total number of assessed risk of bias, methodological, and/or reporting criteria ranging from 2 to 25. The most common criteria assessed were randomization (25/30, 83%), investigator blinding (23/30, 77%), and sample size calculation (18/30, 60%). In general, authors failed to empirically justify why these or other criteria were included. Nearly all (28/30, 93%) of the instruments have not been rigorously tested for validity or reliability.

CONCLUSION

Our review highlights a number of risk of bias assessment criteria that have been empirically tested for animal research, including randomization, concealment of allocation, blinding, and accounting for all animals. In addition, there is a need for empirically testing additional methodological criteria and assessing the validity and reliability of a standard risk of bias assessment instrument.

Data disclosure for chemical evaluations

BACKGROUND

Public disclosure of scientific data used by the government to make regulatory decisions for chemicals is a practical step that can enhance public confidence in the scientific basis of such decisions.

OBJECTIVES

We reviewed the U.S. Environmental Protection Agency's (EPA) current practices regarding disclosure of data underlying regulatory and policy decisions involving chemicals, including pesticides. We sought to identify additional opportunities for the U.S. EPA to disclose data and, more generally, to promote broad access to data it uses, regardless of origin.

DISCUSSION

We recommend that when the U.S. EPA proposes a regulatory determination or other policy decision that relies on scientific research, it should provide sufficient underlying raw data and information about methods to enable reanalysis and attempts to independently reproduce the work, including the sensitivity of results to alternative analyses. This recommendation applies regardless of who conducted the work. If the U.S. EPA is unable to provide such transparency, it should state whether it had full access to all underlying data and methods. A timely version of submitted data cleared of information about confidential business matters and personal privacy should fully meet the standards of transparency described below, including public access sufficient for others to undertake an independent reanalysis.

CONCLUSION

Reliable chemical evaluation is essential for protecting public health and the environment and for ensuring availability of useful chemicals under appropriate conditions. Permitting qualified researchers to endeavor to independently reproduce the analyses used in regulatory determinations of pesticides and other chemicals would increase confidence in the scientific basis of such determinations.

Taking financial relationships into account when assessing research

Many scientific journals, government agencies, and universities require disclosure of sources of funding and financial interests related to research, such as stock ownership, consulting arrangements with companies, and patents. Although disclosure has become one of the central approaches for responding to financial conflicts of interest (COIs) in research, critics contend that information about financial COIs does not serve as a reliable indicator of research credibility, and therefore, studies should be evaluated solely based on their scientific merits. We argue that, while it is indeed important to evaluate studies on their scientific merits, it is often difficult to detect significant influences of financial relationships that affect research credibility. Moreover, at least five factors can be examined to determine whether financial relationships are likely to enhance, undermine, or have no impact on the credibility of research. These include as follows: whether sponsors, institutions, or researchers have a significant financial stake in the outcome of a study; whether the financial interests of the sponsors, institutions, or researchers coincide with the goal of conducting research that is objective and reliable; whether the sponsor, institution, or researchers have a history of biasing research in order to promote their financial goals; how easy it is to manipulate the research in order to achieve financial goals; and whether oversight mechanisms are in place which are designed to minimize bias. Since these factors vary from case to case, evaluating the impact of financial relationships depends on the circumstances. In some situations, one may decide that the financial relationships significantly undermine the study's credibility; in others, one may decide that they have no impact on credibility or even enhance it.