Hypothesis-Based Weight of Evidence: An Approach to Assessing Causation and its Application to Regulatory Toxicology

Issue analysis: key characteristics approach for identifying endocrine disruptors

For more than a decade, weight of evidence (WoE) evaluations have been the standard method for determining whether a chemical meets the definition of an endocrine disrupting chemical (EDC). WoE methods consider all data pertinent to satisfying the EDC definition and evaluating those data with respect to relevance, reliability, strength, and coherence with established endocrine physiology and pharmacology. A new approach for identifying EDC hazards has been proposed that organizes and evaluates data according to ten so-called "Key Characteristics (KCs) of EDCs". The approach claims to address the lack of a widely accepted, systematic approach for identifying EDC hazards, but completely ignores the WoE literature for EDCs. In contrast to WoE methods, the KC approach fails to apply the consensus definition of EDC and is not amenable to empirical testing or validation, is fungible and ensures inconsistent and unreliable results, ignores principles of hormone action and characteristics of dose-response in endocrine pharmacology and toxicology, lacks a means of distinguishing endocrine-mediated from non-endocrine mediated mechanisms, lacks a means to reach a negative conclusion about a chemical's EDC properties or to distinguish EDCs from non-EDCs, and provides no means for developing a valid consensus among experts nor provides a means of resolving conflicting interpretations of data. Instead of shortcuts like the KC approach, which are prone to bias, error, and arbitrary conclusions, identifying EDCs should rely on WoE evaluations that supply the critical components and scientific rigor lacking in the proposed KCs for EDCs.

Biological plausibility in environmental health systematic reviews: a GRADE concept paper

BACKGROUND

"Biological plausibility" is a concept frequently referred to in environmental and public health when researchers are evaluating how confident they are in the results and inferences of a study or evidence review. Biological plausibility is not, however, a domain of one of the most widely-used approaches for assessing the certainty of evidence (CoE) which underpins the findings of a systematic review, the Grading of Recommendations Assessment, Development and Evaluation (GRADE) CoE Framework. Whether the omission of biological plausibility is a potential limitation of the GRADE CoE Framework is a topic that is regularly discussed, especially in the context of environmental health systematic reviews.

OBJECTIVES

We analyse how the concept of "biological plausibility", as applied in the context of assessing certainty of the evidence that supports the findings of a systematic review, is accommodated under the processes of systematic review and the existing GRADE domains.

RESULTS AND DISCUSSION

We argue that "biological plausibility" is a concept which primarily comes into play when direct evidence about the effects of an exposure on a population of concern (usually humans) is absent, at high risk of bias, is inconsistent, or limited in other ways. In such circumstances, researchers look toward evidence from other study designs in order to draw conclusions. In this respect, we can consider experimental animal and in vitro evidence as "surrogates" for the target populations, exposures, comparators and outcomes of actual interest. Through discussion of 10 examples of experimental surrogates, we propose that the concept of biological plausibility consists of two principal aspects: a "generalisability aspect" and a "mechanistic aspect". The "generalisability aspect" concerns the validity of inferences from experimental models to human scenarios, and asks the same question as does the assessment of external validity or indirectness in systematic reviews. The "mechanistic aspect" concerns certainty in knowledge of biological mechanisms and would inform judgements of indirectness under GRADE, and thus the overall CoE. While both aspects are accommodated under the indirectness domain of the GRADE CoE Framework, further research is needed to determine how to use knowledge of biological mechanisms in the assessment of indirectness of the evidence in systematic reviews.

Biological plausibility in environmental health systematic reviews: a GRADE concept paper

BACKGROUND

"Biological plausibility" is a concept frequently referred to in environmental and public health when researchers are evaluating how confident they are in the results and inferences of a study or evidence review. Biological plausibility is not, however, a domain of one of the most widely-used approaches for assessing the certainty of evidence (CoE) which underpins the findings of a systematic review, the Grading of Recommendations Assessment, Development and Evaluation (GRADE) CoE Framework. Whether the omission of biological plausibility is a potential limitation of the GRADE CoE Framework is a topic that is regularly discussed, especially in the context of environmental health systematic reviews.

OBJECTIVES

We analyse how the concept of "biological plausibility", as applied in the context of assessing certainty of the evidence that supports the findings of a systematic review, is accommodated under the processes of systematic review and the existing GRADE domains.

RESULTS AND DISCUSSION

We argue that "biological plausibility" is a concept which primarily comes into play when direct evidence about the effects of an exposure on a population of concern (usually humans) is absent, at high risk of bias, is inconsistent, or limited in other ways. In such circumstances, researchers look toward evidence from other study designs in order to draw conclusions. In this respect, we can consider experimental animal and in vitro evidence as "surrogates" for the target populations, exposures, comparators and outcomes of actual interest. Through discussion of 10 examples of experimental surrogates, we propose that the concept of biological plausibility consists of two principal aspects: a "generalisability aspect" and a "mechanistic aspect". The "generalisability aspect" concerns the validity of inferences from experimental models to human scenarios, and asks the same question as does the assessment of external validity or indirectness in systematic reviews. The "mechanistic aspect" concerns certainty in knowledge of biological mechanisms and would inform judgements of indirectness under GRADE, and thus the overall CoE. While both aspects are accommodated under the indirectness domain of the GRADE CoE Framework, further research is needed to determine how to use knowledge of biological mechanisms in the assessment of indirectness of the evidence in systematic reviews.

Cancer and occupational exposure to pesticides: an umbrella review

Purpose

The aim was to identify the scope of the epidemiology literature reviewed regarding the risk of cancer as related to occupational exposure to pesticides and to compare regulatory toxicity results where feasible.

Methods

Review studies of breast, lung, prostate, non-Hodgkin lymphoma, and colorectal cancer were identified from the published literature from 2010 to 2020 using a priori inclusion and exclusion criteria. Epidemiology observations were first assessed and then compared against carcinogenicity profiles derived from regulatory toxicology studies.

Results

Several active ingredients were associated with specific cancer but overall, there was neither strong nor consistent epidemiologic data supportive of a positive association between pesticide exposure in occupational settings and cancer. Authors noted common themes related to the heterogeneity of exposure, study design, control for confounders, and the challenge to collect these data reliably and validly with an adequate sample size. Toxicology studies in laboratory animals that assessed carcinogenic potential did not reveal cancer outcomes that were concordant with reported epidemiologic findings.

Conclusions

Farming and pesticides represent diverse exposures that are difficult to quantify in epidemiologic studies. Going forward, investigators will need creative and novel approaches for exposure assessment. Integration of epidemiologic and toxicological studies with attention to biological plausibility, mode of toxicological action and relevance to humans will increase the ability to better assess associations between pesticides and cancer.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00420-020-01638-y.

Systematic Review and Weight of Evidence Are Integral to Ecological and Human Health Assessments: They Need an Integrated Framework

Scientific assessments synthesize the various results of scientific research for policy and decision making. Synthesizing evidence in environmental assessments can involve either or both of 2 systems: systematic review (SR) and weight of evidence (WoE). Systematic review was developed to systematically assemble results of clinical trials to be combined by meta-analysis. Weight-of-evidence approaches have evolved from jurisprudence to make inferences from diverse bodies of evidence in various fields. Our objectives are to describe the similarities and differences between SR and WoE and to suggest how their best practices can be combined into a general framework that is applicable to human health and ecological assessments. Integrating SR and WoE is based on the recognition that 2 processes are required, assembling evidence and making an inference. Systematic review is characterized by methodical literature searching, screening, and data extraction, originally for meta-analysis but now for various inferential methods. Weight of evidence is characterized by systematically relating heterogeneous evidence to considerations appropriate to the inference and making the inference by weighing the evidence. Systematic review enables the unbiased assembly of evidence from literature, but methods for assembling other information must be considered as well. If only 1 type of quantitative study estimates the assessment endpoint, meta-analysis is appropriate for inference. Otherwise, the heterogeneous evidence must be weighed. A framework is presented that integrates best practices into a methodical assembly and weighing of evidence. A glossary of terms for the combined practice and a history of the origins of SR and WoE are provided in Supplemental Data. Integr Environ Assess Manag 2020;16:718-728. Published 2020. This article is a US Government work and is in the public domain in the USA.

Using problem formulation to clarify the meaning of weight of evidence and biological relevance in environmental risk assessments for genetically modified crops

Weight of evidence and biological relevance are important concepts for risk assessment and decision-making over the use of GM crops; however, their meanings are not well defined. We use problem formulation to clarify the definition of these concepts and thereby identify data that are relevant for risk assessment. Problem formulation defines criteria for the acceptability of risk and devises rigorous tests of the hypothesis that the criteria are met. Corroboration or falsification of such hypotheses characterize risk and enable predictable and transparent decisions about whether certain risks from using a particular GM crop are acceptable. Decisions based on a weight of evidence approach use a synthesis of several lines of evidence, whereas a "definitive" approach to risk assessment enables some decisions to be based on the results of a single test. Data are biologically relevant for risk assessment only if they test a hypothesis that is useful for decision-making.

Weight of Evidence for Hazard Identification: A Critical Review of the Literature

BACKGROUND

Transparency when documenting and assessing weight of evidence (WOE) has been an area of increasing focus for national and international health agencies.

OBJECTIVE

The objective of this work was to conduct a critical review of WOE analysis methods as a basis for developing a practical framework for considering and assessing WOE in hazard identification in areas of application at the French Agency for Food, Environmental and Occupational Health and Safety (ANSES).

METHODS

Based on a review of the literature and directed requests to 63 international and national agencies, 116 relevant articles and guidance documents were selected. The WOE approaches were assessed based on three aspects: the extent of their prescriptive nature, their purpose-specific relevance, and their ease of implementation.

RESULTS

Twenty-four approaches meeting the specified criteria were identified from selected reviewed documents. Most approaches satisfied one or two of the assessed considerations, but not all three. The approaches were grouped within a practical framework comprising the following four stages: (1) planning the assessment, including scoping, formulating the question, and developing the assessment method; (2) establishing lines of evidence (LOEs), including identifying and selecting studies, assessing their quality, and integrating with studies of similar type; (3) integrating the LOEs to evaluate WOE; and (4) presenting conclusions.

DISCUSSION

Based on the review, considerations for selecting methods for a wide range of applications are proposed. Priority areas for further development are identified. https://doi.org/10.1289/EHP3067.

Six years after the NRC review of EPA's Draft IRIS Toxicological Review of Formaldehyde: Regulatory implications of new science in evaluating formaldehyde leukemogenicity

Shortly after the International Agency for Research on Cancer (IARC) determined that formaldehyde causes leukemia, the United States Environmental Protection Agency (EPA) released its Draft IRIS Toxicological Review of Formaldehyde ("Draft IRIS Assessment"), also concluding that formaldehyde causes leukemia. Peer review of the Draft IRIS Assessment by a National Academy of Science committee noted that "causal determinations are not supported by the narrative provided in the draft" (NRC 2011). They offered recommendations for improving the Draft IRIS assessment and identified several important research gaps. Over the six years since the NRC peer review, significant new science has been published. We identify and summarize key recommendations made by NRC and map them to this new science, including extended analysis of epidemiological studies, updates of earlier occupational cohort studies, toxicological experiments using a sensitive mouse strain, mechanistic studies examining the role of exogenous versus endogenous formaldehyde in bone marrow, and several critical reviews. With few exceptions, new findings are consistently negative, and integration of all available evidence challenges the earlier conclusions that formaldehyde causes leukemia. Given formaldehyde's commercial importance, environmental ubiquity and endogenous production, accurate hazard classification and risk evaluation of whether exposure to formaldehyde from occupational, residential and consumer products causes leukemia are critical.

Using systematic review in occupational safety and health

Evaluation of scientific evidence is critical in developing recommendations to reduce risk. Healthcare was the first scientific field to employ a systematic review approach for synthesizing research findings to support evidence-based decision-making and it is still the largest producer and consumer of systematic reviews. Systematic reviews in the field of occupational safety and health are being conducted, but more widespread use and adoption would strengthen assessments. In 2016, NIOSH asked RAND to develop a framework for applying the traditional systematic review elements to the field of occupational safety and health. This paper describes how essential systematic review elements can be adapted for use in occupational systematic reviews to enhance their scientific quality, objectivity, transparency, reliability, utility, and acceptability.

Framework for the quantitative weight-of-evidence analysis of 'omics data for regulatory purposes

A framework for the quantitative weight-of-evidence (QWoE) analysis of 'omics data for regulatory purposes is presented. The QWoE framework encompasses seven steps to evaluate 'omics data (also together with non-'omics data): (1) Hypothesis formulation, identification and weighting of lines of evidence (LoEs). LoEs conjoin different (types of) studies that are used to critically test the hypothesis. As an essential component of the QWoE framework, step 1 includes the development of templates for scoring sheets that predefine scoring criteria with scores of 0-4 to enable a quantitative determination of study quality and data relevance; (2) literature searches and categorisation of studies into the pre-defined LoEs; (3) and (4) quantitative assessment of study quality and data relevance using the respective pre-defined scoring sheets for each study; (5) evaluation of LoE-specific strength of evidence based upon the study quality and study relevance scores of the studies conjoined in the respective LoE; (6) integration of the strength of evidence from the individual LoEs to determine the overall strength of evidence; (7) characterisation of uncertainties and conclusion on the QWoE. To put the QWoE framework in practice, case studies are recommended to confirm the relevance of its different steps, or to adapt them as necessary.

Glyphosate toxicity and carcinogenicity: a review of the scientific basis of the European Union assessment and its differences with IARC

Glyphosate is the most widely used herbicide worldwide. It is a broad spectrum herbicide and its agricultural uses increased considerably after the development of glyphosate-resistant genetically modified (GM) varieties. Since glyphosate was introduced in 1974, all regulatory assessments have established that glyphosate has low hazard potential to mammals, however, the International Agency for Research on Cancer (IARC) concluded in March 2015 that it is probably carcinogenic. The IARC conclusion was not confirmed by the EU assessment or the recent joint WHO/FAO evaluation, both using additional evidence. Glyphosate is not the first topic of disagreement between IARC and regulatory evaluations, but has received greater attention. This review presents the scientific basis of the glyphosate health assessment conducted within the European Union (EU) renewal process, and explains the differences in the carcinogenicity assessment with IARC. Use of different data sets, particularly on long-term toxicity/carcinogenicity in rodents, could partially explain the divergent views; but methodological differences in the evaluation of the available evidence have been identified. The EU assessment did not identify a carcinogenicity hazard, revised the toxicological profile proposing new toxicological reference values, and conducted a risk assessment for some representatives uses. Two complementary exposure assessments, human-biomonitoring and food-residues-monitoring, suggests that actual exposure levels are below these reference values and do not represent a public concern.

A primer on systematic reviews in toxicology

Systematic reviews, pioneered in the clinical field, provide a transparent, methodologically rigorous and reproducible means of summarizing the available evidence on a precisely framed research question. Having matured to a well-established approach in many research fields, systematic reviews are receiving increasing attention as a potential tool for answering toxicological questions. In the larger framework of evidence-based toxicology, the advantages and obstacles of, as well as the approaches for, adapting and adopting systematic reviews to toxicology are still being explored. To provide the toxicology community with a starting point for conducting or understanding systematic reviews, we herein summarized available guidance documents from various fields of application. We have elaborated on the systematic review process by breaking it down into ten steps, starting with planning the project, framing the question, and writing and publishing the protocol, and concluding with interpretation and reporting. In addition, we have identified the specific methodological challenges of toxicological questions and have summarized how these can be addressed. Ultimately, this primer is intended to stimulate scientific discussions of the identified issues to fuel the development of toxicology-specific methodology and to encourage the application of systematic review methodology to toxicological issues.

Characterizing the Long-Term PM2.5 Concentration-Response Function: Comparing the Strengths and Weaknesses of Research Synthesis Approaches

The magnitude, shape, and degree of certainty in the association between long-term population exposure to ambient fine particulate matter (PM2.5 ) and the risk of premature death is one of the most intensely studied issues in environmental health. For regulatory risk analysis, this relationship is described quantitatively by a concentration-response (C-R) function that relates exposure to ambient concentrations with the risk of premature mortality. Four data synthesis techniques develop the basis for, and derive, this function: systematic review, expert judgment elicitation, quantitative meta-analysis, and integrated exposure-response (IER) assessment. As part of an academic workshop aiming to guide the use of research synthesis approaches, we developed criteria with which to evaluate and select among the approaches for their ability to inform policy choices. These criteria include the quality and extent of scientific support for the method, its transparency and verifiability, its suitability to the policy problem, and the time and resources required for its application. We find that these research methods are both complementary and interdependent. A systematic review of the multidisciplinary evidence is a starting point for all methods, providing the broad conceptual basis for the nature, plausibility, and strength of the associations between PM exposure and adverse health effects. Further, for a data-rich application like PM2.5 and premature mortality, all three quantitative approaches can produce estimates that are suitable for regulatory and benefit analysis. However, when fewer data are available, more resource-intensive approaches such as expert elicitation may be more important for understanding what scientists know, where they agree or disagree, and what they believe to be the most important areas of uncertainty. Whether implicitly or explicitly, all require considerable judgment by scientists. Finding ways for all these methods to acknowledge, appropriately elicit, and examine the implications of that judgment would be an important step forward for research synthesis.

Key Elements for Judging the Quality of a Risk Assessment

BACKGROUND

Many reports have been published that contain recommendations for improving the quality, transparency, and usefulness of decision making for risk assessments prepared by agencies of the U.S. federal government. A substantial measure of consensus has emerged regarding the characteristics that high-quality assessments should possess.

OBJECTIVE

The goal was to summarize the key characteristics of a high-quality assessment as identified in the consensus-building process and to integrate them into a guide for use by decision makers, risk assessors, peer reviewers and other interested stakeholders to determine if an assessment meets the criteria for high quality.

DISCUSSION

Most of the features cited in the guide are applicable to any type of assessment, whether it encompasses one, two, or all four phases of the risk-assessment paradigm; whether it is qualitative or quantitative; and whether it is screening level or highly sophisticated and complex. Other features are tailored to specific elements of an assessment. Just as agencies at all levels of government are responsible for determining the effectiveness of their programs, so too should they determine the effectiveness of their assessments used in support of their regulatory decisions. Furthermore, if a nongovernmental entity wishes to have its assessments considered in the governmental regulatory decision-making process, then these assessments should be judged in the same rigorous manner and be held to similar standards.

CONCLUSIONS

The key characteristics of a high-quality assessment can be summarized and integrated into a guide for judging whether an assessment possesses the desired features of high quality, transparency, and usefulness.

CITATION

Fenner-Crisp PA, Dellarco VL. 2016. Key elements for judging the quality of a risk assessment. Environ Health Perspect 124:1127-1135; http://dx.doi.org/10.1289/ehp.1510483.

Hypothesis-based weight-of-evidence evaluation and risk assessment for naphthalene carcinogenesis

Inhalation of naphthalene causes olfactory epithelial nasal tumors in rats (but not in mice) and benign lung adenomas in mice (but not in rats). The limited available human data have not identified an association between naphthalene exposure and increased respiratory cancer risk. Assessing naphthalene's carcinogenicity in humans, therefore, depends entirely on experimental evidence from rodents. We evaluated the respiratory carcinogenicity of naphthalene in rodents, and its potential relevance to humans, using our Hypothesis-Based Weight-of-Evidence (HBWoE) approach. We systematically and comparatively reviewed data relevant to key elements in the hypothesized modes of action (MoA) to determine which is best supported by the available data, allowing all of the data from each realm of investigation to inform interpretation of one another. Our analysis supports a mechanism that involves initial metabolism of naphthalene to the epoxide, followed by GSH depletion, cytotoxicity, chronic inflammation, regenerative hyperplasia, and tumor formation, with possible weak genotoxicity from downstream metabolites occurring only at high cytotoxic doses, strongly supporting a non-mutagenic threshold MoA in the rat nose. We also conducted a dose-response analysis, based on the likely MoA, which suggests that the rat nasal MoA is not relevant in human respiratory tissues at typical environmental exposures. Our analysis illustrates how a thorough WoE evaluation can be used to support a MoA, even when a mechanism of action cannot be fully elucidated. A non-mutagenic threshold MoA for naphthalene-induced rat nasal tumors should be considered as a basis to determine human relevance and to guide regulatory and risk-management decisions.