The value of animal test information in environmental control decisions

A value of information framework for assessing the trade-offs associated with uncertainty, duration, and cost of chemical toxicity testing

A number of investigators have explored the use of value of information (VOI) analysis to evaluate alternative information collection procedures in diverse decision-making contexts. This paper presents an analytic framework for determining the value of toxicity information used in risk-based decision making. The framework is specifically designed to explore the trade-offs between cost, timeliness, and uncertainty reduction associated with different toxicity-testing methodologies. The use of the proposed framework is demonstrated by two illustrative applications which, although based on simplified assumptions, show the insights that can be obtained through the use of VOI analysis. Specifically, these results suggest that timeliness of information collection has a significant impact on estimates of the VOI of chemical toxicity tests, even in the presence of smaller reductions in uncertainty. The framework introduces the concept of the expected value of delayed sample information, as an extension to the usual expected value of sample information, to accommodate the reductions in value resulting from delayed decision making. Our analysis also suggests that lower cost and higher throughput testing also may be beneficial in terms of public health benefits by increasing the number of substances that can be evaluated within a given budget. When the relative value is expressed in terms of return-on-investment per testing strategy, the differences can be substantial.

The Role of Decision Analysis in Risk Analysis: A Retrospective

In commemorating the 40th anniversary of Risk Analysis, this article takes a retrospective look at some of the ways in which decision analysis (as a "sibling field") has contributed to the development both of the journal, and of risk analysis as a field. I begin with some early foundational papers from the first decade of the journal's history. I then review a number of papers that have applied decision analysis to risk problems over the years, including applications of related methods such as influence diagrams, multicriteria decision analysis, and risk matrices. The article then reviews some recent trends, from roughly the last five years, and concludes with observations about the parallel evolution of risk analysis and decision analysis over the decades-especially with regard to the importance of representing multiple stakeholder perspectives, and the importance of behavioral realism in decision models. Overall, the extensive literature surveyed here supports the view that the incorporation of decision-analytic perspectives has improved the practice of risk analysis.

Expected Value of Sample Information Calculations in Medical Decision Modeling

There has been an increasing interest in using expected value of information (EVI) theory in medical decision making, to identify the need for further research to reduce uncertainty in decision and as a tool for sensitivity analysis. Expected value of sample information (EVSI) has been proposed for determination of optimum sample size and allocation rates in randomized clinical trials. This article derives simple Monte Carlo, or nested Monte Carlo, methods that extend the use of EVSI calculations to medical decision applications with multiple sources of uncertainty, with particular attention to the form in which epidemiological data and research findings are structured. In particular, information on key decision parameters such as treatment efficacy are invariably available on measures of relative efficacy such as risk differences or odds ratios, but not on model parameters themselves. In addition, estimates of model parameters and of relative effect measures in the literature may be heterogeneous, reflecting additional sources of variation besides statistical sampling error. The authors describe Monte Carlo procedures for calculating EVSI for probability, rate, or continuous variable parameters in multi parameter decision models and approximate methods for relative measures such as risk differences, odds ratios, risk ratios, and hazard ratios. Where prior evidence is based on a random effects meta-analysis, the authors describe different ESVI calculations, one relevant for decisions concerning a specific patient group and the other for decisions concerning the entire population of patient groups. They also consider EVSI methods for new studies intended to update information on both baseline treatment efficacy and the relative efficacy of 2 treatments. Although there are restrictions regarding models with prior correlation between parameters, these methods can be applied to the majority of probabilistic decision models. Illustrative worked examples of EVSI calculations are given in an appendix.

A quantitative approach for integrating multiple lines of evidence for the evaluation of environmental health risks

Decision analysis often considers multiple lines of evidence during the decision making process. Researchers and government agencies have advocated for quantitative weight-of-evidence approaches in which multiple lines of evidence can be considered when estimating risk. Therefore, we utilized Bayesian Markov Chain Monte Carlo to integrate several human-health risk assessment, biomonitoring, and epidemiology studies that have been conducted for two common insecticides (malathion and permethrin) used for adult mosquito management to generate an overall estimate of risk quotient (RQ). The utility of the Bayesian inference for risk management is that the estimated risk represents a probability distribution from which the probability of exceeding a threshold can be estimated. The mean RQs after all studies were incorporated were 0.4386, with a variance of 0.0163 for malathion and 0.3281 with a variance of 0.0083 for permethrin. After taking into account all of the evidence available on the risks of ULV insecticides, the probability that malathion or permethrin would exceed a level of concern was less than 0.0001. Bayesian estimates can substantially improve decisions by allowing decision makers to estimate the probability that a risk will exceed a level of concern by considering seemingly disparate lines of evidence.

Potential uncertainty reduction in model-averaged benchmark dose estimates informed by an additional dose study

A methodology is presented for assessing the information value of an additional dosage experiment in existing bioassay studies. The analysis demonstrates the potential reduction in the uncertainty of toxicity metrics derived from expanded studies, providing insights for future studies. Bayesian methods are used to fit alternative dose-response models using Markov chain Monte Carlo (MCMC) simulation for parameter estimation and Bayesian model averaging (BMA) is used to compare and combine the alternative models. BMA predictions for benchmark dose (BMD) are developed, with uncertainty in these predictions used to derive the lower bound BMDL. The MCMC and BMA results provide a basis for a subsequent Monte Carlo analysis that backcasts the dosage where an additional test group would have been most beneficial in reducing the uncertainty in the BMD prediction, along with the magnitude of the expected uncertainty reduction. Uncertainty reductions are measured in terms of reduced interval widths of predicted BMD values and increases in BMDL values that occur as a result of this reduced uncertainty. The methodology is illustrated using two existing data sets for TCDD carcinogenicity, fitted with two alternative dose-response models (logistic and quantal-linear). The example shows that an additional dose at a relatively high value would have been most effective for reducing the uncertainty in BMA BMD estimates, with predicted reductions in the widths of uncertainty intervals of approximately 30%, and expected increases in BMDL values of 5-10%. The results demonstrate that dose selection for studies that subsequently inform dose-response models can benefit from consideration of how these models will be fit, combined, and interpreted.

Probabilistic exposure analysis for chemical risk characterization

This paper summarizes the state of the science of probabilistic exposure assessment (PEA) as applied to chemical risk characterization. Current probabilistic risk analysis methods applied to PEA are reviewed. PEA within the context of risk-based decision making is discussed, including probabilistic treatment of related uncertainty, interindividual heterogeneity, and other sources of variability. Key examples of recent experience gained in assessing human exposures to chemicals in the environment, and other applications to chemical risk characterization and assessment, are presented. It is concluded that, although improvements continue to be made, existing methods suffice for effective application of PEA to support quantitative analyses of the risk of chemically induced toxicity that play an increasing role in key decision-making objectives involving health protection, triage, civil justice, and criminal justice. Different types of information required to apply PEA to these different decision contexts are identified, and specific PEA methods are highlighted that are best suited to exposure assessment in these separate contexts.

Methods for assessing uncertainty in fundamental assumptions and associated models for cancer risk assessment

The distributional approach for uncertainty analysis in cancer risk assessment is reviewed and extended. The method considers a combination of bioassay study results, targeted experiments, and expert judgment regarding biological mechanisms to predict a probability distribution for uncertain cancer risks. Probabilities are assigned to alternative model components, including the determination of human carcinogenicity, mode of action, the dosimetry measure for exposure, the mathematical form of the dose-response relationship, the experimental data set(s) used to fit the relationship, and the formula used for interspecies extrapolation. Alternative software platforms for implementing the method are considered, including Bayesian belief networks (BBNs) that facilitate assignment of prior probabilities, specification of relationships among model components, and identification of all output nodes on the probability tree. The method is demonstrated using the application of Evans, Sielken, and co-workers for predicting cancer risk from formaldehyde inhalation exposure. Uncertainty distributions are derived for maximum likelihood estimate (MLE) and 95th percentile upper confidence limit (UCL) unit cancer risk estimates, and the effects of resolving selected model uncertainties on these distributions are demonstrated, considering both perfect and partial information for these model components. A method for synthesizing the results of multiple mechanistic studies is introduced, considering the assessed sensitivities and selectivities of the studies for their targeted effects. A highly simplified example is presented illustrating assessment of genotoxicity based on studies of DNA damage response caused by naphthalene and its metabolites. The approach can provide a formal mechanism for synthesizing multiple sources of information using a transparent and replicable weight-of-evidence procedure.

Tiered chemical testing: a value of information approach

In December 2000 the EPA initiated the Voluntary Children's Chemical Evaluation Program (VCCEP) by asking manufacturers to voluntarily sponsor toxicological testing in a tiered process for 23 chemicals selected for the pilot phase. The tiered nature of the VCCEP pilot program creates the need for clearly defined criteria for determining when information is sufficient to assess the potential risks to children. This raises questions about how to determine the "adequacy" of the existing information and assess the need to undertake efforts to reduce uncertainty (through further testing). This article applies a value of information analysis approach to determine adequacy by modeling how toxicological and exposure data collected through the VCCEP may be used to inform risk management decisions. The analysis demonstrates the importance of information about the exposure level and control costs in making decisions regarding further toxicological testing. This article accounts for the cost of delaying control action and identifies the optimal testing strategy for a constrained decisionmaker who, absent applicable human data, cannot regulate without bioassay data on a specific chemical. It also quantifies the differences in optimal testing strategy for three decision criteria: maximizing societal net benefits, ensuring maximum exposure control while net benefits are positive (i.e., benefits outweigh costs), and controlling to the maximum extent technologically feasible while the lifetime risk of cancer exceeds a specific level of risk. Finally, this article shows the large differences that exist in net benefits between the three criteria for the range of exposure levels where the optimal actions differ.

Value of information literature analysis: a review of applications in health risk management

This article provides the first comprehensive review of value of information (VOI) analyses related to health risk management published in English in peer-reviewed journals by the end of 2001. VOI analysis represents a decision analytic technique that explicitly evaluates the benefit of collecting additional information to reduce or eliminate uncertainty. Through a content analysis of VOI applications, this article characterizes various attributes of VOI applications, shows the evolution of the methodology and advances in computing tools that allow analysis of increasingly complex problems, and suggests the need for some standardization of reporting methods and results. The authors' analysis shows a lack of cross-fertilization across topic areas and the tendency of articles to focus on demonstrating the usefulness of the VOI approach rather than applications to actual management decisions. This article provides important insights for VOI applications in medical decision making.

Value of information analysis in environmental health risk management decisions: past, present, and future

Experts agree that value of information (VOI) analyses provide useful insights in risk management decisions. However, applications in environmental health risk management (EHRM) remain largely demonstrative thus far because of the complexity in modeling and solving VOI problems. Based on this comprehensive review of all VOI applications published in the peer-reviewed literature of such applications, the complexity of solving VOI problems with continuous probability distributions as inputs in models emerges as the main barrier to greater use of VOI although simulation allows analysts to solve more complex and realistic problems. Several analytical challenges that inhibit greater use of VOI techniques include issues related to modeling decisions, valuing outcomes, and characterizing uncertain and variable model inputs appropriately. This comprehensive review of methods for modeling and solving VOI problems for applications related to EHRM provides the first synthesis of important methodological advances in the field. The insights provide risk analysts and decision scientists with some guidance on how to structure and solve VOI problems focused on evaluating opportunities to collect better information to improve EHRM decisions. They further suggest the need for some efforts to standardize approaches and develop some prescriptive guidance for VOI analysts similar to existing guidelines for conducting cost-effectiveness analyses.

Reconstructing population exposures from dose biomarkers: inhalation of trichloroethylene (TCE) as a case study

Physiologically based pharmacokinetic (PBPK) modeling is a well-established toxicological tool designed to relate exposure to a target tissue dose. The emergence of federal and state programs for environmental health tracking and the availability of exposure monitoring through biomarkers creates the opportunity to apply PBPK models to estimate exposures to environmental contaminants from urine, blood, and tissue samples. However, reconstructing exposures for large populations is complicated by often having too few biomarker samples, large uncertainties about exposures, and large interindividual variability. In this paper, we use an illustrative case study to identify some of these difficulties, and for a process for confronting them by reconstructing population-scale exposures using Bayesian inference. The application consists of interpreting biomarker data from eight adult males with controlled exposures to trichloroethylene (TCE) as if the biomarkers were random samples from a large population with unknown exposure conditions. The TCE concentrations in blood from the individuals fell into two distinctly different groups even though the individuals were simultaneously in a single exposure chamber. We successfully reconstructed the exposure scenarios for both subgroups - although the reconstruction of one subgroup is different than what is believed to be the true experimental conditions. We were however unable to predict with high certainty the concentration of TCE in air.

Use of expert judgment in exposure assessment: part 2. Calibration of expert judgments about personal exposures to benzene

The recent movement of regulatory agencies toward probabilistic analyses of human health and environmental risks has focused greater attention on the quality of the estimates of variability and uncertainty that underlie them. Of particular concern is how uncertainty--a measure of what is not known--is characterized, as uncertainty can play an influential role in analyses of the need for regulatory controls or in estimates of the economic value of additional research. This paper reports the second phase of a study, conducted as an element of the National Human Exposure Assessment Survey (NHEXAS), to obtain and calibrate exposure assessment experts judgments about uncertainty in residential ambient, residential indoor, and personal air benzene concentrations experienced by the nonsmoking, nonoccupationally exposed population in U.S. EPA's Region V. Subjective judgments (i.e., the median, interquartile range, and 90% confidence interval) about the means and 90th percentiles of each of the benzene distributions were elicited from the seven experts participating in the study. The calibration or quality of the experts' judgments was assessed by comparing them to the actual measurements from the NHEXAS Region V study using graphical techniques, a quadratic scoring rule, and surprise and interquartile indices. The results from both quantitative scoring methods suggested that, considered collectively, the experts' judgments were relatively well calibrated although on balance, underconfident. The calibration of individual expert judgments appeared variable, highlighting potential pitfalls in reliance on individual experts. In a surprising finding, the experts' judgments about the 90th percentiles of the benzene distributions were better calibrated than their predictions about the means; the experts tended to be overconfident in their ability to predict the means. This paper is also one of the first calibration studies to demonstrate the importance of taking into account intraexpert correlation on the statistical significance of the findings. When the judgments were assumed to be independent, analysis of the surprise and interquartile indices found evidence of poor calibration (P<0.05). However, when the intraexpert correlation in the study was taken into account, these findings were no longer statistically significant. The analysis further found that the experts' judgments scored better than estimates of Region V benzene concentrations simply drawn from earlier studies of ambient, indoor and personal benzene levels in other U.S. cities. These results suggest the value of careful elicitation of expert judgments in characterizing exposures in probabilistic form. Additional calibration studies need to be undertaken to corroborate and extend these findings.

Use of expert judgment in exposure assessment. Part I. Characterization of personal exposure to benzene

This paper presents the results of the first phase of a study, conducted as an element of the National Human Exposure Assessment Survey (NHEXAS), to demonstrate the use of expert subjective judgment elicitation techniques to characterize the magnitude of and uncertainty in environmental exposure to benzene. In decisions about the value of exposure research or of regulatory controls, the characterization of uncertainty can play an influential role. Classical methods for characterizing uncertainty may be sufficient when adequate amounts of relevant data are available. Frequently, however, data are neither abundant nor directly relevant, making it necessary to rely to varying degrees on subjective judgment. Since the 1950s, methods to elicit and quantify subjective judgments have been explored but have rarely been applied to the field of environmental exposure assessment. In this phase of the project, seven experts in benzene exposure assessment were selected through a peer nomination process, participated in a 2-day workshop, and were interviewed individually to elicit their judgments about the distributions of residential ambient, residential indoor, and personal air benzene concentrations (6-day integrated average) experienced by both the non-smoking, non-occupationally exposed target and study populations of the US EPA Region V pilot study. Specifically, each expert was asked to characterize, in probabilistic form, the arithmetic means and the 90th percentiles of these distributions. This paper presents the experts' judgments about the concentrations of benzene encountered by the target population. The experts' judgments about levels of benzene in personal air were demonstrative of patterns observed in the judgments about the other distributions. They were in closest agreement about their predictions of the mean; with one exception, their best estimates of the mean fell within 7-11 microg/m(3) although they exhibited striking differences in the degree of uncertainty expressed. Their estimates of the 90th percentile were more varied with the best estimates ranging from 12 to 26 microg/m(3) for all but one expert. However, their predictions of the 90th percentile were far more uncertain. The paper demonstrates that coherent subjective judgments can be elicited from exposure assessment scientists and critically examines the challenges and potential benefits of a subjective judgment approach. The results of the second phase of the project, in which measurements from the NHEXAS field study in Region V are used to calibrate the experts' judgments about the benzene exposures in the study population, will be presented in a second paper.

Risk-based environmental remediation: Bayesian Monte Carlo analysis and the expected value of sample information

A methodology that simulates outcomes from future data collection programs, utilizes Bayesian Monte Carlo analysis to predict the resulting reduction in uncertainty in an environmental fate-and-transport model, and estimates the expected value of this reduction in uncertainty to a risk-based environmental remediation decision is illustrated considering polychlorinated biphenyl (PCB) sediment contamination and uptake by winter flounder in New Bedford Harbor, MA. The expected value of sample information (EVSI), the difference between the expected loss of the optimal decision based on the prior uncertainty analysis and the expected loss of the optimal decision from an updated information state, is calculated for several sampling plan. For the illustrative application we have posed, the EVSI for a sampling plan of two data points is $9.4 million, for five data points is $10.4 million, and for ten data points is $11.5 million. The EVSI for sampling plans involving larger numbers of data points is bounded by the expected value of perfect information, $15.6 million. A sensitivity analysis is conducted to examine the effect of selected model structure and parametric assumptions on the optimal decision and the EVSI. The optimal decision (total area to be dredged) is sensitive to the assumption of linearity between PCB sediment concentration and flounder PCB body burden and to the assumed relationship between area dredged and the harbor-wide average sediment PCB concentration; these assumptions also have a moderate impact on the computed EVSI. The EVSI is most sensitive to the unit cost of remediation and rather insensitive to the penalty cost associated with under-remediation.