Addressing uncertainty in medical cost-effectiveness analysis implications of expected utility maximization for methods to perform sensitivity analysis and the use of cost-effectiveness analysis to set priorities for medical research

Net Monetary Benefit Lines Augmented with Value-of-Information Measures to Present the Results of Economic Evaluations under Uncertainty

BACKGROUND

Methods to present the result of cost-effectiveness analyses under parameter uncertainty include cost-effectiveness planes (CEPs), cost-effectiveness acceptability curves/frontier (CEACs/CEAF), expected loss curves (ELCs), and net monetary benefit (NMB) lines. We describe how NMB lines can be augmented to present NMB values that could be achieved by reducing or resolving parameter uncertainty. We evaluated the ability of these methods to correctly 1) identify the alternative with the highest expected NMB and 2) communicate the magnitude of parameter and decision uncertainty.

METHODS

We considered 4 hypothetical decision problems representing scenarios with high variance or correlated cost and effect estimates and alternatives with similar cost-effectiveness ratios. We used these decision problems to demonstrate the limitations of existing methods and the potential of augmented NMB lines to resolve these issues.

RESULTS

CEPs and CEACs/CEAF could falsely imply the lack of sufficient evidence to identify the optimal option if cost and effect estimates have high variance, are correlated across alternatives, or when alternatives have similar cost-effectiveness ratios. The augmented NMB lines and ELCs can correctly identify the option with the highest expected NMB and communicate the potential benefit of resolving uncertainties. Like ELCs, the augmented NMB lines provide information about the value of resolving parameter uncertainties, but augmented NMB lines may be easier to interpret for decision makers.

CONCLUSIONS

Our analysis supports recommending the augment NMB lines as an important method to present the results of economic evaluation studies under parameter uncertainty.

HIGHLIGHTS

The results of cost-effectiveness analyses (CEAs) when the cost and effect estimates of alternatives are uncertain are commonly presented using cost-effectiveness planes (CEPs), cost-effectiveness acceptability curves/frontier (CEACs/CEAF), and expected loss curves (ELCs).Although currently not often used, net monetary benefit (NMB) lines could present the results of cost-effectiveness to identify the alternative with the highest expected NMB values given the current level of uncertainty. Furthermore, NMB lines can be augmented to 1) show metrics of value of information, which measure the value of additional research to reduce or eliminate the decision uncertainty, and 2) display the confidence intervals along the NMB lines to ensure that NMB values are estimated accurately using a sufficiently large number of parameter samples.Using several decision problems, we demonstrate the limitation of existing methods to present the results of CEAs under parameter uncertainty and how augmented NMB lines could resolve these issues.Our analysis supports recommending augmented NMB lines as an important method to present the results of CEA under uncertainty since they 1) correctly identify the alternative with the highest expected NMB value given the current evidence, 2) provide information about the potential value of additional research to improve the decision by reducing or resolving uncertainty in model parameters, 3) assist the analysis to visually ensure that enough parameter samples are used to estimate the expected NMB of alternatives, and 4) are easier to interpret for decision makers compared with other methods.

How Many Monte Carlo Samples are Needed for Probabilistic Cost-Effectiveness Analyses?

OBJECTIVES

Probabilistic sensitivity analysis (PSA) is conducted to account for the uncertainty in cost and effect of decision options under consideration. PSA involves obtaining a large sample of input parameter values (N) to estimate the expected cost and effect of each alternative in the presence of parameter uncertainty. When the analysis involves using stochastic models (eg, individual-level models), the model is further replicated P times for each sampled parameter set. We study how N and P should be determined.

METHODS

We show that PSA could be structured such that P can be an arbitrary number (say, P=1). To determine N, we derive a formula based on Chebyshev's inequality such that the error in estimating the incremental cost-effectiveness ratio (ICER) of alternatives (or equivalently, the willingness-to-pay value at which the optimal decision option changes) is within a desired level of accuracy. We described 2 methods to confirm, visually and quantitatively, that the N informed by this method results in ICER estimates within the specified level of accuracy.

RESULTS

When N is arbitrarily selected, the estimated ICERs could be substantially different from the true ICER (even as P increases), which could lead to misleading conclusions. Using a simple resource allocation model, we demonstrate that the proposed approach can minimize the potential for this error.

CONCLUSIONS

The number of parameter samples in probabilistic cost-effectiveness analyses should not be arbitrarily selected. We describe 3 methods to ensure that enough parameter samples are used in probabilistic cost-effectiveness analyses.

Toward Patient-Centered Drug Approval for Treatment of Rare Diseases

OBJECTIVES

This commentary seeks to improve the design and analysis of trials undertaken to obtain approval of drugs for treatment of rare diseases.

METHODS

Methodological analysis reveals that use of hypothesis testing in the Food and Drug Administration drug approval process is harmful. Conventional asymmetric error probabilities bias the approval process against approval of new drugs. Hypothesis testing is inattentive to the relative magnitudes of losses to patient welfare when types 1 and 2 errors occur. Requiring the sample size to be large enough to guarantee the specified statistical power particularly inhibits the development of new drugs for treating rare diseases. Rarity of a disease makes it difficult to enroll the number of trial subjects needed to meet the statistical power standards for drug approval.

RESULTS

Use of statistical decision theory in drug approval would overcome these serious deficiencies of hypothesis testing. Sample size would remain relevant to drug approval, but the criterion used to evaluate sample size would change. Rather than judging sample size by statistical power, the Food and Drug Administration could require a sample to be large enough to provide a specified nearness to optimality of the approval decision.

CONCLUSIONS

Using nearness to optimality to set sample size and making approval decisions to minimize distance from optimality would particularly benefit the evaluation of drugs for treatment of rare diseases. It would enable a dramatic reduction in sample size relative to current norms, without compromising the clinical informativeness of trials.

The Value of Health in a Cost-Effectiveness Analysis: Theory Versus Practice

A cost-effectiveness analysis has become an important method to inform allocation decisions and reimbursement of new technologies in healthcare. A cost-effectiveness analysis requires a threshold to which the cost effectiveness of a new intervention can be compared. In principle, the threshold ought to reflect opportunity costs of reimbursing a new technology. In this paper, we contrast the practical use of this threshold within a CEA with its theoretical underpinnings. We argue that several assumptions behind the theoretical models underlying this threshold are violated in practice. This implies that a simple application of the decision rules of CEA using a single estimate of the threshold does not necessarily improve population health or societal welfare. Conceptual differences regarding the interpretation of the threshold, widely varying estimates of its value, and an inconsistent use within and outside the healthcare sector are important challenges in informing policy makers on optimal reimbursement decision and setting appropriate healthcare budgets.

Statistical Decision Properties of Imprecise Trials Assessing Coronavirus Disease 2019 (COVID-19) Drugs

OBJECTIVES

Researchers studying treatment of coronavirus disease 2019 (COVID-19) have reported findings of randomized trials comparing standard care with care augmented by experimental drugs. Many trials have small sample sizes, so estimates of treatment effects are imprecise. Hence, clinicians may find it difficult to decide when to treat patients with experimental drugs. A conventional practice when comparing standard care and an innovation is to choose the innovation only if the estimated treatment effect is positive and statistically significant. This practice defers to standard care as the status quo. We study treatment choice from the perspective of statistical decision theory, which considers treatment options symmetrically when assessing trial findings.

METHODS

We use the concept of near-optimality to evaluate criteria for treatment choice. This concept jointly considers the probability and magnitude of decision errors. An appealing criterion from this perspective is the empirical success rule, which chooses the treatment with the highest observed average patient outcome in the trial.

RESULTS

Considering the design of some COVID-19 trials, we show that the empirical success rule yields treatment choices that are much closer to optimal than those generated by prevailing decision criteria based on hypothesis tests.

CONCLUSION

Using trial findings to make near-optimal treatment choices rather than perform hypothesis tests should improve clinical decision making.

Value of Information Analysis for Research Decisions-An Introduction: Report 1 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force

Healthcare resource allocation decisions made under conditions of uncertainty may turn out to be suboptimal. In a resource constrained system in which there is a fixed budget, these suboptimal decisions will result in health loss. Consequently, there may be value in reducing uncertainty, through the collection of new evidence, to make better resource allocation decisions. This value can be quantified using a value of information (VOI) analysis. This report, from the ISPOR VOI Task Force, introduces VOI analysis, defines key concepts and terminology, and outlines the role of VOI for supporting decision making, including the steps involved in undertaking and interpreting VOI analyses. The report is specifically aimed at those tasked with making decisions about the adoption of healthcare or the funding of healthcare research. The report provides a number of recommendations for good practice when planning, undertaking, or reviewing the results of VOI analyses.

Reasonable patient care under uncertainty

This paper discusses how limited ability to predict illness and treatment response may affect the welfare achieved in patient care. The discussion covers both decentralized clinical decision making and care that adheres to clinical practice guidelines. I explain why predictive ability has been limited, calling attention to questionable methodological practices in the research that supports evidence-based medicine. I summarize research on identification whose objective is to yield credible prediction of patient outcomes. Recognizing that uncertainty will continue to afflict medical decision making, I apply basic decision theory to suggest reasonable decision criteria with well-understood welfare properties. Previous research on medical decision making has largely embraced Bayesian decision theory. I summarize research studying the minimax-regret criterion, which seeks uniformly near-optimal decisions.

Estimating the Economic Value of Information for Screening in Disseminating and Targeting Effective School-based Preventive Interventions: An Illustrative Example

When candidates for school-based preventive interventions are heterogeneous in their risk of poor outcomes, an intervention's expected economic net benefits may be maximized by targeting candidates for whom the intervention is most likely to yield benefits, such as those at high risk of poor outcomes. Although increasing amounts of information about candidates may facilitate more accurate targeting, collecting information can be costly. We present an illustrative example to show how cost-benefit analysis results from effective intervention demonstrations can help us to assess whether improved targeting accuracy justifies the cost of collecting additional information needed to make this improvement.

Cost-effectiveness of Antihypertensive Medication: Exploring Race and Sex Differences Using Data From the REasons for Geographic and Racial Differences in Stroke Study

BACKGROUND

Antihypertensive medication decreases risk of cardiovascular disease (CVD) events in adults with hypertension. Although black adults have higher prevalence of hypertension and worse CVD outcomes compared with whites, limited attention has been given to the cost-effectiveness of antihypertensive medication for blacks.

OBJECTIVE

To compare the cost-effectiveness of antihypertensive medication treatment versus no-treatment in white and black adults.

RESEARCH DESIGN

We constructed a State Transition Model to assess the costs and quality-adjusted life-years (QALYs) associated with either antihypertensive medication treatment or no-treatment using data from the REasons for Geographic and Racial Differences in Stroke (REGARDS) study and published literature. CVD events and health states considered in the model included stroke, coronary heart disease, heart failure, chronic kidney disease, and end-stage renal disease.

SUBJECTS

White and black adults with hypertension in the United States, 45 years of age and above.

MEASURES

Yearly risk of CVD was determined using REGARDS data and published literature. Antihypertensive medication costs were determined using Medicare claims. Event and health state costs were estimated from published literature. All costs were adjusted to 2012 US dollars. Effectiveness was assessed using QALYs.

RESULTS

Antihypertensive medication treatment was cost-saving and increased QALYs compared with no-treatment for white men ($7387; 1.14 QALYs), white women ($7796; 0.89 QALYs), black men ($8400; 1.66 QALYs), and black women ($10,249; 1.79 QALYs).

CONCLUSIONS

Antihypertensive medication treatment is cost-saving and increases QALYs for all groups considered in the model, particularly among black adults.

New Metrics for Economic Evaluation in the Presence of Heterogeneity: Focusing on Evaluating Policy Alternatives Rather than Treatment Alternatives

BACKGROUND

Cost-effectiveness analysis (CEA) methods fail to acknowledge that where cost-effectiveness differs across subgroups, there may be differential adoption of technology. Also, current CEA methods are not amenable to incorporating the impact of policy alternatives that potentially influence the adoption behavior. Unless CEA methods are extended to allow for a comparison of policies rather than simply treatments, their usefulness to decision makers may be limited.

METHODS

We conceptualize new metrics, which estimate the realized value of technology from policy alternatives, through introducing subgroup-specific adoption parameters into existing metrics, incremental cost-effectiveness ratios (ICERs) and Incremental Net Monetary Benefits (NMBs). We also provide the Loss with respect to Efficient Diffusion (LED) metrics, which link with existing value of information metrics but take a policy evaluation perspective. We illustrate these metrics using policies on treatment with combination therapy with a statin plus a fibrate v. statin monotherapy for patients with diabetes and mixed dyslipidemia.

RESULTS

Under the traditional approach, the population-level ICER of combination v. monotherapy was $46,000/QALY. However, after accounting for differential rates of adoption of the combination therapy (7.2% among males and 4.3% among females), the modified ICER was $41,733/QALY, due to the higher rate of adoption in the more cost-effective subgroup (male). The LED metrics showed that an education program to increase the uptake of combination therapy among males would provide the largest economic returns due to the significant underutilization of the combination therapy among males under the current policy.

CONCLUSION

This framework may have the potential to improve the decision-making process by producing metrics that are better aligned with the specific policy decisions under consideration for a specific technology.

A Review of Methods for Analysis of the Expected Value of Information

In recent years, value-of-information analysis has become more widespread in health economic evaluations, specifically as a tool to guide further research and perform probabilistic sensitivity analysis. This is partly due to methodological advancements allowing for the fast computation of a typical summary known as the expected value of partial perfect information (EVPPI). A recent review discussed some approximation methods for calculating the EVPPI, but as the research has been active over the intervening years, that review does not discuss some key estimation methods. Therefore, this paper presents a comprehensive review of these new methods. We begin by providing the technical details of these computation methods. We then present two case studies in order to compare the estimation performance of these new methods. We conclude that a method based on nonparametric regression offers the best method for calculating the EVPPI in terms of accuracy, computational time, and ease of implementation. This means that the EVPPI can now be used practically in health economic evaluations, especially as all the methods are developed in parallel with R functions and a web app to aid practitioners.

Understanding the Value of Individualized Information: The Impact of Poor Calibration or Discrimination in Outcome Prediction Models

BACKGROUND

Risk prediction models allow for the incorporation of individualized risk and clinical effectiveness information to identify patients for whom therapy is most appropriate and cost-effective. This approach has the potential to identify inefficient (or harmful) care in subgroups at different risks, even when the overall results appear favorable. Here, we explore the value of personalized risk information and the factors that influence it.

METHODS

Using an expected value of individualized care (EVIC) framework, which monetizes the value of customizing care, we developed a general approach to calculate individualized incremental cost effectiveness ratios (ICERs) as a function of individual outcome risk. For a case study (tPA v. streptokinase to treat possible myocardial infarction), we used a simulation to explore how an EVIC is influenced by population outcome prevalence, model discrimination (c-statistic) and calibration, and willingness-to-pay (WTP) thresholds.

RESULTS

In our simulations, for well-calibrated models, which do not over- or underestimate predicted v. observed event risk, the EVIC ranged from $0 to $700 per person, with better discrimination (higher c-statistic values) yielding progressively higher EVIC values. For miscalibrated models, the EVIC ranged from -$600 to $600 in different simulated scenarios. The EVIC values decreased as discrimination improved from a c-statistic of 0.5 to 0.6, before becoming positive as the c-statistic reached values of ~0.8.

CONCLUSIONS

Individualizing treatment decisions using risk may produce substantial value but also has the potential for net harm. Good model calibration ensures a non-negative EVIC. Improvements in discrimination generally increase the EVIC; however, when models are miscalibrated, greater discriminating power can paradoxically reduce the EVIC under some circumstances.

Burden of illness and research investments in translational sciences for pharmaceuticals in metastatic cancers

AIM

To explore whether investments in translational sciences for six metastatic cancers follow idiosyncratic returns to those investments rather than levels of burden of illness (BI).

METHODS

Associate the number of translational clinical trials in the USA involving oncolytic drugs approved during 2008-2013 and the level (in 2008) and changes (2002-2008 and 2008-2014) in cancer-specific years of life lost.

RESULTS

Investments in trials were positively associated only with contemporary changes in BI (2008-2014). The relationship was stronger for government-sponsored comparative-effectiveness trials than for industry.

CONCLUSION

Translational research investments follow anticipated changes to BI levels. Systematic quantification of these expected returns from specific investments can help guide investment decisions in translational health sciences and generate productive dialogue across stakeholders.

Sufficient trial size to inform clinical practice

Medical research has evolved conventions for choosing sample size in randomized clinical trials that rest on the theory of hypothesis testing. Bayesian statisticians have argued that trials should be designed to maximize subjective expected utility in settings of clinical interest. This perspective is compelling given a credible prior distribution on treatment response, but there is rarely consensus on what the subjective prior beliefs should be. We use Wald's frequentist statistical decision theory to study design of trials under ambiguity. We show that ε-optimal rules exist when trials have large enough sample size. An ε-optimal rule has expected welfare within ε of the welfare of the best treatment in every state of nature. Equivalently, it has maximum regret no larger than ε We consider trials that draw predetermined numbers of subjects at random within groups stratified by covariates and treatments. We report exact results for the special case of two treatments and binary outcomes. We give simple sufficient conditions on sample sizes that ensure existence of ε-optimal treatment rules when there are multiple treatments and outcomes are bounded. These conditions are obtained by application of Hoeffding large deviations inequalities to evaluate the performance of empirical success rules.

Calculating Partial Expected Value of Perfect Information via Monte Carlo Sampling Algorithms

Partial expected value of perfect information (EVPI) calculations can quantify the value of learning about particular subsets of uncertain parameters in decision models. Published case studies have used different computational approaches. This article examines the computation of partial EVPI estimates via Monte Carlo sampling algorithms. The mathematical definition shows 2 nested expectations, which must be evaluated separately because of the need to compute a maximum between them. A generalized Monte Carlo sampling algorithm uses nested simulation with an outer loop to sample parameters of interest and, conditional upon these, an inner loop to sample remaining uncertain parameters. Alternative computation methods and shortcut algorithms are discussed and mathematical conditions for their use considered. Maxima of Monte Carlo estimates of expectations are biased upward, and the authors show that the use of small samples results in biased EVPI estimates. Three case studies illustrate 1) the bias due to maximization and also the inaccuracy of shortcut algorithms 2) when correlated variables are present and 3) when there is nonlinearity in net benefit functions. If relatively small correlation or nonlinearity is present, then the shortcut algorithm can be substantially inaccurate. Empirical investigation of the numbers of Monte Carlo samples suggests that fewer samples on the outer level and more on the inner level could be efficient and that relatively small numbers of samples can sometimes be used. Several remaining areas for methodological development are set out. A wider application of partial EVPI is recommended both for greater understanding of decision uncertainty and for analyzing research priorities.

A Framework for Prioritizing Research Investments in Precision Medicine

INTRODUCTION

The adoption of precision medicine (PM) has been limited in practice to date, and yet its promise has attracted research investments. Developing foundational economic approaches for directing proper use of PM and stimulating growth in this area from multiple perspectives is thus quite timely.

METHODS

Building on our previously developed expected value of individualized care (EVIC) framework, we conceptualize new decision-relevant metrics to better understand and forecast the expected value of PM. Several aspects of behavior at the patient, physician, and payer levels are considered that can inform the rate and manner in which PM innovations diffuse throughout the relevant population. We illustrate this framework and the methods using a retrospective evaluation of the use of OncotypeDx genomic test among breast cancer patients.

RESULTS

The enriched metrics can help inform many facets of PM decision making, such as evaluating alternative reimbursement levels for PM tests, implementation and education programs for physicians and patients, and decisions around research investments by manufacturers and public entities. We replicated prior published results on evaluation of OncotypeDx among breast cancer patients but also illustrated that those results are based on assumptions that are often not met in practice. Instead, we show how incorporating more practical aspects of behavior around PM could lead to drastically different estimates of value.

CONCLUSION

We believe that the framework and the methods presented can provide decision makers with more decision-relevant tools to explore the value of PM. There is a growing recognition that data on adoption is important to decision makers. More research is needed to develop prediction models for potential diffusion of PM technologies.

Estimating the expected value of partial perfect information in health economic evaluations using integrated nested Laplace approximation

The Expected Value of Perfect Partial Information (EVPPI) is a decision‐theoretic measure of the ‘cost’ of parametric uncertainty in decision making used principally in health economic decision making. Despite this decision‐theoretic grounding, the uptake of EVPPI calculations in practice has been slow. This is in part due to the prohibitive computational time required to estimate the EVPPI via Monte Carlo simulations. However, recent developments have demonstrated that the EVPPI can be estimated by non‐parametric regression methods, which have significantly decreased the computation time required to approximate the EVPPI. Under certain circumstances, high‐dimensional Gaussian Process (GP) regression is suggested, but this can still be prohibitively expensive. Applying fast computation methods developed in spatial statistics using Integrated Nested Laplace Approximations (INLA) and projecting from a high‐dimensional into a low‐dimensional input space allows us to decrease the computation time for fitting these high‐dimensional GP, often substantially. We demonstrate that the EVPPI calculated using our method for GP regression is in line with the standard GP regression method and that despite the apparent methodological complexity of this new method, R functions are available in the package BCEA to implement it simply and efficiently. © 2016 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.

Cost collection and analysis for health economic evaluation

OBJECTIVE

To improve the understanding of common health care cost collection, estimation, analysis, and reporting methodologies.

DATA SOURCES

Ovid MEDLINE (1947 to December 2012), Cochrane Central register of Controlled Trials, Database of Systematic Reviews, Health Technology Assessment, and National Health Service Economic Evaluation Database.

REVIEW METHODS

This article discusses the following cost collection methods: defining relevant resources, quantification of consumed resources, and resource valuation. It outlines the recommendations for cost reporting in economic evaluations and reviews the techniques on how to handle cost data uncertainty. Last, it discusses the controversial topics of future costs and patient productivity losses.

CONCLUSION

Health care cost collection and estimation can be challenging, and an organized approach is required to optimize accuracy of economic evaluation outcomes.

IMPLICATIONS FOR PRACTICE

Understanding health care cost collection and estimation techniques will improve both critical appraisal and development of future economic evaluations.

Health economic evaluation: important principles and methodology

OBJECTIVES/HYPOTHESIS

To discuss health economic evaluation and improve the understanding of common methodology.

RESULTS

This article discusses the methodology for the following types of economic evaluations: cost-minimization, cost-effectiveness, cost-utility, cost-benefit, and economic modeling. Topics include health-state utility measures, the quality-adjusted life year (QALY), uncertainty analysis, discounting, decision tree analysis, and Markov modeling.

CONCLUSION

Economic evaluation is the comparative analysis of alternative courses of action in terms of both their costs and consequences. With increasing health care expenditure and limited resources, it is important for physicians to consider the economic impact of their interventions. Understanding common methodology involved in health economic evaluation will improve critical appraisal of the literature and optimize future economic evaluations.

Implementing a Monte‐Carlo simulation on admission decisions

Purpose

Although very significant and applicable, there have been no formal justifications for the use of Monte‐Carlo models and Markov chains in evaluating hospital admission decisions or concrete data supporting their use. For these reasons, this research was designed to provide a deeper understanding of these models. The purpose of this paper is to examine the usefulness of a computerized Monte‐Carlo simulation of admission decisions under the constraints of emergency departments.

Design/methodology/approach

The authors construct a simple decision tree using the expected utility method to represent the complex admission decision process terms of quality adjusted life years (QALY) then show the advantages of using a Monte‐Carlo simulation in evaluating admission decisions in a cohort simulation, using a decision tree and a Markov chain.

Findings

After showing that the Monte‐Carlo simulation outperforms an expected utility method without a simulation, the authors develop a decision tree with such a model. real cohort simulation data are used to demonstrate that the integration of a Monte‐Carlo simulation shows which patients should be admitted.

Research limitations/implications

This paper may encourage researchers to use Monte‐Carlo simulation in evaluating admission decision implications. The authors also propose applying the model when using a computer simulation that deals with various CVD symptoms in clinical cohorts.

Originality/value

Aside from demonstrating the value of a Monte‐Carlo simulation as a powerful analysis tool, the paper's findings may prompt researchers to conduct a decision analysis with a Monte‐Carlo simulation in the healthcare environment.

Diagnostic testing and treatment under ambiguity: using decision analysis to inform clinical practice

Partial knowledge of patient health status and treatment response is a pervasive concern in medical decision making. Clinical practice guidelines (CPGs) make recommendations intended to optimize patient care, but optimization typically is infeasible with partial knowledge. Decision analysis shows that a clinician's objective, knowledge, and decision criterion should jointly determine the care he prescribes. To demonstrate, this paper studies a common scenario regarding diagnostic testing and treatment. A patient presents to a clinician, who obtains initial evidence on health status. The clinician can prescribe a treatment immediately or he can order a test yielding further evidence that may be useful in predicting treatment response. In the latter case, he prescribes a treatment after observation of the test result. I analyze this scenario in three steps. The first poses a welfare function and characterizes optimal care. The second describes partial knowledge of response to testing and treatment that might realistically be available. The third considers decision criteria. I conclude with reconsideration of clinical practice guidelines.

A systematic and critical review of the evolving methods and applications of value of information in academia and practice

OBJECTIVE

This article provides a systematic and critical review of the evolving methods and applications of value of information (VOI) in academia and practice and discusses where future research needs to be directed.

METHODS

Published VOI studies were identified by conducting a computerized search on Scopus and ISI Web of Science from 1980 until December 2011 using pre-specified search terms. Only full-text papers that outlined and discussed VOI methods for medical decision making, and studies that applied VOI and explicitly discussed the results with a view to informing healthcare decision makers, were included. The included papers were divided into methodological and applied papers, based on the aim of the study.

RESULTS

A total of 118 papers were included of which 50 % (n = 59) are methodological. A rapidly accumulating literature base on VOI from 1999 onwards for methodological papers and from 2005 onwards for applied papers is observed. Expected value of sample information (EVSI) is the preferred method of VOI to inform decision making regarding specific future studies, but real-life applications of EVSI remain scarce. Methodological challenges to VOI are numerous and include the high computational demands, dealing with non-linear models and interdependency between parameters, estimations of effective time horizons and patient populations, and structural uncertainties.

CONCLUSION

VOI analysis receives increasing attention in both the methodological and the applied literature bases, but challenges to applying VOI in real-life decision making remain. For many technical and methodological challenges to VOI analytic solutions have been proposed in the literature, including leaner methods for VOI. Further research should also focus on the needs of decision makers regarding VOI.

Private manufacturers' thresholds to invest in comparative effectiveness trials

The recent rush of enthusiasm for public investment in comparative effectiveness research (CER) in the US has focussed attention on these public investments. However, little attention has been given to how changing public investment in CER may affect private manufacturers' incentives for CER, which has long been a major source of CER. In this work, based on a simple revenue maximizing economic framework, we generate predictions on thresholds to invest in CER for a private manufacturer that compares its own product to a competitor's product in head-to-head trials. Our analysis shows that private incentives to invest in CER are determined by how the results of CER may affect the price and quantity of the product sold and the duration over which resulting changes in revenue would accrue, given the time required to complete CER and the time from the completion of CER to the time of patent expiration. We highlight the result that private incentives may often be less than public incentives to invest in CER and may even be negative if the likelihood of adverse findings is sufficient. We find that these incentives imply a number of predictions about patterns of CER and how they will be affected by changes in public financing of CER and CER methods. For example, these incentives imply that incumbent patent holders may be less likely to invest in CER than entrants and that public investments in CER may crowd out similar private investments. In contrast, newer designs and methods for CER, such as Bayesian adaptive trials, which can reduce ex post risk of unfavourable results and shorten the time for the production of CER, may increase the expected benefits of CER and may tend to increase private investment in CER as long as the costs of such innovative designs are not excessive. Bayesian approaches to design also naturally highlight the dynamic aspects of CER, allowing less expensive initial studies to guide decisions about future investments and thereby encouraging greater initial investments in CER. However, whether the potential effects we highlight of public funding of CER and of Bayesian approaches to trial design actually produce changes in private investment in CER remains an empirical question.

Prioritization in comparative effectiveness research: the CANCERGEN Experience

BACKGROUND

Systematic approaches to stakeholder-informed research prioritization are a central focus of comparative effectiveness research. Genomic testing in cancer is an ideal area to refine such approaches given rapid innovation and potentially significant impacts on patient outcomes.

OBJECTIVE

To develop and pilot test a stakeholder-informed approach to prioritizing genomic tests for future study in collaboration with the cancer clinical trials consortium SWOG.

METHODS

We conducted a landscape analysis to identify genomic tests in oncology using a systematic search of published and unpublished studies, and expert consultation. Clinically valid tests suitable for evaluation in a comparative study were presented to an external stakeholder group. Domains to guide the prioritization process were identified with stakeholder input, and stakeholders ranked tests using multiple voting rounds.

RESULTS

A stakeholder group was created including representatives from patient-advocacy groups, payers, test developers, regulators, policy makers, and community-based oncologists. We identified 9 domains for research prioritization with stakeholder feedback: population impact; current standard of care, strength of association; potential clinical benefits, potential clinical harms, economic impacts, evidence of need, trial feasibility, and market factors. The landscape analysis identified 635 studies; of 9 tests deemed to have sufficient clinical validity, 6 were presented to stakeholders. Two tests in lung cancer (ERCC1 and EGFR) and 1 test in breast cancer (CEA/CA15-3/CA27.29) were identified as top research priorities.

CONCLUSIONS

Use of a diverse stakeholder group to inform research prioritization is feasible in a pragmatic and timely manner. Additional research is needed to optimize search strategies, stakeholder group composition, and integration with existing prioritization mechanisms.

Economic analysis of nutrition interventions for chronic disease prevention: methods, research, and policy

Increased interest in the potential societal benefit of incorporating health economics as a part of clinical translational science, particularly nutrition interventions, led the Office of Dietary Supplements at the National Institutes of Health to sponsor a conference to address key questions about the economic analysis of nutrition interventions to enhance communication among health economic methodologists, researchers, reimbursement policy makers, and regulators. Issues discussed included the state of the science, such as what health economic methods are currently used to judge the burden of illness, interventions, or healthcare policies, and what new research methodologies are available or needed to address knowledge and methodological gaps or barriers. Research applications included existing evidence-based health economic research activities in nutrition that are ongoing or planned at federal agencies. International and US regulatory, policy, and clinical practice perspectives included a discussion of how research results can help regulators and policy makers within government make nutrition policy decisions, and how economics affects clinical guideline development.

Minimal modeling approaches to value of information analysis for health research

Value of information (VOI) techniques can provide estimates of the expected benefits from clinical research studies that can inform decisions about the design and priority of those studies. Most VOI studies use decision-analytic models to characterize the uncertainty of the effects of interventions on health outcomes, but the complexity of constructing such models can pose barriers to some practical applications of VOI. However, because some clinical studies can directly characterize uncertainty in health outcomes, it may sometimes be possible to perform VOI analysis with only minimal modeling. This article 1) develops a framework to define and classify minimal modeling approaches to VOI, 2) reviews existing VOI studies that apply minimal modeling approaches, and 3) illustrates and discusses the application of the minimal modeling to 2 new clinical applications to which the approach appears well suited because clinical trials with comprehensive outcomes provide preliminary estimates of the uncertainty in outcomes. The authors conclude that minimal modeling approaches to VOI can be readily applied in some instances to estimate the expected benefits of clinical research.

Estimating expected value of sample information for incomplete data models using Bayesian approximation

Expected value of sample information (EVSI) involves simulating data collection, Bayesian updating, and reexamining decisions. Bayesian updating in incomplete data models typically requires Markov chain Monte Carlo (MCMC). This article describes a revision to a form of Bayesian Laplace approximation for EVSI computation to support decisions in incomplete data models. The authors develop the approximation, setting out the mathematics for the likelihood and log posterior density function, which are necessary for the method. They compare the accuracy of EVSI estimates in a case study cost-effectiveness model using first- and second-order versions of their approximation formula and traditional Monte Carlo. Computational efficiency gains depend on the complexity of the net benefit functions, the number of inner-level Monte Carlo samples used, and the requirement or otherwise for MCMC methods to produce the posterior distributions. This methodology provides a new and valuable approach for EVSI computation in health economic decision models and potential wider benefits in many fields requiring Bayesian approximation.

The relationship between economic evaluations and HIV and AIDS treatment policies

PURPOSE OF REVIEW

Economics, and specifically economic evaluations, are increasingly being utilized to provide treatment policy guidance to decision makers. This article reviews work that has contributed to understanding of the relationship.

RECENT FINDINGS

There is a paucity of research explicitly investigating the association between economic evaluations and HIV and AIDS treatment policy. Where it does exist, it is weak. Factors contributing to the limited impact include lack of reliable and trusted data; absence of local cost-effectiveness data for different interventions; contradictory results; challenges associated with understanding complex economic/mathematical models; inefficient implementation of HIV and AIDS policies; inability to pursue long-term health planning needs; and political will.

SUMMARY

Consideration of the ways in which economic evaluations can have greater influence over HIV and AIDS policies is needed. The weak relationship between the two reflects the complicated and multifaceted decision-making process that is often influenced by socioeconomic and political factors. If an economic evaluation is to influence policy, then cognizance of this is important. Extending the economic toolkit to include broader-based models that incorporate political economy variables, but do not compromise on comprehension, validity and robustness, will offer better informed policy recommendations.

Implications of an economic model of health states worse than dead

We introduce a formal definition of health equivalent to dead into a standard model to develop previously unrecognized insights. We find that the health state viewed as equivalent to dead will depend on an individual's health prognosis, probability of survival, and rate of time preference. Our work on maximum endurable time shows that using QALY scores based on long-run preferences to value health states that last for shorter periods can alter cardinal and ordinal valuations. Simulations show that errors of substantial magnitude in QALY scores can consequently result. We describe situations where biases are likely and identify possible corrections.

Social science insights into improving workforce effectiveness: examples from the developing field of hospital medicine

The translation of insights from the biological sciences to medical practice requires actions by clinicians, patients, and others involved in healthcare. This makes insights from the social sciences critical to improving medical care. The recent emergence of hospitalists in the United States--physicians who specialize in the care of hospitalized patients--is an important innovation in how biomedical knowledge is translated into clinical care. This article discusses work by my colleagues and me examining the emergence of the hospitalist model by using the tools of the social sciences to understand whether, and under what conditions, hospitalists reduce the costs and improve the outcomes of care, and developing tools to measure and improve the quality of hospital care. The social scientific concepts and tools that we have drawn upon reflect a wide range of the social sciences, including economics, sociology, psychology, and related fields, such as clinical epidemiology and program evaluation. Many of these issues we have examined, including how professionals learn from experience and from their peers and how to measure and reward productivity, have important potential to address challenges faced by the public health workforce and reflect the broad potential for insights from the social sciences to inform public health workforce policy.

The economics of comparative effectiveness studies: societal and private perspectives and their implications for prioritizing public investments in comparative effectiveness research

Comparative effectiveness research (CER) can provide valuable information for patients, providers and payers. These stakeholders differ in their incentives to invest in CER. To maximize benefits from public investments in CER, it is important to understand the value of CER from the perspectives of these stakeholders and how that affects their incentives to invest in CER. This article provides a conceptual framework for valuing CER, and illustrates the potential benefits of such studies from a number of perspectives using several case studies. We examine cases in which CER provides value by identifying when one treatment is consistently better than others, when different treatments are preferred for different subgroups, and when differences are small enough that decisions can be made based on price. We illustrate these findings using value-of-information techniques to assess the value of research, and by examining changes in pharmaceutical prices following publication of a comparative effectiveness study. Our results suggest that CER may have high societal value but limited private return to providers or payers. This suggests the importance of public efforts to promote the production of CER. We also conclude that value-of-information tools may help inform policy decisions about how much public funds to invest in CER and how to prioritize the use of available public funds for CER, in particular targeting public CER spending to areas where private incentives are low relative to social benefits.

Comparative effectiveness research for antipsychotic medications: how much is enough?

Second-generation antipsychotics have attracted practitioners' and policy-makers' attention, because of concerns over their health effects and costs. Comparative effectiveness data from the Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE)-a high-profile National Institutes of Health (NIH)-funded study-have been used to argue for restricting coverage for these costly drugs. But concerns about the design of CATIE and its associated cost-effectiveness analysis and uncertainty about the precision of these findings raise questions about this interpretation. Our work suggests that additional research to increase the precision of comparisons of the effectiveness of antipsychotics would be well worth the cost.

Setting priorities for research: a practical application of 'payback' and expected value of information

BACKGROUND

Setting priorities for research using economic in addition to scientific criteria can ensure that resources are spent efficiently and equitably.

OBJECTIVE

This study applies two priority setting methods 'payback' and expected value of information (EVI) to two research areas (osteoporosis and pressure ulcers) and where appropriate to four clinical trials: the Record Trial, the Vitamin D and Calcium Trial and the Hip Protector Trial (osteoporosis), and the Pressure Trial (wound care).

METHODS

Two decision-analytic models were developed. For 'payback', the PATHS model was used to estimate the expected net benefits of conducting the four clinical trials. An EVI framework was applied to estimate the cost-effectiveness of conducting further research in the two disease areas investigated.

RESULTS

The application of 'payback' suggests that the Record Trial and the Vitamin D and Calcium Trial would be cost-effective. The Hip Protector and the Pressure Ulcer Trial are cost-effective under certain assumptions concerning the likelihood of obtaining positive, negative or inconclusive results. The EVI method suggests that research would be potentially cost-effective in these areas in the populations considered.

CONCLUSION

EVI provides strategic information for setting priorities for research between disease areas and study populations. 'Payback' provides information on the cost-effectiveness of specific research designs. However, further work in this area, particularly concerning the issue of implementation of research, is required.

Expected value of sample information for Weibull survival data

Expected value of sample information (EVSI) involves simulating data collection, Bayesian updating, and re-examining decisions. Bayesian updating in Weibull models typically requires Markov chain Monte Carlo (MCMC). We examine five methods for calculating posterior expected net benefits: two heuristic methods (data lumping and pseudo-normal); two Bayesian approximation methods (Tierney & Kadane, Brennan & Kharroubi); and the gold standard MCMC. A case study computes EVSI for 25 study options. We compare accuracy, computation time and trade-offs of EVSI versus study costs. Brennan & Kharroubi (B&K) approximates expected net benefits to within +/-1% of MCMC. Other methods, data lumping (+54%), pseudo-normal (-5%) and Tierney & Kadane (+11%) are less accurate. B&K also produces the most accurate EVSI approximation. Pseudo-normal is also reasonably accurate, whilst Tierney & Kadane consistently underestimates and data lumping exhibits large variance. B&K computation is 12 times faster than the MCMC method in our case study. Though not always faster, B&K provides most computational efficiency when net benefits require appreciable computation time and when many MCMC samples are needed. The methods enable EVSI computation for economic models with Weibull survival parameters. The approach can generalize to complex multi-state models and to survival analyses using other smooth parametric distributions.

Methods for determining cost-benefit ratios for pharmaceuticals in Germany

The aim of this methodological paper is to summarize evidence on how to implement cost-benefit assessment according to the new German legislative framework (Competition Enhancement Act). Given the complexity of existing health policy frameworks within industrialised countries in adapting health economics in their respective regulatory scheme, no clear international scientific consensus on which health economic methods should be chosen for assessment can be determined. Nevertheless, a broad consensus on the internal properties of methods itself can be found. Based on these common international standards in methodology, this work provides a minimum catalogue of methods and criteria that meet legal and local German requirements with regard to specific factors of its health care system. Aside from categorising clearly defined standards (e.g., study forms, cost and benefit categories) the suggested catalogue specifies some intensively debated areas in Germany (e.g., the QALY, modelling, the perspective used in the assessment). After the proposition of certain methods the paper leads to a first recommendation of a detailed assessment-process itself specific for the German way in implementing cost-benefit ratios within regulatory decision making in Germany.

Efficient computation of partial expected value of sample information using Bayesian approximation

We describe a novel process for transforming the efficiency of partial expected value of sample information (EVSI) computation in decision models. Traditional EVSI computation begins with Monte Carlo sampling to produce new simulated data-sets with a specified sample size. Each data-set is synthesised with prior information to give posterior distributions for model parameters, either via analytic formulae or a further Markov Chain Monte Carlo (MCMC) simulation. A further 'inner level' Monte Carlo sampling then quantifies the effect of the simulated data on the decision. This paper describes a novel form of Bayesian Laplace approximation, which can be replace both the Bayesian updating and the inner Monte Carlo sampling to compute the posterior expectation of a function. We compare the accuracy of EVSI estimates in two case study cost-effectiveness models using 1st and 2nd order versions of our approximation formula, the approximation of Tierney and Kadane, and traditional Monte Carlo. Computational efficiency gains depend on the complexity of the net benefit functions, the number of inner level Monte Carlo samples used, and the requirement or otherwise for MCMC methods to produce the posterior distributions. This methodology provides a new and valuable approach for EVSI computation in health economic decision models and potential wider benefits in many fields requiring Bayesian approximation.

Cost-effectiveness of peginterferon alfa-2a (40kDa) plus ribavirin in patients with HIV and hepatitis C virus co-infection

BACKGROUND

A multinational trial (APRICOT) showed that peginterferon alfa-2a (40kDa) plus ribavirin is efficacious for treatment of HIV-HCV co-infection. The cost-effectiveness of treating these patients with peginterferon alfa-2a/ribavirin has yet to be explored from a US societal perspective.

OBJECTIVE

To predict the cost-effectiveness of peginterferon alfa-2a/ribavirin with interferon/ribavirin (IFN/RBV) or no treatment in HIV-HCV co-infected patients.

STUDY DESIGN

A Markov model was constructed with liver progression estimates based on published literature. Sustained virological response and baseline characteristics of the reference case were based on APRICOT. Quality of life and costs in 2004 US dollars (US$) were based on literature estimates and discounted at 3%.

RESULTS

Peginterferon alfa-2a/ribavirin compared with IFN/RBV or no treatment is predicted to increase quality-adjusted life-years (QALYs) by 0.73 and 0.94 years, respectively, in HCV-genotype-1 patients. The incremental cost-effectiveness ratio of peginterferon alfa-2a/ribavirin compared with IFN/RBV and no treatment for all patients is respectively US$ 2,082 and 5,187/QALY gained.

CONCLUSIONS

Anti-HCV treatment is predicted to decrease the risk of cirrhosis and increase quality-adjusted survival of HIV-HCV co-infected patients compared with IFN/RBV and no treatment. Peginterferon alfa-2a/ribavirin's cost per QALY gained relative to these options falls within the cost-effectiveness level of many health technologies commonly adopted in the US.

Incorporating direct and indirect evidence using bayesian methods: an applied case study in ovarian cancer

OBJECTIVE

To demonstrate the application of a Bayesian mixed treatment comparison (MTC) model to synthesize data from clinical trials to inform decisions based on all relevant evidence.

METHODS

The value of an MTC model is demonstrated using a probabilistic decision-analytic model developed to assess the cost-effectiveness of second-line chemotherapy in ovarian cancer. Three clinical trials were found that each made a different pair-wise comparison of three treatments of interest in the overall patient population. As no common comparator existed between the three trials, an MTC model was used to assess the combined weight of evidence on survival from all three trials simultaneously. This analysis was compared to an alternative approach that combined two of the trials to make the same comparison of all three treatments using a common comparator, and an informal approach that did not synthesize the available evidence.

RESULTS

By including all three trials using an MTC model, the credible intervals around estimated overall survival were reduced compared with making the same comparison using only two trials and a common comparator. Nevertheless, the survival estimates from the MTC model result in greater uncertainty around the optimal treatment strategy at a cost-effectiveness threshold of 30,000 pounds per quality-adjusted life-year.

CONCLUSIONS

MTC models can be used to combine more data than would typically be included in a traditional meta-analysis that relies on a common comparator. They can formally quantify the combined uncertainty from all available evidence, and can be conducted using the same analytical approaches as standard meta-analyses.

Estimating transitions between symptom severity states over time in schizophrenia: a Bayesian meta-analytic approach

We obtain the posterior predictive distribution of transition probabilities between symptom severity states over time for patients with schizophrenia by (i) employing a Bayesian meta-analysis of published clinical trials and observational studies to estimate the posterior distribution of parameters that guide changes in Positive and Negative Syndrome Scale (PANSS) scores over time and under the influence of various drugs and (ii) by propagating the variability from the posterior distributions of the parameters through a micro-simulation model that is formulated based on schizophrenia progression. Results show detailed differences among haloperidol, risperidone and olanzapine in controlling various levels of severities of positive, negative and joint symptoms over time. For example, risperidone seems best in controlling severe positive symptoms while olanzapine is the worst in that during the first quarter of drug treatment; however, olanzapine seems to be best in controlling severe negative symptoms across all four quarters of treatment while haloperidol is the worst in this regard. These details may further serve to better estimate quality of life of patients and aid in resource utilization decisions in treating schizophrenic patients. In addition, consistent estimation of uncertainty in the time-profile parameters also has important implications for the practice of cost-effectiveness analysis and for future resource allocation policies in schizophrenia treatment.

Optimal allocation of resources over health care programmes: dealing with decreasing marginal utility and uncertainty

This paper addresses the problem of how to value health care programmes with different ratios of costs to effects, specifically when taking into account that these costs and effects are uncertain. First, the traditional framework of maximising health effects with a given health care budget is extended to a flexible budget using a value function over money and health effects. Second, uncertainty surrounding costs and effects is included in the model using expected utility. Other approaches to uncertainty that do not specify a utility function are discussed and it is argued that these also include implicit notions about risk attitude.

Setting priorities for research

Setting priorities for research should be conducted in order to make the most efficient use of scarce resources. Yet the uptake in practice of such methods by researchers and commissioners of research alike has been slow, in part because the methodologies available to do so have not been widely disseminated. This paper argues that an appropriate priority-setting methodology should meet the objectives of the health system, that is to provide the most health benefits to the population that it serves within the budget constraint and while respecting equity considerations. A condition for these criteria to be met is to construct and operationalise an appropriate definition of the value of research. Five different ways that have been used in practice to value research and set priorities were reviewed. Shortcomings in the ways research is valued make it unlikely that the application of subjective methods, burden of disease methods, and clinical variations and payback methods meet the objectives of the health system. Using the fifth method, value of information, priority-setting can meet the objectives of the health system because it expresses the value of research using the same overall cost-effectiveness framework that is employed for decisions on service provision. However, this method still requires further work to evaluate how research outcomes can then be communicated to clinical practitioners and how practitioners can be encouraged to implement them.

Probabilistic sensitivity analysis for NICE technology assessment: not an optional extra

Recently the National Institute for Clinical Excellence (NICE) updated its methods guidance for technology assessment. One aspect of the new guidance is to require the use of probabilistic sensitivity analysis with all cost-effectiveness models submitted to the Institute. The purpose of this paper is to place the NICE guidance on dealing with uncertainty into a broader context of the requirements for decision making; to explain the general approach that was taken in its development; and to address each of the issues which have been raised in the debate about the role of probabilistic sensitivity analysis in general. The most appropriate starting point for developing guidance is to establish what is required for decision making. On the basis of these requirements, the methods and framework of analysis which can best meet these needs can then be identified. It will be argued that the guidance on dealing with uncertainty and, in particular, the requirement for probabilistic sensitivity analysis, is justified by the requirements of the type of decisions that NICE is asked to make. Given this foundation, the main issues and criticisms raised during and after the consultation process are reviewed. Finally, some of the methodological challenges posed by the need fully to characterise decision uncertainty and to inform the research agenda will be identified and discussed.

Future challenges for the economic evaluation of healthcare: patient preferences, risk attitudes and beyond

The continued growth in the economic evaluation of healthcare over the past 25 years has led to a shortage of trained health economists globally, leading to a number of universities and/or national governments developing specialised health economics programmes to train more health economists. One of the common problems with many of these training programmes is that they only educate new health economists to the Masters level, and as such they are unable to cover the many skills needed by a successful health economist. Furthermore, government and industry interests have ensured that economic evaluation is a heavily regulated environment that gives little incentive to seek further education. These two related factors (under-education and over-regulation) have lead to a situation where economic evaluation methods may adversely limit innovation of therapeutics and devices in clinical areas that perform badly when evaluated on the cost per QALY scale. The good news, however, is that the tide is turning and theoretically sound adjustments (such as risk adjustments and stated preferences) to the current paradigm are now being considered. This, of cause, is just the tip of the iceberg with other important issues such as time preference and the endogeneity of preference remaining very much under-researched areas in health. This paper concludes that many of these real-world issues, such as patient preferences, can be avoided by using artificial objective functions such as cost per QALY, but this comes at the cost of irrelevance and the misallocation of resources. If we are to meet all of the future challenges in economic evaluation in healthcare then we must focus more on advanced education and far less on the regulation of health economists.