Cost-Effectiveness Uncertainty Analysis Methods: A Comparison of One-Way Sensitivity, Analysis of Covariance, and Expected Value of Partial Perfect Information

Methods to Quantify the Importance of Parameters for Model Updating and Distributional Adaptation

PURPOSE

Decision models are time-consuming to develop; therefore, adapting previously developed models for new purposes may be advantageous. We provide methods to prioritize efforts to 1) update parameter values in existing models and 2) adapt existing models for distributional cost-effectiveness analysis (DCEA).

METHODS

Methods exist to assess the influence of different input parameters on the results of a decision models, including value of information (VOI) and 1-way sensitivity analysis (OWSA). We apply 1) VOI to prioritize searches for additional information to update parameter values and 2) OWSA to prioritize searches for parameters that may vary by socioeconomic characteristics. We highlight the assumptions required and propose metrics that quantify the extent to which parameters in a model have been updated or adapted. We provide R code to quickly carry out the analysis given inputs from a probabilistic sensitivity analysis (PSA) and demonstrate our methods using an oncology case study.

RESULTS

In our case study, updating 2 of 21 probabilistic model parameters addressed 71.5% of the total VOI and updating 3 addressed approximately 100% of the uncertainty. Our proposed approach suggests that these are the 3 parameters that should be prioritized. For model adaptation for DCEA, 46.3% of the total OWSA variation came from a single parameter, while the top 10 input parameters were found to account for more than 95% of the total variation, suggesting efforts should be aimed toward these.

CONCLUSIONS

These methods offer a systematic approach to guide research efforts in updating models with new data or adapting models to undertake DCEA. The case study demonstrated only very small gains from updating more than 3 parameters or adapting more than 10 parameters.

HIGHLIGHTS

It can require considerable analyst time to search for evidence to update a model or to adapt a model to take account of equity concerns.In this article, we provide a quantitative method to prioritze parameters to 1) update existing models to reflect potential new evidence and 2) adapt existing models to estimate distributional outcomes.We define metrics that quantify the extent to which the parameters in a model have been updated or adapted.We provide R code that can quickly rank parameter importance and calculate quality metrics using only the results of a standard probabilistic sensitivity analysis.

Uncertainty Analysis in Intervention Impact on Health Inequality for Resource Allocation Decisions

Cost-effectiveness analysis, routinely used in health care to inform funding decisions, can be extended to consider impact on health inequality. Distributional cost-effectiveness analysis (DCEA) incorporates socioeconomic differences in model parameters to capture how an intervention would affect both overall population health and differences in health between population groups. In DCEA, uncertainty analysis can consider the decision uncertainty around on both impacts (i.e., the probability that an intervention will increase overall health and the probability that it will reduce inequality). Using an illustrative example assessing smoking cessation interventions (2 active interventions and a “no-intervention” arm), we demonstrate how the uncertainty analysis could be conducted in DCEA to inform policy recommendations. We perform value of information (VOI) analysis and analysis of covariance (ANCOVA) to identify what additional evidence would add most value to the level of confidence in the DCEA results. The analyses were conducted for both national and local authority-level decisions to explore whether the conclusions about decision uncertainty based on the national-level estimates could inform local policy. For the comparisons between active interventions and “no intervention,” there was no uncertainty that providing the smoking cessation intervention would increase overall health but increase inequality. However, there was uncertainty in the direction of both impacts when comparing between the 2 active interventions. VOI and ANCOVA show that uncertainty in socioeconomic differences in intervention effectiveness and uptake contributes most to the uncertainty in the DCEA results. This suggests potential value of collecting additional evidence on intervention-related inequalities for this evaluation. We also found different levels of decision uncertainty between settings, implying that different types and levels of additional evidence are required for decisions in different localities.

Comparison of Decision Modeling Approaches for Health Technology and Policy Evaluation

We discuss tradeoffs and errors associated with approaches to modeling health economic decisions. Through an application in pharmacogenomic (PGx) testing to guide drug selection for individuals with a genetic variant, we assessed model accuracy, optimal decisions, and computation time for an identical decision scenario modeled 4 ways: using 1) coupled-time differential equations (DEQ), 2) a cohort-based discrete-time state transition model (MARKOV), 3) an individual discrete-time state transition microsimulation model (MICROSIM), and 4) discrete event simulation (DES). Relative to DEQ, the net monetary benefit for PGx testing (v. a reference strategy of no testing) based on MARKOV with rate-to-probability conversions using commonly used formulas resulted in different optimal decisions. MARKOV was nearly identical to DEQ when transition probabilities were embedded using a transition intensity matrix. Among stochastic models, DES model outputs converged to DEQ with substantially fewer simulated patients (1 million) v. MICROSIM (1 billion). Overall, properly embedded Markov models provided the most favorable mix of accuracy and runtime but introduced additional complexity for calculating cost and quality-adjusted life year outcomes due to the inclusion of "jumpover" states after proper embedding of transition probabilities. Among stochastic models, DES offered the most favorable mix of accuracy, reliability, and speed.

Cost-Effectiveness of Routine Screening for Autoimmune Encephalitis in Patients With First-Episode Psychosis in the United States

OBJECTIVE

Autoimmune encephalitis (AE) is a highly treatable neurologic condition that can cause psychosis. Screening for AE is not currently recommended in routine workup for first-episode psychosis (FEP), owing partly to the high cost of testing for AE-associated neuronal autoantibodies.

METHODS

This study used a decision-analytic model to estimate the cost-effectiveness of routine serum screening for AE compared with clinically targeted screening in patients with FEP. Model parameters drawn from prior published literature included the prevalence of neuronal autoantibodies in FEP (4.5%), serum autoantibody panel cost (US $291), remission probability with antipsychotics (0.58), and remission probability with immunotherapy for patients diagnosed with AE (0.85). Outcomes included quality-adjusted life-years (QALYs), costs, and incremental cost-effectiveness ratios (ICERs), assessed over a 5-year horizon from the US health care sector and societal perspectives. ICER thresholds of $50,000/QALY to $150,000/QALY were used to define cost-effectiveness. The analysis was conducted between June 2018 and January 2020.

RESULTS

Routine screening led to mean QALY gains of 0.008 among all patients and 0.174 among the subgroup of patients with neuronal autoantibodies. Mean costs increased by $780 from a societal perspective and $1,150 from a health care sector perspective, resulting in ICERs of $99,330/QALY and $147,460/QALY, respectively. Incorporating joint input data uncertainty, the likelihood routine screening has an ICER ≤ $150,000/QALY was 55% from a societal perspective and 37% from a health care sector perspective. The model parameter with the greatest contribution to overall uncertainty was the effectiveness of immunotherapy relative to antipsychotics.

CONCLUSIONS

Routine screening for AE in patients with FEP may be cost-effective in the United States. As further immunotherapy effectiveness data become available, a more definitive recommendation to perform routine screening could be warranted.

The Cost-Effectiveness of Bimodal Stimulation Compared to Unilateral and Bilateral Cochlear Implant Use in Adults with Bilateral Severe to Profound Deafness

OBJECTIVES

An increasing number of severe-profoundly deaf adult unilateral cochlear implant (CI) users receive bimodal stimulation; that is, they use a conventional acoustic hearing aid (HA) in their nonimplanted ear. The combination of electric and contralateral acoustic hearing provides additional benefits to hearing and also to general health-related quality of life compared with unilateral CI use. Bilateral CI is a treatment alternative to both unilateral CI and bimodal stimulation in some healthcare systems. The objective of this study was to conduct an economic evaluation of bimodal stimulation compared with other management options for adults with bilateral severe to profound deafness.

DESIGN

The economic evaluation took the form of a cost-utility analysis and compared bimodal stimulation (CI+HA) to two treatment alternatives: unilateral and bilateral CI. The analysis used a public healthcare system perspective based on data from the United Kingdom and the United States. Costs and health benefits were identified for both alternatives and estimated across a patient's lifetime using Markov state transition models. Utilities were based on Health Utilities Index estimates, and health outcomes were expressed in Quality Adjusted Life Years (QALYs). The results were presented using the Incremental Cost-Effectiveness Ratio and the Net Monetary Benefit approach to determine the cost-effectiveness of bimodal stimulation. Probabilistic sensitivity analyses explored the degree of overall uncertainty using Monte Carlo simulation. Deterministic sensitivity analyses and analysis of covariance identified parameters to which the model was most sensitive; that is, whose values had a strong influence on the intervention that was determined to be most cost-effective. A value of information analysis was performed to determine the potential value to be gained from additional research on bimodal stimulation.

RESULTS

The base case model showed that bimodal stimulation was the most cost-effective treatment option with a decision certainty of 72 and 67% in the United Kingdom and United States, respectively. Despite producing more QALYs than either unilateral CI or bimodal stimulation, bilateral CI was found not to be cost-effective because it was associated with excessive costs. Compared with unilateral CI, the increased costs of bimodal stimulation were outweighed by the gain in quality of life. Bimodal stimulation was found to cost an extra £174 per person in the United Kingdom ($937 in the US) and yielded an additional 0.114 QALYs compared with unilateral CI, resulting in an Incremental Cost-Effectiveness Ratio of £1521 per QALY gained in the United Kingdom ($8192/QALY in the United States). The most influential variable was the utility gained from the simultaneous use of both devices (CI+HA) compared with Unilateral CI. The value of further research was £4,383,922 at £20,000/QALY ($86,955,460 at $50,000/QALY in the United States).

CONCLUSIONS

This study provides evidence of the most cost-effective treatment alternative for adults with bilateral severe to profound deafness from publicly funded healthcare perspectives of the United Kingdom and United States. Bimodal stimulation was found to be more cost-effective than unilateral and bilateral CI across a wide range of willingness-to-pay thresholds. If there is scope for future research, conducting interventional designs to obtain utilities for bimodal stimulation compared with unilateral CI would reduce decision uncertainty considerably.

The Cost-Effectiveness of Cognitive Behavioral Therapy Versus Second-Generation Antidepressants for Initial Treatment of Major Depressive Disorder in the United States: A Decision Analytic Model

Background

Most guidelines for major depressive disorder recommend initial treatment with either a second-generation antidepressant (SGA) or cognitive behavioral therapy (CBT). Although most trials suggest that these treatments have similar efficacy, their health economic implications are uncertain.

Objective

To quantify the cost-effectiveness of CBT versus SGA for initial treatment of depression.

Design

Decision analytic model.

Data Sources

Relative effectiveness data from a meta-analysis of randomized controlled trials; additional clinical and economic data from other publications.

Target Population

Adults with newly diagnosed major depressive disorder in the United States.

Time Horizon

1 to 5 years.

Perspectives

Health care sector and societal.

Intervention

Initial treatment with either an SGA or group and individual CBT.

Outcome Measures

Costs in 2014 U.S. dollars, quality-adjusted life-years (QALYs), and incremental cost-effectiveness ratios.

Results of Base-Case Analysis

In model projections, CBT produced higher QALYs (3 days more at 1 year and 20 days more at 5 years) with higher costs at 1 year (health care sector, $900; societal, $1500) but lower costs at 5 years (health care sector, -$1800; societal, -$2500).

Results of Sensitivity Analysis

In probabilistic sensitivity analyses, SGA had a 64% to 77% likelihood of having an incremental cost-effectiveness ratio of $100 000 or less per QALY at 1 year; CBT had a 73% to 77% likelihood at 5 years. Uncertainty in the relative risk for relapse of depression contributed the most to overall uncertainty in the optimal treatment.

Limitation

Long-term trials comparing CBT and SGA are lacking.

Conclusion

Neither SGAs nor CBT provides consistently superior cost-effectiveness relative to the other. Given many patients' preference for psychotherapy over pharmacotherapy, increasing patient access to CBT may be warranted.

Primary Funding Source

Department of Veterans Affairs, National Institute of Mental Health.

Cost-effectiveness of scaling-up HCV prevention and treatment in the United States for people who inject drugs

AIMS

To examine the cost-effectiveness of hepatitis C virus (HCV) treatment of people who inject drugs (PWID), combined with medication-assisted treatment (MAT) and syringe-service programs (SSP), to tackle the increasing HCV epidemic in the United States.

DESIGN

HCV transmission and disease progression models with cost-effectiveness analysis using a health-care perspective.

SETTING

Rural Perry County, KY (PC) and urban San Francisco, CA (SF), USA. Compared with PC, SF has a greater proportion of PWID with access to MAT or SSP. HCV treatment of PWID is negligible in both settings.

PARTICIPANTS

PWID data were collected between 1998 and 2015 from Social Networks Among Appalachian People, U Find Out, Urban Health Study and National HIV Behavioral Surveillance System studies.

INTERVENTIONS AND COMPARATOR

Three intervention scenarios modeled: baseline-existing SSP and MAT coverage with HCV screening and treatment with direct-acting antiviral for ex-injectors only as per standard of care; intervention 1-scale-up of SSP and MAT without changes to treatment; and intervention 2-scale-up as intervention 1 combined with HCV screening and treatment for current PWID.

MEASUREMENTS

Incremental cost-effectiveness ratios (ICERs) and uncertainty using cost-effectiveness acceptability curves. Benefits were measured in quality-adjusted life-years (QALYs).

FINDINGS

For both settings, intervention 2 is preferred to intervention 1 and the appropriate comparator for intervention 2 is the baseline scenario. Relative to baseline, for PC intervention 2 averts 1852 more HCV infections, increases QALYS by 3095, costs $21.6 million more and has an ICER of $6975/QALY. For SF, intervention 2 averts 36 473 more HCV infections, increases QALYs by 7893, costs $872 million more and has an ICER of $11 044/QALY. The cost-effectiveness of intervention 2 was robust to several sensitivity analysis.

CONCLUSIONS

Hepatitis C screening and treatment for people who inject drugs, combined with medication-assisted treatment and syringe-service programs, is a cost-effective strategy for reducing hepatitis C burden in the United States.

A utility of model input uncertainty analysis in transferring tobacco control-related economic evidence to countries with scarce resources: results from the EQUIPT study

AIMS

To inform the transferability of tobacco control-related economic evidence to resource-poor countries.

METHODS

We ran a univariate sensitivity analysis on a return on investment (ROI) model, the European study on Quantifying Utility of Investment in Protection from Tobacco model (EQUIPTMOD), to identify key input values to which the ROI estimates were sensitive. The EQUIPTMOD used a Markov-based state transition model to estimate the ROI of several tobacco control interventions in five European countries (England, Germany, Spain, Hungary and the Netherlands). Base case ROI estimates were obtained through average values of model inputs (throughout the five countries), which were then replaced one at a time with country-specific values. Tornado diagrams were used to evaluate the significance of sensitivity, defined as a ≥ 10% difference in ROI estimates from the base case estimates.

RESULTS

The ROI estimates were sensitive to 18 (of 46) input values. Examples of model inputs to which ROI estimates were sensitive included: smoking rate, costs of smoking-related diseases (e.g. lung cancer) and general population attributes.

CONCLUSION

Countries that have limited research time and other resources can adapt EQUIPTMOD to their own settings by choosing to collect data on a small number of model inputs. EQUIPTMOD can therefore facilitate transfer of tobacco control related economic evidence to new jurisdictions.

Effectiveness and cost-effectiveness of serum B-type natriuretic peptide testing and monitoring in patients with heart failure in primary and secondary care: an evidence synthesis, cohort study and cost-effectiveness model

BACKGROUND

Heart failure (HF) affects around 500,000 people in the UK. HF medications are frequently underprescribed and B-type natriuretic peptide (BNP)-guided therapy may help to optimise treatment.

OBJECTIVE

To evaluate the clinical effectiveness and cost-effectiveness of BNP-guided therapy compared with symptom-guided therapy in HF patients.

DESIGN

Systematic review, cohort study and cost-effectiveness model.

SETTING

A literature review and usual care in the NHS.

PARTICIPANTS

(a) HF patients in randomised controlled trials (RCTs) of BNP-guided therapy; and (b) patients having usual care for HF in the NHS.

INTERVENTIONS

Systematic review: BNP-guided therapy or symptom-guided therapy in primary or secondary care. Cohort study: BNP monitored (≥ 6 months' follow-up and three or more BNP tests and two or more tests per year), BNP tested (≥ 1 tests but not BNP monitored) or never tested. Cost-effectiveness model: BNP-guided therapy in specialist clinics.

MAIN OUTCOME MEASURES

Mortality, hospital admission (all cause and HF related) and adverse events; and quality-adjusted life-years (QALYs) for the cost-effectiveness model.

DATA SOURCES

Systematic review: Individual participant or aggregate data from eligible RCTs. Cohort study: The Clinical Practice Research Datalink, Hospital Episode Statistics and National Heart Failure Audit (NHFA).

REVIEW METHODS

A systematic literature search (five databases, trial registries, grey literature and reference lists of publications) for published and unpublished RCTs.

RESULTS

Five RCTs contributed individual participant data (IPD) and eight RCTs contributed aggregate data (1536 participants were randomised to BNP-guided therapy and 1538 participants were randomised to symptom-guided therapy). For all-cause mortality, the hazard ratio (HR) for BNP-guided therapy was 0.87 [95% confidence interval (CI) 0.73 to 1.04]. Patients who were aged < 75 years or who had heart failure with a reduced ejection fraction (HFrEF) received the most benefit [interactions (p = 0.03): < 75 years vs. ≥ 75 years: HR 0.70 (95% CI 0.53 to 0.92) vs. 1.07 (95% CI 0.84 to 1.37); HFrEF vs. heart failure with a preserved ejection fraction (HFpEF): HR 0.83 (95% CI 0.68 to 1.01) vs. 1.33 (95% CI 0.83 to 2.11)]. In the cohort study, incident HF patients (1 April 2005-31 March 2013) were never tested (n = 13,632), BNP tested (n = 3392) or BNP monitored (n = 71). Median survival was 5 years; all-cause mortality was 141.5 out of 1000 person-years (95% CI 138.5 to 144.6 person-years). All-cause mortality and hospital admission rate were highest in the BNP-monitored group, and median survival among 130,433 NHFA patients (1 January 2007-1 March 2013) was 2.2 years. The admission rate was 1.1 patients per year (interquartile range 0.5-3.5 patients). In the cost-effectiveness model, in patients aged < 75 years with HFrEF or HFpEF, BNP-guided therapy improves median survival (7.98 vs. 6.46 years) with a small QALY gain (5.68 vs. 5.02) but higher lifetime costs (£64,777 vs. £58,139). BNP-guided therapy is cost-effective at a threshold of £20,000 per QALY.

LIMITATIONS

The limitations of the trial were a lack of IPD for most RCTs and heterogeneous interventions; the inability to identify BNP monitoring confidently, to determine medication doses or to distinguish between HFrEF and HFpEF; the use of a simplified two-state Markov model; a focus on health service costs and a paucity of data on HFpEF patients aged < 75 years and HFrEF patients aged ≥ 75 years.

CONCLUSIONS

The efficacy of BNP-guided therapy in specialist HF clinics is uncertain. If efficacious, it would be cost-effective for patients aged < 75 years with HFrEF. The evidence reviewed may not apply in the UK because care is delivered differently.

FUTURE WORK

Identify an optimal BNP-monitoring strategy and how to optimise HF management in accordance with guidelines; update the IPD meta-analysis to include the Guiding Evidence Based Therapy Using Biomarker Intensified Treatment (GUIDE-IT) RCT; collect routine long-term outcome data for completed and ongoing RCTs.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN37248047 and PROSPERO CRD42013005335.

FUNDING

This project was funded by the NIHR Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 21, No. 40. See the NIHR Journals Library website for further project information. The British Heart Foundation paid for Chris A Rogers' and Maria Pufulete's time contributing to the study. Syed Mohiuddin's time is supported by the NIHR Collaboration for Leadership in Applied Health Research and Care West at University Hospitals Bristol NHS Foundation Trust. Rachel Maishman contributed to the study when she was in receipt of a NIHR Methodology Research Fellowship.

A Review of Methods for Analysis of the Expected Value of Information

In recent years, value-of-information analysis has become more widespread in health economic evaluations, specifically as a tool to guide further research and perform probabilistic sensitivity analysis. This is partly due to methodological advancements allowing for the fast computation of a typical summary known as the expected value of partial perfect information (EVPPI). A recent review discussed some approximation methods for calculating the EVPPI, but as the research has been active over the intervening years, that review does not discuss some key estimation methods. Therefore, this paper presents a comprehensive review of these new methods. We begin by providing the technical details of these computation methods. We then present two case studies in order to compare the estimation performance of these new methods. We conclude that a method based on nonparametric regression offers the best method for calculating the EVPPI in terms of accuracy, computational time, and ease of implementation. This means that the EVPPI can now be used practically in health economic evaluations, especially as all the methods are developed in parallel with R functions and a web app to aid practitioners.

Value of Information in Asia: Concepts, Current Use, and Future Directions

Health technology assessment is a form of health policy research that provides policymakers with information relevant to decisions about policy alternatives. Findings from cost-effectiveness analysis (CEA) are one of the important aspects of health technology assessment. Nevertheless, the more advanced method of value of information (VOI), which is recommended by the International Society for Pharmacoeconomics and Outcomes Research and Society for Medical Decision Making Modeling Good Research Practices Task Force, has rarely been applied in CEA studies in Asia. The lack of VOI in Asian CEA studies may be due to limited understanding of VOI methods and what VOI can and cannot help policy decision makers accomplish. This concept article offers audiences a practical primer in understanding the calculation, presentation, and policy implications of VOI. In addition, it provides a rapid survey of health technology assessment guidelines and literature related to VOI in Asia and discusses the future directions of VOI use in Asia and its potential barriers. This article will enable health economists, outcomes researchers, and policymakers in Asia to better understand the importance of VOI analysis and its implications, leading to the appropriate use of VOI in Asia.

Estimating the expected value of partial perfect information in health economic evaluations using integrated nested Laplace approximation

The Expected Value of Perfect Partial Information (EVPPI) is a decision‐theoretic measure of the ‘cost’ of parametric uncertainty in decision making used principally in health economic decision making. Despite this decision‐theoretic grounding, the uptake of EVPPI calculations in practice has been slow. This is in part due to the prohibitive computational time required to estimate the EVPPI via Monte Carlo simulations. However, recent developments have demonstrated that the EVPPI can be estimated by non‐parametric regression methods, which have significantly decreased the computation time required to approximate the EVPPI. Under certain circumstances, high‐dimensional Gaussian Process (GP) regression is suggested, but this can still be prohibitively expensive. Applying fast computation methods developed in spatial statistics using Integrated Nested Laplace Approximations (INLA) and projecting from a high‐dimensional into a low‐dimensional input space allows us to decrease the computation time for fitting these high‐dimensional GP, often substantially. We demonstrate that the EVPPI calculated using our method for GP regression is in line with the standard GP regression method and that despite the apparent methodological complexity of this new method, R functions are available in the package BCEA to implement it simply and efficiently. © 2016 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.