Multivariable and Structural Uncertainty Analyses for Cost-Effectiveness Estimates: Back to the Future

A Guide to an Iterative Approach to Model-Based Decision Making in Health and Medicine: An Iterative Decision-Making Framework

Decision makers frequently face decisions about optimal resource allocation. A model-based economic evaluation can be used to guide decision makers in their choices by systematically evaluating the magnitude of expected health effects and costs of decision options and by making trade-offs explicit. We provide a guide to an iterative approach to the medical decision-making process by following a coherent framework, and outline the overarching iterative steps of model-based decision making. We systematized the framework by performing three steps. First, we compiled the existing guidelines provided by the ISPOR-SMDM Modeling Good Research Practices Task Force, and the ISPOR Value of Information Task Force. Second, we identified other previous work related to frameworks and guidelines for model-based decision analyses through a literature search in PubMed. Third, we assessed the role of the evidence and iterative process in decision making and formalized key steps in a model-based decision-making framework. We provide guidance on an iterative approach to medical decision making by applying the compiled iterative model-based decision-making framework. The framework formally combines the decision problem conceptualization (Part I), the model conceptualization and development (Part II), and the process of model-based decision analysis (Part III). Following the overarching steps of the framework ensures compliance to the principles of evidence-based medicine and regular updates of the evidence, given that value of information analysis represents an essential component of model-based decision analysis in the framework. Following the provided guide and the steps outlined in the framework can help inform various health care decisions, and therefore it has the potential to improve decision making.

Real-World Cost-Effectiveness Analysis: How Much Uncertainty Is in the Results?

Cost-effectiveness analyses of new cancer treatments in real-world settings (e.g., post-clinical trials) inform healthcare decision makers about their healthcare investments for patient populations. The results of these analyses are often, though not always, presented with statistical uncertainty. This paper identifies five ways to characterize statistical uncertainty: (1) a 95% confidence interval (CI) for the incremental cost-effectiveness ratio (ICER); (2) a 95% CI for the incremental net benefit (INB); (3) an INB by willingness-to-pay (WTP) plot; (4) a cost-effectiveness acceptability curve (CEAC); and (5) a cost-effectiveness scatterplot. It also explores their usage in 22 articles previously identified by a rapid review of real-world cost effectiveness of novel cancer treatments. Seventy-seven percent of these articles presented uncertainty results. The majority those papers (59%) used administrative data to inform their analyses while the remaining were conducted using models. Cost-effectiveness scatterplots were the most commonly used method (34.3%), with 40% indicating high levels of statistical uncertainty, suggesting the possibility of a qualitatively different result from the estimate given. Understanding the necessity for and the meaning of uncertainty in real-world cost-effectiveness analysis will strengthen knowledge translation efforts to improve patient outcomes in an efficient manner.

Exploring Structural Uncertainty and Impact of Health State Utility Values on Lifetime Outcomes in Diabetes Economic Simulation Models: Findings from the Ninth Mount Hood Diabetes Quality-of-Life Challenge

BACKGROUND

Structural uncertainty can affect model-based economic simulation estimates and study conclusions. Unfortunately, unlike parameter uncertainty, relatively little is known about its magnitude of impact on life-years (LYs) and quality-adjusted life-years (QALYs) in modeling of diabetes. We leveraged the Mount Hood Diabetes Challenge Network, a biennial conference attended by international diabetes modeling groups, to assess structural uncertainty in simulating QALYs in type 2 diabetes simulation models.

METHODS

Eleven type 2 diabetes simulation modeling groups participated in the 9th Mount Hood Diabetes Challenge. Modeling groups simulated 5 diabetes-related intervention profiles using predefined baseline characteristics and a standard utility value set for diabetes-related complications. LYs and QALYs were reported. Simulations were repeated using lower and upper limits of the 95% confidence intervals of utility inputs. Changes in LYs and QALYs from tested interventions were compared across models. Additional analyses were conducted postchallenge to investigate drivers of cross-model differences.

RESULTS

Substantial cross-model variability in incremental LYs and QALYs was observed, particularly for HbA1c and body mass index (BMI) intervention profiles. For a 0.5%-point permanent HbA1c reduction, LY gains ranged from 0.050 to 0.750. For a 1-unit permanent BMI reduction, incremental QALYs varied from a small decrease in QALYs (-0.024) to an increase of 0.203. Changes in utility values of health states had a much smaller impact (to the hundredth of a decimal place) on incremental QALYs. Microsimulation models were found to generate a mean of 3.41 more LYs than cohort simulation models (P = 0.049).

CONCLUSIONS

Variations in utility values contribute to a lesser extent than uncertainty captured as structural uncertainty. These findings reinforce the importance of assessing structural uncertainty thoroughly because the choice of model (or models) can influence study results, which can serve as evidence for resource allocation decisions.HighlightsThe findings indicate substantial cross-model variability in QALY predictions for a standardized set of simulation scenarios and is considerably larger than within model variability to alternative health state utility values (e.g., lower and upper limits of the 95% confidence intervals of utility inputs).There is a need to understand and assess structural uncertainty, as the choice of model to inform resource allocation decisions can matter more than the choice of health state utility values.

Financial Estimation of the Uncertainty in Medicine Using Present Value of Medical Fees and a Mortality Risk Prediction Model: a Retrospective Cohort Study

This study aimed to develop a method to enable the financial estimation of each patient’s uncertainty without focusing on healthcare technology. We define financial uncertainty (FU) as the difference between an actual amount of claim (AC) and the discounted present value of the AC (DAC). DAC can be calculated based on a discounted present value calculated using a cash flow, a period of investment, and a discount rate. The present study considered these three items as AC, the length of hospital stay, and the predicted mortality rate. The mortality prediction model was built using typical data items in standard level electronic medical records such as sex, age, and disease information. The performance of the prediction model was moderate because an area under curve was approximately 85%. The empirical analysis primarily compares the FU of the top 20 diseases with the actual AC using a retrospective cohort in the University of Miyazaki Hospital. The observational period is 5 years, from April 1, 2013, to March 31, 2018. The analysis demonstrates that the proportion of FU to actual AC is higher than 20% in low-weight children, patients with leukemia, brain tumor, myeloid leukemia, or non-Hodgkin’s lymphoma. For these diseases, patients cannot avoid long hospitalization; therefore, the medical fee payment system should be designed based on uncertainty. Our method is both practical and generalizable because it uses a small number of data items that are required in standard electronic medical records. This method contributes to the decision-making processes of health policymakers.

Are COVID-19 models blind to the social determinants of health? A systematic review protocol

INTRODUCTION

Infectious disease models are important tools to inform public health policy decisions. These models are primarily based on an average population approach and often ignore the role of social determinants in predicting the course of a pandemic and the impact of policy interventions. Ignoring social determinants in models may cause or exacerbate inequalities. This limitation has not been previously explored in the context of the current pandemic, where COVID-19 has been found to disproportionately affect marginalised racial, ethnic and socioeconomic groups. Therefore, our primary goal is to identify the extent to which COVID-19 models incorporate the social determinants of health in predicting outcomes of the pandemic.

METHODS AND ANALYSIS

We will search MEDLINE, EMBASE, Cochrane Library and Web of Science databases from December 2019 to August 2020. We will assess all infectious disease modelling studies for inclusion of social factors that meet the following criteria: (a) focused on human spread of SARS-CoV-2; (b) modelling studies; (c) interventional or non-interventional studies; and (d) focused on one of the following outcomes: COVID-19-related outcomes (eg, cases, deaths), non-COVID-19-related outcomes (ie, impacts of the pandemic or control policies on other health conditions or health services), or impact of the pandemic or control policies on economic outcomes. Data will only be extracted from models incorporating social factors. We will report the percentage of models that considered social factors, indicate which social factors were considered, and describe how social factors were incorporated into the conceptualisation and implementation of the infectious disease models. The extracted data will also be used to create a narrative synthesis of the results.

ETHICS AND DISSEMINATION

Ethics approval is not required as only secondary data will be collected. The results of this systematic review will be disseminated through peer-reviewed publication and conference proceedings.

PROSPERO REGISTRATION NUMBER

CRD42020207706.