Unremarked or Unperformed? Systematic Review on Reporting of Validation Efforts of Health Economic Decision Models in Seasonal Influenza and Early Breast Cancer

A Blueprint for Multi-use Disease Modeling in Health Economics: Results from Two Expert-Panel Consultations

BACKGROUND

The current use of health economic decision models in HTA is mostly confined to single use cases, which may be inefficient and result in little consistency over different treatment comparisons, and consequently inconsistent health policy decisions, for the same disorder. Multi-use disease models (MUDMs) (other terms: generic models, whole disease models, disease models) may offer a solution. However, much is uncertain about their definition and application. The current research aimed to develop a blueprint for the application of MUDMs.

METHODS

We elicited expert opinion using a two-round modified Delphi process. The panel consisted of experts and stakeholders in health economic modelling from various professional backgrounds. The first questionnaire concerned definition, terminology, potential applications, issues and recommendations for MUDMs and was based on an exploratory scoping review. In the second round, the panel members were asked to reconsider their input, based on feedback regarding first-round results, and to score issues and recommendations for priority. Finally, adding input from external advisors and policy makers in a structured way, an overview of issues and challenges was developed during two team consensus meetings.

RESULTS

In total, 54 respondents contributed to the panel results. The term 'multi-use disease models' was proposed and agreed upon, and a definition was provided. The panel prioritized 10 potential applications (with comparing alternative policies and supporting resource allocation decisions as the top 2), while 20 issues (with model transparency and stakeholders' roles as the top 2) were identified as challenges. Opinions on potential features concerning operationalization of multi-use models were given, with 11 of these subsequently receiving high priority scores (regular updates and revalidation after updates were the top 2).

CONCLUSIONS

MUDMs would improve on current decision support regarding cost-effectiveness information. Given feasibility challenges, this would be most relevant for diseases with multiple treatments, large burden of disease and requiring more complex models. The current overview offers policy makers a starting point to organize the development, use, and maintenance of MUDMs and to support choices concerning which diseases and policy decisions they will be helpful for.

Validating Health Economic Models With the Probabilistic Analysis Check dashBOARD

OBJECTIVES

Health economic (HE) models are often considered as "black boxes" because they are not publicly available and lack transparency, which prevents independent scrutiny of HE models. Additionally, validation efforts and validation status of HE models are not systematically reported. Methods to validate HE models in absence of their full underlying code are therefore urgently needed to improve health policy making. This study aimed to develop and test a generic dashboard to systematically explore the workings of HE models and validate their model parameters and outcomes.

METHODS

The Probabilistic Analysis Check dashBOARD (PACBOARD) was developed using insights from literature, health economists, and a data scientist. Functionalities of PACBOARD are (1) exploring and validating model parameters and outcomes using standardized validation tests and interactive plots, (2) visualizing and investigating the relationship between model parameters and outcomes using metamodeling, and (3) predicting HE outcomes using the fitted metamodel. To test PACBOARD, 2 mock HE models were developed, and errors were introduced in these models, eg, negative costs inputs, utility values exceeding 1. PACBOARD metamodeling predictions of incremental net monetary benefit were validated against the original model's outcomes.

RESULTS

PACBOARD automatically identified all errors introduced in the erroneous HE models. Metamodel predictions were accurate compared with the original model outcomes.

CONCLUSIONS

PACBOARD is a unique dashboard aiming at improving the feasibility and transparency of validation efforts of HE models. PACBOARD allows users to explore the working of HE models using metamodeling based on HE models' parameters and outcomes.

Model and Empirical Data-Based Cost-Utility Studies of Continuous Glucose Monitoring in Individuals with Type 1 Diabetes: A Protocol of a Systematic Review on Methodology and Quality

INTRODUCTION

This review aims to critically appraise differences in methodology and quality of model-based and empirical-data-based cost-utility studies to address key limitations, opportunities, and challenges to inform future cost-utility analyses of continuous glucose monitoring (CGM) in type 1 diabetes. This protocol is registered at PROSPERO (CRD42023391284).

METHODS

The review will be conducted in accordance with the PRISMA guideline for systematic reviews. Searches will be conducted in MEDLINE, Embase, Web of Science, Cochrane Library, and Econlit from 2000 to January 2023. Model and empirical data-based studies evaluating the cost-utility of any CGM system in type 1 diabetes will be considered for inclusion. Studies that only report on cost per life year or any other clinical outcome, or reporting only costs or only clinical outcomes studies in type 2 diabetes populations, and studies on bi-hormonal closed loops and do-it-yourself hybrid closed loop devices will be excluded. Two reviewers will independently screen each study for inclusion. Data on the intervention, population, model settings (such as perspective, time horizon), model type and structure, clinical outcomes used to populate the model, validation, and uncertainty will be extracted and qualitatively synthesised. Quality will be assessed using the Philips et al. 2006 (model-based studies) or Consensus Health Economic Criteria (empirical data-based studies) checklists. Model validation will be assessed using the AdViSHE checklist.

DISCUSSION

Now that CGM is being used more broadly in practice, critical appraisal of existing cost-utility methodology and quality is important to inform future cost-utility analyses of CGM in type 1 diabetes in various settings.

Simulation modeling validity and utility in colorectal cancer screening delivery: A systematic review

OBJECTIVE

This study sought to assess the impact and validity of simulation modeling in informing decision making in a complex area of healthcare delivery: colorectal cancer (CRC) screening.

MATERIALS AND METHODS

We searched 10 electronic databases for English-language articles published between January 1, 2008, and March 1, 2019, that described the development of a simulation model with a focus on average-risk CRC screening delivery. Included articles were reviewed for evidence that the model was validated, and provided real or potential contribution to informed decision making using the GRADE EtD (Grading of Recommendations Assessment, Development, and Evaluation Evidence to Decision) framework.

RESULTS

A total of 43 studies met criteria. The majority used Markov modeling (n = 31 [72%]) and sought to determine cost-effectiveness, compare screening modalities, or assess effectiveness of screening. No study reported full model validation and only (58%) reported conducting any validation. Majority of models were developed to address a specific health systems or policy question; few articles report the model's impact on this decision (n = 39 [91%] vs. n = 5 [12%]). Overall, models provided evidence relevant to every element important to decision makers as outlined in the GRADE EtD framework.

DISCUSSION AND CONCLUSION

Simulation modeling contributes evidence that is considered valuable to decision making in CRC screening delivery, particularly in assessing cost-effectiveness and comparing screening modalities. However, the actual impact on decisions and validity of models is lacking in the literature. Greater validity testing, impact assessment, and standardized reporting of both is needed to understand and demonstrate the reliability and utility of simulation modeling.

Use of Simulation Modeling to Inform Decision Making for Health Care Systems and Policy in Colorectal Cancer Screening: Protocol for a Systematic Review

Background

Simulation modeling has frequently been used to assess interventions in complex aspects of health care, such as colorectal cancer (CRC) screening, where clinical trials are not feasible. Simulation models provide estimates of outcomes, unintended consequences, and costs of an intervention; thus offering an invaluable decision aid for policy makers and health care leaders. However, the contribution that simulation models have made to policy and health system decisions is unknown.

Objective

This study aims to assess if simulation modeling has supported evidence-informed decision making in CRC screening.

Methods

A preliminary literature search and pilot screening of 100 references were conducted by three independent reviewers to define and refine the inclusion criteria of this systematic review. Using the developed inclusion criteria, a search of the academic and gray literature published between January 1, 2008, and March 1, 2019, will be conducted to identify studies that developed a simulation model focusing on the delivery of CRC screening of average-risk individuals. The three independent reviewers will assess the validation process and the extent to which the study contributed evidence toward informed decision making (both reported and potential). Validation will be assessed based on adherence to the best practice recommendations described by the International Society for Pharmacoeconomics and Outcomes Research-Society for Medical Decision Making (ISPOR-SMDM). Criteria for potential contribution to decision making will be defined as outlined in the internationally recognized Grading of Recommendations Assessment, Development and Evaluation Evidence to Decision (GRADE EtD) framework. These criteria outline information that the health system and policy decision makers should consider when making an evidence-informed decision including an intervention’s resource utilization, cost-effectiveness, impact on health equity, and feasibility. Subgroup analysis of articles based on their GRADE EtD criteria will be conducted to identify methods associated with decision support capacity (ie, participatory, quantitative, or mixed methods).

Results

A database search of the literature yielded 484 references to screen for inclusion in the systematic review. We anticipate that this systematic review will provide an insight into the contribution of simulation modeling methods to informed decision making in CRC screening delivery and discuss methods that may be associated with a stronger impact on decision making. The project was funded in May 2019. Data collection took place from January 2008 to March 2019. Data analysis was completed in November 2019, and are expected to be published in spring 2020.

Conclusions

Our findings will help guide researchers and health care leaders to mobilize the potential for simulation modeling to inform evidence-informed decisions in CRC screening delivery. The methods of this study may also be replicated to assess the utility of simulation modeling in other areas of complex health care decision making.

International Registered Report Identifier (IRRID)

DERR1-10.2196/16103

Trial Registration

PROSPERO no. 130823; https://www.crd.york.ac.uk/PROSPERO

A Microsimulation Model for Evaluating the Effectiveness of Cancer Risk Management for BRCA Pathogenic Variant Carriers: miBRovaCAre

OBJECTIVES

To develop a validated model for evaluating the real-world effectiveness of long-term clinical management strategies for women with germline BRCA1 or BRCA2 pathogenic variants.

METHODS

A microsimulation model was developed that included a BRCA-specific natural history for breast and ovarian cancer, a clinical framework for carrier follow-up, and cancer risk management strategies (breast screening, risk-reducing mastectomy, and bilateral salpingo-oophorectomy). Adherence rates and outcomes for breast screening and risk-reducing surgery were obtained from BRCA carriers seen through a familial cancer service in Melbourne, Australia. The model was assessed for internal and external validity. The model was used to compare women perfectly adhering to screening recommendations versus actual adherence of the clinical cohort.

RESULTS

The model accurately predicted cancer incidence, pathology, and mortality. Using actual adherence for breast screening resulted in additional breast cancer deaths (per 1000 women: BRCA1, 2.7; BRCA2, 1.6) compared with perfect screening adherence. This decreased average life expectancy by 0.30 life-years for BRCA1 and 0.07 life-years for BRCA2. When carriers had access to risk-reducing mastectomy, the benefit from improved screening adherence was not significant.

CONCLUSIONS

The developed model is a good descriptor of BRCA carriers' lifetime trajectory and its modification by use of risk management strategies alone or in combination. Evaluations of breast screening in BRCA carriers may overestimate the benefits of screening programs unless adherence is considered. By incorporating real-world clinical practice and patient behavior, this model can assist in developing clinical services and improving clinical outcomes for carriers.

A systematic review of the health economic consequences of quadrivalent influenza vaccination

BACKGROUND

Quadrivalent influenza vaccines (QIVs) contain antigens derived from an additional influenza type B virus as compared with currently used trivalent influenza vaccines (TIVs). This should overcome a potential reduced vaccine protection due to mismatches between TIV and circulating B viruses. In this study, we systematically reviewed the available literature on health economic evaluations of switching from TIV to QIV. Areas covered: The databases of Medline and Embase were searched systematically to identify health economic evaluations of QIV versus TIV published before September 2016.A total of sixteen studies were included, thirteen cost-effectiveness analyses and three cost-comparisons. Expert commentary: Published evidence on the cost-effectiveness of QIV suggests that switching from TIV to QIV would be a valuable intervention from both the public health and economic viewpoint. However, more research seems mandatory. Our main recommendations for future research include: 1) more extensive use of dynamic models in order to estimate the full impact of QIV on influenza transmission including indirect effects, 2) improved availability of data on disease outcomes and costs related to influenza type B viruses, and 3) more research on immunogenicity of natural influenza infection and vaccination, with emphasis on cross-reactivity between different influenza B viruses and duration of protection.