Development and Validation of PREDICT-DM: A New Microsimulation Model to Project and Evaluate Complications and Treatments of Type 2 Diabetes Mellitus

A literature review of quality assessment and applicability to HTA of risk prediction models of coronary heart disease in patients with diabetes

This literature review had two objectives: to identify models for predicting the risk of coronary heart diseases in patients with diabetes (DM); and to assess model quality in terms of risk of bias (RoB) and applicability for the purpose of health technology assessment (HTA). We undertook a targeted review of journal articles published in English, Dutch, Chinese, or Spanish in 5 databases from 1st January 2016 to 18th December 2022, and searched three systematic reviews for the models published after 2012. We used PROBAST (Prediction model Risk Of Bias Assessment Tool) to assess RoB, and used findings from Betts et al. 2019, which summarized recommendations and criticisms of HTA agencies on cardiovascular risk prediction models, to assess model applicability for the purpose of HTA. As a result, 71 % and 67 % models reporting C-index showed good discrimination abilities (C-index >= 0.7). Of the 26 model studies and 30 models identified, only one model study showed low RoB in all domains, and no model was fully applicable for HTA. Since the major cause of high RoB is inappropriate use of analysis method, we advise clinicians to carefully examine the model performance declared by model developers, and to trust a model if all PROBAST domains except analysis show low RoB and at least one validation study conducted in the same setting (e.g. country) is available. Moreover, since general model applicability is not informative for HTA, novel adapted tools may need to be developed.

A Systematic Review of Methodologies Used in Models of the Treatment of Diabetes Mellitus

BACKGROUND

Diabetes mellitus is a chronic and complex disease, increasing in prevalence and consequent health expenditure. Cost-effectiveness models with long time horizons are commonly used to perform economic evaluations of diabetes' treatments. As such, prediction accuracy and structural uncertainty are important features in cost-effectiveness models of chronic conditions.

OBJECTIVES

The aim of this systematic review is to identify and review published cost-effectiveness models of diabetes treatments developed between 2011 and 2022 regarding their methodological characteristics. Further, it also appraises the quality of the methods used, and discusses opportunities for further methodological research.

METHODS

A systematic literature review was conducted in MEDLINE and Embase to identify peer-reviewed papers reporting cost-effectiveness models of diabetes treatments, with time horizons of more than 5 years, published in English between 1 January 2011 and 31 of December 2022. Screening, full-text inclusion, data extraction, quality assessment and data synthesis using narrative synthesis were performed. The Philips checklist was used for quality assessment of the included studies. The study was registered in PROSPERO (CRD42021248999).

RESULTS

The literature search identified 30 studies presenting 29 unique cost-effectiveness models of type 1 and/or type 2 diabetes treatments. The review identified 26 type 2 diabetes mellitus (T2DM) models, 3 type 1 DM (T1DM) models and one model for both types of diabetes. Fifteen models were patient-level models, whereas 14 were at cohort level. Parameter uncertainty was assessed thoroughly in most of the models, whereas structural uncertainty was seldom addressed. All the models where validation was conducted performed well. The methodological quality of the models with respect to structure was high, whereas with respect to data modelling it was moderate.

CONCLUSIONS

Models developed in the past 12 years for health economic evaluations of diabetes treatments are of high-quality and make use of advanced methods. However, further developments are needed to improve the statistical modelling component of cost-effectiveness models and to provide better assessment of structural uncertainty.

Development and Validation of a Simulation Model for Treatment to Maintain Remission in Antineutrophil Cytoplasmic Antibody-Associated Vasculitis

OBJECTIVE

Fixed and tailored rituximab retreatment strategies to maintain remission in antineutrophil cytoplasmic antibody (ANCA)-associated vasculitis (AAV) are associated with tradeoffs. The current study was undertaken to develop a simulation model (AAV-Sim) to project clinical outcomes with these strategies.

METHODS

We developed the AAV-Sim, a microsimulation model of clinical events among individuals with AAV initiating treatment to maintain remission. Individuals transition between health states of remission or relapse and are at risk for severe infection, end-stage renal disease, or death. We estimated transition rates from published literature, stratified by individual-level characteristics. We performed validation using the mean average percent error (MAPE) and the coefficient of variation of root mean square error (CV-RMSE). In internal validation, we compared model-projected outcomes over 28 months with outcomes observed in the Rituximab versus Azathioprine in ANCA-Associated Vasculitis 2 (MAINRITSAN2) trial, which compared fixed versus tailored retreatment. In external validation, we compared outcomes with fixed rituximab retreatment from the AAV-Sim to outcomes from the MAINRITSAN1 trial and an observational study.

RESULTS

The AAV-Sim projected outcomes similar to those in the MAINRITSAN2 trial, including minor (AAV-Sim 6.0% fixed versus 7.3% tailored; MAINRITSAN2 6.2% versus 8.6%; MAPE 3% and 15%) and major relapse (AAV-Sim 3.5% versus 5.5%; MAINRITSAN2 3.7% versus 7.4%; MAPE 5% and 26%), severe infection (AAV-Sim 19.4% versus 11.1%; MAINRITSAN2 19.8% versus 10.2%; MAPE 2% and 9%), and relapse-free survival (AAV-Sim 84.8% versus 82.3%; MAINRITSAN2 86% versus 84%; CV-RMSE 2.3% and 2.5%). Similar performance was observed in external validation.

CONCLUSION

The AAV-Sim projected a range of clinical outcomes for different treatment approaches that were validated against published data. The AAV-Sim has the potential to inform management guidelines and research priorities.

Prediction of complications in health economic models of type 2 diabetes: a review of methods used

AIM

Diabetes health economic (HE) models play important roles in decision making. For most HE models of diabetes 2 diabetes (T2D), the core model concerns the prediction of complications. However, reviews of HE models pay little attention to the incorporation of prediction models. The objective of the current review is to investigate how prediction models have been incorporated into HE models of T2D and to identify challenges and possible solutions.

METHODS

PubMed, Web of Science, Embase, and Cochrane were searched from January 1, 1997, to November 15, 2022, to identify published HE models for T2D. All models that participated in The Mount Hood Diabetes Simulation Modeling Database or previous challenges were manually searched. Data extraction was performed by two independent authors. Characteristics of HE models, their underlying prediction models, and methods of incorporating prediction models were investigated.

RESULTS

The scoping review identified 34 HE models, including a continuous-time object-oriented model (n = 1), discrete-time state transition models (n = 18), and discrete-time discrete event simulation models (n = 15). Published prediction models were often applied to simulate complication risks, such as the UKPDS (n = 20), Framingham (n = 7), BRAVO (n = 2), NDR (n = 2), and RECODe (n = 2). Four methods were identified to combine interdependent prediction models for different complications, including random order evaluation (n = 12), simultaneous evaluation (n = 4), the 'sunflower method' (n = 3), and pre-defined order (n = 1). The remaining studies did not consider interdependency or reported unclearly.

CONCLUSIONS

The methodology of integrating prediction models in HE models requires further attention, especially regarding how prediction models are selected, adjusted, and ordered.

Application of decision analytical models to diabetes in low- and middle-income countries: a systematic review

BACKGROUND

Decision analytical models (DAMs) are used to develop an evidence base for impact and health economic evaluations, including evaluating interventions to improve diabetes care and health services-an increasingly important area in low- and middle-income countries (LMICs), where the disease burden is high, health systems are weak, and resources are constrained. This study examines how DAMs-in particular, Markov, system dynamic, agent-based, discrete event simulation, and hybrid models-have been applied to investigate non-pharmacological population-based (NP) interventions and how to advance their adoption in diabetes research in LMICs.

METHODS

We systematically searched peer-reviewed articles published in English from inception to 8th August 2022 in PubMed, Cochrane, and the reference list of reviewed articles. Articles were summarised and appraised based on publication details, model design and processes, modelled interventions, and model limitations using the Health Economic Evaluation Reporting Standards (CHEERs) checklist.

RESULTS

Twenty-three articles were fully screened, and 17 met the inclusion criteria of this qualitative review. The majority of the included studies were Markov cohort (7, 41%) and microsimulation models (7, 41%) simulating non-pharmacological population-based diabetes interventions among Asian sub-populations (9, 53%). Eleven (65%) of the reviewed studies evaluated the cost-effectiveness of interventions, reporting the evaluation perspective and the time horizon used to track cost and effect. Few studies (6,35%) reported how they validated models against local data.

CONCLUSIONS

Although DAMs have been increasingly applied in LMICs to evaluate interventions to control diabetes, there is a need to advance the use of DAMs to evaluate NP diabetes policy interventions in LMICs, particularly DAMs that use local research data. Moreover, the reporting of input data, calibration and validation that underlies DAMs of diabetes in LMICs needs to be more transparent and credible.

Development and validation of the age-associated dementia policy (AgeD-Pol) computer simulation model in the USA and Europe

OBJECTIVE

To develop and validate a novel, microsimulation model that accounts for the prevalence and incidence of age-associated dementias (AAD), disease progression and associated mortality.

DESIGN, DATA SOURCES AND OUTCOME MEASURES

We developed the AAD policy (AgeD-Pol) model, a microsimulation model to simulate the natural history, morbidity and mortality associated with AAD. We populated the model with age-stratified and sex-stratified data on AAD prevalence, AAD incidence and mortality among people with AAD. We first performed internal validation using data from the Adult Changes in Thought (ACT) cohort study. We then performed external validation of the model using data from the Framingham Heart Study, the Rotterdam Study and Kaiser Permanente Northern California (KPNC). We compared model-projected AAD cumulative incidence and mortality with published cohort data using mean absolute percentage error (MAPE) and root-mean-square error (RMSE).

RESULTS

In internal validation, the AgeD-Pol model provided a good fit to the ACT cohort for cumulative AAD incidence, 10.4% (MAPE, 0.2%) and survival, 66.5% (MAPE, 8.8%), after 16 years of follow-up among those initially aged 65-69 years. In the external validations, the model-projected lifetime cumulative incidence of AAD was 30.5%-32.4% (females) and 16.7%-23.0% (males), using data from the Framingham and Rotterdam cohorts, and AAD cumulative incidence was 21.5% over 14 years using KPNC data. Model projections demonstrated a good fit to all three cohorts (MAPE, 0.9%-9.0%). Similarly, model-projected survival provided good fit to the Rotterdam (RMSE, 1.9-3.6 among those with and without AAD) and KPNC cohorts (RMSE, 7.6-18.0 among those with AAD).

CONCLUSIONS

The AgeD-Pol model performed well when validated to published data for AAD cumulative incidence and mortality and provides a useful tool to project the AAD disease burden for health systems planning in the USA.

Modeling Chronic Kidney Disease in Type 2 Diabetes Mellitus: A Systematic Literature Review of Models, Data Sources, and Derivation Cohorts

INTRODUCTION

As novel therapies for chronic kidney disease (CKD) in type 2 diabetes mellitus (T2DM) become available, their long-term benefits should be evaluated using CKD progression models. Existing models offer different modeling approaches that could be reused, but it may be challenging for modelers to assess commonalities and differences between the many available models. Additionally, the data and underlying population characteristics informing model parameters may not always be evident. Therefore, this study reviewed and summarized existing modeling approaches and data sources for CKD in T2DM, as a reference for future model development.

METHODS

This systematic literature review included computer simulation models of CKD in T2DM populations. Searches were implemented in PubMed (including MEDLINE), Embase, and the Cochrane Library, up to October 2021. Models were classified as cohort state-transition models (cSTM) or individual patient simulation (IPS) models. Information was extracted on modeled kidney disease states, risk equations for CKD, data sources, and baseline characteristics of derivation cohorts in primary data sources.

RESULTS

The review identified 49 models (21 IPS, 28 cSTM). A five-state structure was standard among state-transition models, comprising one kidney disease-free state, three kidney disease states [frequently including albuminuria and end-stage kidney disease (ESKD)], and one death state. Five models captured CKD regression and three included cardiovascular disease (CVD). Risk equations most commonly predicted albuminuria and ESKD incidence, while the most predicted CKD sequelae were mortality and CVD. Most data sources were well-established registries, cohort studies, and clinical trials often initiated decades ago in predominantly White populations in high-income countries. Some recent models were developed from country-specific data, particularly for Asian countries, or from clinical outcomes trials.

CONCLUSION

Modeling CKD in T2DM is an active research area, with a trend towards IPS models developed from non-Western data and single data sources, primarily recent outcomes trials of novel renoprotective treatments.

Economic Simulation Modeling in Type 2 Diabetes

PURPOSE OF REVIEW

This paper provides an overview of type 2 diabetes economic simulation modeling and reviews current topics of discussion and major challenges in the field.

RECENT FINDINGS

Important challenges in the field include increasing the generalizability of models and improving transparency in model reporting. To identify and address these issues, modeling groups have organized through the Mount Hood Diabetes Challenge meetings and developed tools (i.e., checklist, impact inventory) to standardize modeling methods and reporting of results. Accordingly, many newer diabetes models have begun utilizing these tools, allowing for improved comparability between diabetes models. In the last two decades, type 2 diabetes simulation models have improved considerably, due to the collaborative work performed through the Mount Hood Diabetes Challenge meetings. To continue to improve diabetes models, future work must focus on clarifying diabetes progression in racial/ethnic minorities and incorporating equity considerations into health economic analysis.

Novel microsimulation model of tobacco use behaviours and outcomes: calibration and validation in a US population

BACKGROUND AND OBJECTIVE

Simulation models can project effects of tobacco use and cessation and inform tobacco control policies. Most existing tobacco models do not explicitly include relapse, a key component of the natural history of tobacco use. Our objective was to develop, calibrate and validate a novel individual-level microsimulation model that would explicitly include smoking relapse and project cigarette smoking behaviours and associated mortality risks.

METHODS

We developed the Simulation of Tobacco and Nicotine Outcomes and Policy (STOP) model, in which individuals transition monthly between tobacco use states (current/former/never) depending on rates of initiation, cessation and relapse. Simulated individuals face tobacco use-stratified mortality risks. For US women and men, we conducted cross-validation with a Cancer Intervention and Surveillance Modeling Network (CISNET) model. We then incorporated smoking relapse and calibrated cessation rates to reflect the difference between a transient quit attempt and sustained abstinence. We performed external validation with the National Health Interview Survey (NHIS) and the linked National Death Index. Comparisons were based on root-mean-square error (RMSE).

RESULTS

In cross-validation, STOP-generated projections of current/former/never smoking prevalence fit CISNET-projected data well (coefficient of variation (CV)-RMSE≤15%). After incorporating smoking relapse, multiplying the CISNET-reported cessation rates for women/men by 7.75/7.25, to reflect the ratio of quit attempts to sustained abstinence, resulted in the best approximation to CISNET-reported smoking prevalence (CV-RMSE 2%/3%). In external validation using these new multipliers, STOP-generated cumulative mortality curves for 20-year-old current smokers and never smokers each had CV-RMSE ≤1% compared with NHIS. In simulating those surveyed by NHIS in 1997, the STOP-projected prevalence of current/former/never smokers annually (1998-2009) was similar to that reported by NHIS (CV-RMSE 12%).

CONCLUSIONS

The STOP model, with relapse included, performed well when validated to US smoking prevalence and mortality. STOP provides a flexible framework for policy-relevant analysis of tobacco and nicotine product use.