Comparing methods of data synthesis: re-estimating parameters of an existing probabilistic cost-effectiveness model

Different evidence summaries have implications for contextualizing findings of meta-analysis of diagnostic tests

OBJECTIVE

To evaluate diagnostic tests, analysts use meta-analyses to provide inputs to parameters in decision models. Choosing parameter estimands from meta-analyses requires understanding the meta-analytic and decision-making contexts.

STUDY DESIGN AND SETTING

We expand on an analysis comparing positron emission tomography (PET), PET with computed tomography (PET/CT), and conventional workup (CW) in women with suspected recurrent breast cancer. We discuss Bayesian meta-analytic summaries (posterior mean over a set of existing studies, posterior estimate in an existing study, posterior predictive mean in a new study) used to estimate diagnostic test parameters (prevalence, sensitivity, specificity) needed to calculate quality-adjusted life years in a decision model contextualizing PET, PET/CT, and CW.

RESULTS

The mean and predictive mean give similar estimates, but the latter displays greater uncertainty. Namely, PET/CT outperforms CW on average but may not do better than CW when implemented in future settings.

CONCLUSION

Selecting estimands for decision model parameters from meta-analyses requires understanding the relationship between decision settings and meta-analysis studies' settings, specifically whether the former resemble one or all study settings or represents new settings. We provide an algorithm recommending appropriate estimands as input parameters in decision models for diagnostic tests to obtain output parameters consistent with the decision context.

Current recommendations on the estimation of transition probabilities in Markov cohort models for use in health care decision-making: a targeted literature review

OBJECTIVE

Although Markov cohort models represent one of the most common forms of decision-analytic models used in health care decision-making, correct implementation of such models requires reliable estimation of transition probabilities. This study sought to identify consensus statements or guidelines that detail how such transition probability matrices should be estimated.

METHODS

A literature review was performed to identify relevant publications in the following databases: Medline, Embase, the Cochrane Library, and PubMed. Electronic searches were supplemented by manual-searches of health technology assessment (HTA) websites in Australia, Belgium, Canada, France, Germany, Ireland, Norway, Portugal, Sweden, and the UK. One reviewer assessed studies for eligibility.

RESULTS

Of the 1,931 citations identified in the electronic searches, no studies met the inclusion criteria for full-text review, and no guidelines on transition probabilities in Markov models were identified. Manual-searching of the websites of HTA agencies identified ten guidelines on economic evaluations (Australia, Belgium, Canada, France, Germany, Ireland, Norway, Portugal, Sweden, and UK). All identified guidelines provided general guidance on how to develop economic models, but none provided guidance on the calculation of transition probabilities. One relevant publication was identified following review of the reference lists of HTA agency guidelines: the International Society for Pharmacoeconomics and Outcomes Research taskforce guidance. This provided limited guidance on the use of rates and probabilities.

CONCLUSIONS

There is limited formal guidance available on the estimation of transition probabilities for use in decision-analytic models. Given the increasing importance of cost-effectiveness analysis in the decision-making processes of HTA bodies and other medical decision-makers, there is a need for additional guidance to inform a more consistent approach to decision-analytic modeling. Further research should be done to develop more detailed guidelines on the estimation of transition probabilities.

Mix and match. A simulation study on the impact of mixed-treatment comparison methods on health-economic outcomes

Background

Decision-analytic cost-effectiveness (CE) models combine many parameters, often obtained after meta-analysis.

Aim

We compared different methods of mixed-treatment comparison (MTC) to combine transition and event probabilities derived from several trials, especially with respect to health-economic (HE) outcomes like (quality adjusted) life years and costs.

Methods

Trials were drawn from a simulated reference population, comparing two of four fictitious interventions. The goal was to estimate the CE between two of these. The amount of heterogeneity between trials was varied in scenarios. Parameter estimates were combined using direct comparison, MTC methods proposed by Song and Puhan, and Bayesian generalized linear fixed effects (GLMFE) and random effects models (GLMRE). Parameters were entered into a Markov model. Parameters and HE outcomes were compared with the reference population using coverage, statistical power, bias and mean absolute deviation (MAD) as performance indicators. Each analytical step was repeated 1,000 times.

Results

The direct comparison was outperformed by the MTC methods on all indicators, Song’s method yielded low bias and MAD, but uncertainty was overestimated. Puhan’s method had low bias and MAD and did not overestimate uncertainty. GLMFE generally had the lowest bias and MAD, regardless of the amount of heterogeneity, but uncertainty was overestimated. GLMRE showed large bias and MAD and overestimated uncertainty. Song’s and Puhan’s methods lead to the least amount of uncertainty, reflected in the shape of the CE acceptability curve. GLMFE showed slightly more uncertainty.

Conclusions

Combining direct and indirect evidence is superior to using only direct evidence. Puhan’s method and GLMFE are preferred.

A Systematic Review of Health Economics Simulation Models of Chronic Obstructive Pulmonary Disease

BACKGROUND

Many decision-analytic models with varying structures have been developed to inform resource allocation in chronic obstructive pulmonary disease (COPD).

OBJECTIVES

To review COPD models for their adherence to the best practice modeling recommendations and their assumptions regarding important aspects of the natural history of COPD.

METHODS

A systematic search of English articles reporting on the development or application of a decision-analytic model in COPD was performed in MEDLINE, Embase, and citations within reviewed articles. Studies were summarized and evaluated on the basis of their adherence to the Consolidated Health Economic Evaluation Reporting Standards. They were also evaluated for the underlying assumptions about disease progression, heterogeneity, comorbidity, and treatment effects.

RESULTS

Forty-nine models of COPD were included. Decision trees and Markov models were the most popular techniques (43 studies). Quality of reporting and adherence to the guidelines were generally high, especially in more recent publications. Disease progression was modeled through clinical staging in most studies. Although most studies (n = 43) had incorporated some aspects of COPD heterogeneity, only 8 reported the results across subgroups. Only 2 evaluations explicitly considered the impact of comorbidities. Treatment effect had been mostly modeled (20) as both reduction in exacerbation rate and improvement in lung function.

CONCLUSIONS

Many COPD models have been developed, generally with similar structural elements. COPD is highly heterogeneous, and comorbid conditions play an important role in its burden. These important aspects, however, have not been adequately addressed in most of the published models.

A choice that matters? Smulation study on the impact of direct meta-analysis methods on health economic outcomes

BACKGROUND

Decision-analytic cost-effectiveness (CE) models combine many different parameters like transition probabilities, event probabilities, utilities and costs, which are often obtained after meta-analysis. The method of meta-analysis may affect the CE estimate.

AIM

Our aim was to perform a simulation study that compares the performance of different methods of meta-analysis, especially with respect to model-based health economic (HE) outcomes.

METHODS

A reference patient population of 50,000 was simulated from which sets of samples were drawn. Each sample drawn represented a clinical trial comparing two fictitious interventions. In several scenarios, the heterogeneity between these trials was varied, by drawing one or more of the trials from predefined subpopulations. Parameter estimates from these trials were combined using frequentist fixed (FFE) and random effects (FRE), and Bayesian fixed (BFE) and random effects (BRE) meta-analysis. The pooled parameter estimates were entered into a probabilistic cost-effectiveness Markov model. The four methods of meta-analysis resulted in different parameter estimates and HE outcomes, which were compared with the true values in the reference population. Performance statistics were: (1) the percentage of repetitions that the confidence interval of the probabilistic sensitivity analysis covers the true value (coverage), (2) the difference between the estimated and true value (bias), (3) the mean absolute value of the bias (MAD) and (4) the percentage of repetitions that result in a statistically significant difference between the two interventions (statistical power). As the differences between methods could be due to chance, we repeated every step of the analysis 1,000 times to study whether differences were systematic.

RESULTS

FFE, FRE and BFE lead to different parameter estimates, but, when entered into the model, they do not lead to large differences in the point estimates of the HE outcomes, even in scenarios where we built in heterogeneity. Random effects methods do not necessarily reduce bias when heterogeneity is added to the trials, and may even increase bias in certain situations. BRE tends to overestimate uncertainty reflected in the CE acceptability curve.

CONCLUSION

FFE, FRE and BFE lead to comparable HE outcomes. BRE tends to overestimate uncertainty. Based on this study, we recommend FRE as the preferred method of meta-analysis.