Evaluating evidence from a decision analysis

Characterizing structural uncertainty in decision analytic models: a review and application of methods

BACKGROUND

The characterization of uncertainty is critical in cost-effectiveness analysis, particularly when considering whether additional evidence is needed. In addition to parameter and methodological uncertainty, there are other sources of uncertainty which include simplifications and scientific judgments that have to be made when constructing and interpreting a model of any sort. These have been classified in a number of different ways but can be referred to collectively as structural uncertainties.

MATERIALS AND METHODS

Separate reviews were undertaken to identify what forms these other sources of uncertainty take and what other forms of potential methods to explicitly characterize these types of uncertainties in decision analytic models. These methods were demonstrated through application to four decision models each representing one of the four types of uncertainty.

RESULTS

These sources of uncertainty fall into four general themes: 1) inclusion of relevant comparators; 2) inclusion of relevant events; 3) alternative statistical estimation methods; and 4) clinical uncertainty.Two methods to explicitly characterize such uncertainties were identified: model selection and model averaging. In addition, an alternative approach, adding uncertain parameters to represent the source of uncertainty was also considered.The applications demonstrate that cost-effectiveness may be sensitive to these uncertainties and the methods used to characterize them. The value of research was particularly sensitive to these uncertainties and the methods used to characterize it. It is therefore important, for decision-making purposes, to incorporate such uncertainties into the modeling process.

CONCLUSION

Only parameterizing the uncertainty directly in the model can inform the decision to conduct further research to resolve this source of uncertainty.

Cost-effectiveness analysis of strategies for diagnosing celiac disease

OBJECTIVE

To compare strategies for diagnosing celiac disease (CD).

METHODS

A decision analytic model was used to compare five strategies on diagnostic performance and costs.

RESULTS

First, tTG screening alone is the least costly strategy ($22/individual). While the NPV is high (99.8%), the PPV is low (63.4%). Second, if tTG-positive patients undergo esophagogastroduodenoscopy (EGD) to confirm CD, the PPV increases to 100% ($2,237/false-positive diagnosis avoided). Third, if EGDs are restricted to only those who are both tTG and HLA DQ2/8 positive, costs are slightly reduced ($59 vs. $63/individual), while PPV and NPV remain unchanged. Fourth, screening tTG-negative patients for IgA deficiency increases the NPV to 99.9% ($32,605/false-negative diagnosis avoided). Sensitivity analyses revealed that as the prevalence of CD increases, the cost of avoiding a false-positive diagnosis by adding EGD to the tTG alone strategy increases considerably.

CONCLUSIONS

When the pre-test probability of CD is low, patients with positive tTG serology should undergo EGD with biopsy-either directly or after positive screening for HLA DQ2/8-to confirm CD. As the pre-test probability of CD increases, the added cost of EGD should be weighed against the consequences of a false-positive diagnosis. Routinely screening for IgA deficiency in order to avoid a false-negative diagnosis is quite costly.

Good practice guidelines for decision-analytic modelling in health technology assessment: a review and consolidation of quality assessment

The use of decision-analytic modelling for the purpose of health technology assessment (HTA) has increased dramatically in recent years. Several guidelines for best practice have emerged in the literature; however, there is no agreed standard for what constitutes a 'good model' or how models should be formally assessed. The objective of this paper is to identify, review and consolidate existing guidelines on the use of decision-analytic modelling for the purpose of HTA and to develop a consistent framework against which the quality of models may be assessed. The review and resultant framework are summarised under the three key themes of Structure, Data and Consistency. 'Structural' aspects relate to the scope and mathematical structure of the model including the strategies under evaluation. Issues covered under the general heading of 'Data' include data identification methods and how uncertainty should be addressed. 'Consistency' relates to the overall quality of the model. The review of existing guidelines showed that although authors may provide a consistent message regarding some aspects of modelling, such as the need for transparency, they are contradictory in other areas. Particular areas of disagreement are how data should be incorporated into models and how uncertainty should be assessed. For the purpose of evaluation, the resultant framework is applied to a decision-analytic model developed as part of an appraisal for the National Institute for Health and Clinical Excellence (NICE) in the UK. As a further assessment, the review based on the framework is compared with an assessment provided by an independent experienced modeller not using the framework. It is hoped that the framework developed here may form part of the appraisals process for assessment bodies such as NICE and decision models submitted to peer review journals. However, given the speed with which decision-modelling methodology advances, there is a need for its continual update.