The validity of QALYs: an experimental test of constant proportional tradeoff and utility independence

Are QALYs based on time trade-off comparable?--A systematic review of TTO methodologies

A wide range of methods is used to elicit quality-of-life weights of different health states to generate 'Quality-adjusted life years' (QALYs). The comparability between different types of health outcomes at a numerical level is the main advantage of using a 'common currency for health' such as the QALY. It has been warned that results of different methods and perspectives should not be directly compared in QALY league tables. But do we know that QALYs are comparable if they are based on the same method and perspective?The Time trade-off (TTO) consists in a hypothetical trade-off between living shorter and living healthier. We performed a literature review of the TTO methodology used to elicit quality-of-life weights for own, current health. Fifty-six journal articles, with quality-of-life weights assigned to 102 diagnostic groups were included. We found extensive differences in how the TTO question was asked. The time frame varied from 1 month to 30 years, and was not reported for one-fourth of the weights. The samples in which the quality-of-life weights were elicited were generally small with a median size of 53 respondents. Comprehensive inclusion criteria were given for half the diagnostic groups. Co-morbidity was described in less than one-tenth of the groups of respondents. For two-thirds of the quality-of-life weights, there was no discussion of the influence of other factors, such as age, sex, employment and children. The different methodological approaches did not influence the TTO weights in a predictable or clear pattern. Whether or not it is possible to standardise the TTO method and the sampling procedure, and whether or not the TTO will then give valid quality-of-life weights, remains an open question.This review of the TTO elicited on own behalf, shows that limiting cost-utility analysis to include only quality life weights from one method and one perspective is not enough to ensure that QALYs are comparable.

Cost-effectiveness versus cost-utility analysis of interventions for cancer: does adjusting for health-related quality of life really matter?

OBJECTIVE

The US Public Health Service Panel on Cost-Effectiveness has recommended the use of quality-adjusted life-years (QALYs) as the best way to estimate outcomes in a cost-effectiveness analysis. We evaluate the importance of this recommendation by assessing whether adjusting for health-related quality of life affects the ultimate resource allocation decision implied by the cost-effectiveness ratio for interventions aimed at cancer prevention and control.

METHODS

We identified 110 interventions in 39 articles for which both cost/life-year and cost/QALY were reported. Interventions were forms of prevention, early detection, or treatment of cancer. We calculated a Spearman correlation to assess the ordinal relationship between cost/life-year and cost/QALY. In addition, we employed various decision thresholds to assess whether the use of cost/life-year would yield different resource allocation decisions than the use of cost/QALY.

RESULTS

The correlation between cost/life-year and cost/QALY is 0.96 (P <.0001). Assuming a US dollars 50000 decision threshold, adjustment for quality of life would affect the implied choice in 5% of cases. With a US dollars 400000 threshold, adjustment for quality of life would affect choice for 2% of interventions.

CONCLUSIONS

For interventions aimed at cancer, the outcome measures of cost/life-year and cost/QALY are highly correlated with one another. Although adjusting for quality of life can make an important difference in the evaluation of alternative approaches to cancer prevention and control, it often does not.

A consistency test of the time trade-off

This paper tests the internal consistency of time trade-off utilities. We find significant violations of consistency in the direction predicted by loss aversion. The violations disappear for higher gauge durations. We show that loss aversion can also explain that for short gauge durations time trade-off utilities exceed standard gamble utilities. Our results suggest that time trade-off measurements that use relatively short gauge durations, like the widely used EuroQol algorithm, are affected by loss aversion and lead to utilities that are too high.

A test of the QALY model when health varies over time

Quality-adjusted life years (QALYs) estimate the utility derived from health profiles by taking account of life expectancy and quality of life. In applying QALYs to situations where health varies over time, it is usual to assume that we can add the utilities from constituent health states. This paper investigates the QALY approach to combining health states over time using two tests. The first test rejects additive independence, the central assumption of the QALY model, for individual respondents. The second test is equivocal. The tests are, therefore, unable to conclusively reject the QALY approach to combining health states over time.

The validity of time trade-off values in calculating QALYs: constant proportional time trade-off versus the proportional heuristic

In order to calculate quality adjusted life years (QALYs) from time trade-off (TTO) responses, individual preferences are required to satisfy the constant proportional time trade-off (CPTTO) assumption. Respondents who use a simple proportional heuristic may appear to satisfy CPTTO but will in fact generate preference reversals for states that are associated with a maximal endurable time (MET). Using data from 91 respondents, the study reported here examines the extent to which valuations satisfy the CPTTO assumption and the extent to which they might be generated by the proportional heuristic. The results suggest that respondents are using a proportional heuristic that casts doubt on the validity of using the TTO method to calculate QALYs for health states that are associated with MET preferences.

Are health states 'timeless'? A case study of an acute condition: post-chemotherapy nausea and vomiting

RATIONALE, AIMS AND OBJECTIVES

The objective was to test whether individuals' responses to standard gamble (SG) and visual analogue scale (VAS) questions do not depend on the time horizon of the health scenario presented.

METHODS

Face-to-face interviews were conducted in a convenience sample of 18 women aged 22-50 years with no history of breast cancer or cancer requiring chemotherapy. Data were collected from March 2000 to June 2000 at a university in the Midwest of the United States of America. Preference weights were estimated using SG top-down titration method and VAS scaled from zero (death) to one (perfect health). Subjects were asked to rate their preferences if faced with two scenarios: post-chemotherapy nausea and vomiting (PCNV) occurring for 3 days (scenario 1), and PCNV lasting for the rest of their lives (scenario 2). Three PCNV health states of varying severity were tested: complete alleviation, partial alleviation, and no alleviation.

RESULTS

Paired-t-test analysis showed statistically significantly lower preference weights (P < 0.05) when the health state was for the rest of the respondent's life vs. 3 days. Mean SG weights for scenario 1 vs. scenario 2 were: 0.968 vs. 0.927 (complete alleviation), 0.942 vs. 0.810 (partial alleviation) and 0.866 vs. 0.644 (no alleviation). Mean VAS weights for scenario 1 vs. scenario 2 were: 0.741 vs. 0.676 (complete alleviation), 0.490 vs. 0.307 (partial alleviation) and 0.276 vs. 0.136 (no alleviation).

DISCUSSION AND CONCLUSIONS

For the majority of respondents the utility independence assumption for SG and VAS did not hold. Similar to Bala et al., the results of this study indicated that preference weights as measured by SG and VAS techniques were not 'timeless'. Regardless of the preference measure used, both SG and VAS yielded higher scores when PCNV lasted for a shorter period of time.

Quality of life in patients with Barrett's esophagus undergoing surveillance

OBJECTIVES

Practice guidelines recommend surveillance for Barrett's esophagus (BE) because of the risk of esophageal cancer. The quality of life of patients undergoing surveillance is unknown. The objectives of this study were to develop a new utility instrument to measure quality of life of patients undergoing BE surveillance and determine if Quality of Life in Reflux and Dyspepsia (QOLRD) scores correlate with utility ratings.

METHODS

Fifteen patients were administered 16 scenarios describing possible BE surveillance outcomes. Each scenario was rated from 0 (equivalent to being dead) to 10 (equivalent to being in perfect health). Each patient also completed the QOLRD, a validated instrument. A t test was performed to compare the QOLRD means with published means. The Spearman's rank correlation coefficient was calculated for the median QOLRD score and the median utility rating.

RESULTS

QOLRD means ranged from 5.80 to 6.65 (previously published means 4.3-5.4). Lower scores denoted a worsened quality of life. The difference was significant (p < 0.001). The correlation coefficient of median QOLRD score (6.8) and median utility rating (4.0) was 0.10 (p = 0.71).

CONCLUSIONS

This population of BE patients had significantly higher QOLRD scores than a previously published population referred for endoscopy. Quality of life using the utility measure was reduced. The utility measure did not correlate with the disease-specific instrument, suggesting that the concerns of patients undergoing surveillance are distinct from their reflux symptoms.

The distribution problem in economic evaluation: income and the valuation of costs and consequences of health care programmes

To date, a common view in the health economics literature is that the applicability of cost-benefit analysis (CBA) is limited, due to the distribution problem which underlies its main method of valuation (e.g. willingness to pay). One view is that cost effectiveness analysis (CEA) overcomes these problems. We show that the same distributional concerns apply to non-monetary valuations of health consequences, to measurement of costs and to the decision rules of CEA. Hence adopting CEA over CBA cannot be justified on the basis of "avoiding" distributional considerations. The implications of our results are discussed, including alternative strategies for the use of "income-based" research findings in social decision-making.

Preference-based measures in economic evaluation in health care

Estimating preferences for states of health has been an active area of research in recent years. Unlike psychophysical approaches, which discriminate levels of health status, preference-based approaches incorporate values or utilities for health outcomes and can be used in cost-effectiveness analyses to aid resource allocation decisions. This chapter considers issues and controversies involved in using preference-based measures in economic evaluation in health care, with a particular emphasis on cost-utility analysis and the estimation of quality-adjusted life years. Topics considered include techniques for measuring preferences, the use of preference-based classification systems, the relationship between patient and community preferences, methods for obtaining utilities from clinical trials, mapping health status from health utilities, the development of "off-the-shelf" preference weights, and proposed alternatives to quality-adjusted life years. We also consider applications of cost-utility analyses to public health interventions. Although cost-utility analyses have become more popular recently, many challenges remain for the field. Widespread acceptance of the methodology likely awaits more consensus on measurement techniques, as well as educational efforts in the public health and medical communities on the usefulness of the approach.

On individual preferences and aggregation in economic evaluation in healthcare

For practical reasons, in order to carry out economic evaluations of collective decisions, total costs will generally be compared with total benefits; hence, individuals' willingness to pay (WTP) or quality-adjusted life-years (QALYs) have to be estimated at an aggregate level. So far, aggregation has usually been done by taking the individuals' mean WTP or the unweighted number of QALYs. Since the aggregation process is closely related to the way that income, health and/or utility of different individuals are compared and weighted, it also has significant equity implications. Thus. the explicit (or, more often, implicit) assumptions behind the aggregation process will largely affect how health and welfare are distributed is society. The aggregation problem in economic evaluation is certainly not trivial, but is seldom addressed in current practice. This paper shows the underlying assumptions of aggregate cost-benefit analysis (CBA) and cost-effectiveness analysis/cost-utility analysis (CEA/CUA), and it emphasises the particularly strong assumptions which have to be made when QALYs are interpreted as utilities in the welfare economics sense. Naturally, the appropriate method to choose depends on what is to be maximised: welfare or health. If decisions of resource allocation are to be based on economic welfare theory, then CBA should be preferred. However, if QALYs are interpreted as measures of health, rather than as utilities, then CEA/CUA would be appropriate.

On hypothetical bias and calibration in cost-benefit studies

Despite a sound foundation in economic welfare theory, willingness to pay (WTP) has not been used as a measure of benefits in economic evaluations of health and health care to the same extent as in other fields. Some have suggested that this is due to non-economists' reluctance to placing dollar values on the benefits of health care. However, another potential reason could be uncertainties about the validity of the WTP measure. In this paper, we outline the bias problems with the WTP method, and specifically focus on hypothetical bias; i.e. whether the WTP from hypothetical elicitation methods overstates the real WTP or not. This is done by examining the literature in this field, with emphasis on economic experiments where there is a greater possibility for comparison. The findings are that hypothetical WTP in general significantly overestimates real WTP, but that calibration methods to reduce or eliminate this difference are currently being developed. We conclude that while the area is still very much under development, there seem to be reasons to view the use of cost-benefit analysis as a reasonable alternative to the more common cost-effectiveness analysis.

Establishing health state valuations for disease specific states: an example from heart disease

This study considers the feasibility of defining a QALY from disease-specific data using the New York Heart Association (NYHA) classification of heart failure. The study derives health state values for the four different NYHA classifications of disease progression using the time trade-off (TTO) instrument associated with the five dimensional (EQ-5D) health state valuation method. Consistent mappings between the disease classification and the chosen QALY instrument are found. With this being the case, the assumption of constant proportionality, which is necessary to define the QALY as an acceptable measure of health related preferences, is considered. It is found that constant proportionality does not hold across the more severe health states, thus questioning the use of QALYs as representing cardinal preference structures.

A customisable framework for the assessment of therapies in the solution of therapy decision tasks

In current medical research, a growing interest can be observed in the definition of a global therapy-evaluation framework which integrates considerations such as patients preferences and quality-of-life results. In this article, we propose the use of the research results in this domain as a source of knowledge in the design of support systems for therapy decision analysis, in particular with a view to application in oncology. We discuss the incorporation of these considerations in the definition of the therapy-assessment methods involved in the solution of a generic therapy decision task, described in the context of AI software development methodologies such as CommonKADS. The goal of the therapy decision task is to identify the ideal therapy, for a given patient, in accordance with a set of objectives of a diverse nature. The assessment methods applied are based either on data obtained from statistics or on the specific idiosyncrasies of each patient, as identified from their responses to a suite of psychological tests. In the analysis of the therapy decision task we emphasise the importance, from a methodological perspective, of using a rigorous approach to the modelling of domain ontologies and domain-specific data. To this aim we make extensive use of the semi-formal object oriented analysis notation UML to describe the domain level.

Life-cycle preferences over consumption and health: when is cost-effectiveness analysis equivalent to cost-benefit analysis?

This paper studies life-cycle preferences over consumption and health status. We show that cost-effectiveness analysis is consistent with cost-benefit analysis if the lifetime utility function is additive over time, multiplicative in the utility of consumption and the utility of health status, and if the utility of consumption is constant over time. We derive the conditions under which the lifetime utility function takes this form, both under expected utility theory and under rank-dependent utility theory, which is currently the most important nonexpected utility theory. If cost-effectiveness analysis is consistent with cost-benefit analysis, it is possible to derive tractable expressions for the willingness to pay for quality-adjusted life-years (QALYs). The willingness to pay for QALYs depends on wealth, remaining life expectancy, health status, and the possibilities for intertemporal substitution of consumption.

Are health states "timeless"? The case of the standard gamble method

The standard gamble method, as currently recommended for use in health care program evaluation, provides an individual's preference score or "utility weight" for living in a given health state for the rest of the individual's life. Many researchers interpret this value as a time-independent or "timeless" one and order health states on a scale of zero (death) to one (full health), regardless of the time spent in the health state. This article examines whether preference scores for a severe pain health state are "timeless," or in other words whether the utility independence assumption is satisfied. Our study results suggest that for the majority of respondents, the preference scores are not independent of time.

A comparison of patient and social tariff values derived from the time trade-off method

A social tariff of EuroQol time trade-off (TTO) values was recently presented. We compared the social tariff and patient TTO values among 104 women with mild and severe menopausal symptoms. The social tariff and patient TTO values were elicited both after and before hormone replacement therapy (HRT). There was a close correspondence between social-tariff values and patient TTO values for relatively good health states, whereas the social tariff TTO values were lower than the patient TTO values for severe health states.

Probability Weighting and Utility Curvature in QALY-Based Decision Making

Quality-Adjusted Life-Years (QALYs) are currently the most important utility model in medical decision making. QALYs are calculated by adjusting years of life for the utility of the health state in which these years are spent. For normative reasons the standard gamble is the preferred method to measure health state utilities, but concern exists about its descriptive properties. Recent theoretical work has suggested that probability weighting can explain anomalies in standard gamble measurement. This paper shows that applying probability weighting in standard gamble measurement increases the consistency of QALYs with individual preferences. The consistency of QALYs with individual preferences is not significantly increased further if utility curvature is also taken into account. Copyright 1999 Academic Press.

Holistic versus composite preferences for lifetime treatment sequences for type 2 diabetes

This study of patient preferences for lifetime treatment sequences for type 2 diabetes had three objectives: to assess the feasibility of obtaining holistic preference assessments using the time-tradeoff (TTO) technique; to compare composite and holistic preference scores for the same lifetime treatment paths; and to assess the validity of composite and holistic preference measures in terms of their congruence with an individual's rank-order preferences. 101 persons with type 2 diabetes provided preference ratings for hyperglycemic treatments lasting 30 years, including eight discrete treatment states and four treatment paths. Scenarios described drug and glucose-testing regimens, efficacy of glucose control, and side effects. After ranking and rating scenarios on a thermometer scale, subjects provided TTO preferences for each treatment state or path scenario. Holistic assessment of treatment paths was feasible with the TTO technique, in terms of useable data (88% of interviews) and effect on coefficients of variation. Holistic and composite preference scores were not statistically different. Agreement was poor between rankings implied by holistic and composite scores and direct rankings. The authors conclude that lifetime treatment paths with minor differences in health effects can be assessed using either composite (QALY) or holistic (HYE) measures. The validity of these TTO-based preference measures remains unknown.

Assessment of the time-tradeoff values for prophylactic mastectomy of women with a suspected genetic predisposition to breast cancer

BACKGROUND

Female carriers of the breast-cancer-susceptibility genes BRCA1 and BRCA2 are at high risk for breast cancer (85%). They face the choice between prophylactic mastectomy (PM) and breast cancer screening. For this treatment choice, a shared-decision-making program was developed. In this program, the time tradeoff (TTO) was used to assess preferences for PM.

PURPOSE

Assessment of the feasibility, constant proportional tradeoff, and reliability of using the TTO for this purpose.

METHODS

Fifty-four women suspected to carry the BRCA1/2 mutation were provided with comprehensive relevant information. Their preferences for PM were assessed on two occasions. Discrepancies between preferences indicated by the two tests were resolved by testing a third time. The preferences assessed on the last occasion were used for individual decision analyses. In order to test constant proportional tradeoff, the TTO consisted of four items with different numbers of life years.

RESULTS

Forty-two women (78%) completed the TTO twice and nine women (17%) performed the test a third time. Three women (5%) completed the TTO only once. The mean TTO value for PM at the last replication was 0.69 (SD=0.30). Violations of constant proportional tradeoff were significant: the largest tradeoffs were recorded for the shortest durations. Pearson's correlation coefficient between the TTO values for the two last sessions was 0.96.

CONCLUSION

Assessment of individual preferences by the TTO in this patient group is feasible and reliable. Therefore, the TTO can be used in clinical settings to elicit treatment preferences of women proven or suspected to have a genetic predisposition to breast cancer.

Sequence effects, health profiles, and the QALY model: in search of realistic modeling

The authors conducted an experiment to determine whether the sequence of presentation of states in a health profile would affect the valuations assigned to them. The empirical task was part of a large standardized experiment involving 104 students. Thirteen health states were valued using two variations of the time-tradeoff method. At the group level, a small but distinct overall effect of the sequence of the tradeoffs was detected after accounting for discounting effects. The respondents were not preference-indifferent concerning the sequence of health states presented. Detailed analysis at the individual level indicated that the overall sequence effect was attributable to two groups of respondents who were sensitive to the sequence of events. One small group, referred to as "best-things-first" respondents, preferred the best years first; the other group, classified as "happy-end" respondents, preferred the reverse sequence. The majority of the respondents, however, were indifferent to the sequence. These results suggest that 1) in valuation experiments involving the time-tradeoff method and 2) in applying valuation results to the evaluation of real-life health consequences, a varying lifetime health profile may not be regarded as simply a chain of independent separately valued and discounted QALY periods. Even elementary valuation tasks cannot safely assume ignorance of prognosis, as the additive utility independence assumption of the QALY model does not hold. The sequence effect at least supplements the conventional general time-preference concept, and specific strategies are suggested to disentangle quantitatively the sequence effect and the time-preference effect.

Use of influence diagrams to structure medical decisions

Influence diagrams are compact representations of decision problems that are mathematically equivalent to decision trees. The authors present five important principles for structuring a decision as an influence diagram: 1) start at the value node and work back to the decision nodes; 2) draw the arcs in the direction that makes the probabilities easiest to assess; 3) use informational arcs to specify which events will have been observed at the time each decision is made; 4) ensure that missing arcs reflect intentional assertions about conditional independence and the timing of observations; and 5) ensure that there are no cycles in the influence diagram. They then build an influence diagram for the problem of staging non-small-cell lung cancer as an illustration. Influence diagrams offer several strengths for structuring medical decisions. They represent graphically and compactly the probabilistic relationships between parameters in the model. Influence diagrams also allow the model to be structured in a fashion that eases the necessary probability assessments, regardless of whether the assessments are based on available evidence or on expert judgment. Influence diagrams provide an important complement to decision trees, especially for representing probabilistic relationships among variables in a decision model.