Qualitative and Quantitative Analysis of Interviewer Help Answering the Time Tradeoff

The Egyptian EQ-5D-5L Extensive Pilot Study: Lessons Learned

OBJECTIVES

To investigate the effect of an extensive pilot phase in improving protocol compliance, face validity, reduction of interviewer effect and prediction errors in the composite time trade-off (cTTO) data elicited as part of the EQ-5D-5L valuation study in Egypt.

METHODS

This study used the cTTO data and quality control (QC) reports from the Egyptian EQ-5D-5L valuation study. Three-level mixed models were estimated to test whether interviewer effects were reduced during the pilot phase and subsequent rounds of collected cTTO data. Ordinary least square (OLS) regression analysis was conducted for each interviewer separately to test whether the mean absolute error (MAE) improved as interviewers completed more interviews. Moreover, improvement in protocol compliance, face validity and reduction of prediction errors in the cTTO data were tested.

RESULTS

1180 interviews were conducted by nine interviewers and included in the final analysis, of which 206 interviews were pilot and 974 interviews were actual. There was substantial improvement in the face validity and reduction of prediction errors in the cTTO data where the MAE of the actual data was 0.37, which is much lower than that of the pilot data, which was 0.44. However, there was an initial high level of protocol compliance in terms of the four indicators of the QC tool and the variance attributed to the interviewers was small throughout the whole study.

CONCLUSIONS

This study clarified the benefits of the pilot phase and the strict implementation of the QC tool in improving the face validity and the prediction accuracy of the cTTO data. However, a more extensive pilot phase may be more beneficial in EQ-5D-5L valuation studies that have issues initially with protocol compliance and interviewer effects.

Completing the time trade-off with respondents who are older, in poorer health or with an immigrant background in an EQ-5D-5L valuation study

OBJECTIVES

To determine the effects of age, immigrant background, and poor self-reported health in a general population sample on the probability of non-completion or slow completion of the time trade-off (TTO).

METHODS

We used data from an interrupted Norwegian EQ-5D-5L valuation study conducted between 2019 and 2020. All participants responded to background items, irrespective of completion. We used mixed effect logistic regression analysis to assess the effect of old age, poor health, and immigrant background on the probability of non-completion of the TTO, and, for those who completed the TTO, of slow completion times.

RESULTS

First experiences from a Norwegian valuation study were that 29 (5.5%) respondents failed to complete the TTO tasks. For those reporting age over 65 years, poor health, or an immigrant background, 12% failed to complete the TTO. Adjusted odds ratios for predictors of non-completion were statistically significant (age > 65 years, 8.3; EQ-VAS ≤ 50, 3.49; immigrant background, 4.56). Being over 65 years or with an immigrant background also predicted slow completion of both the introduction and TTO tasks.

CONCLUSIONS

High age, poor health, and immigrant status increased the risk of not being able to complete the TTO tasks, and of slow completion. Higher non-completion rates and increased completion times suggest that elements of the TTO may be demanding for some respondent groups, with possible implications for representativeness.

Time trade-off with someone to live for: impact of having significant others on time trade-off valuations of hypothetical health states

Background

The TTO task involves giving up life years, i.e. living a shorter life, to avoid an undesirable health state. Despite being a hypothetical task, some respondents take other life factors into account when completing the task. This study explored the effect of having children and/or a partner on TTO valuations of hypothetical EQ-5D-5L health states in a valuation study of the general population.

Methods

The study used TTO data collected in a Norwegian EQ-5D-5L valuation study in 2019–2020, by one-to-one pc-assisted interviews following the EQ-VT protocol. We used regression modelling to determine the effect of significant others (having children or a partner) on disutility per health state from the TTO valuations.

Results

430 respondents were included [mean age 43.8 (SD 15.9) years, 58% female, 48% with children, 68% with a partner, 25% with neither children nor partner]. Having children and/or a partner was associated with lowered willingness to trade life years translating to higher elicited health state utilities (p < 0.01).

Conclusion

Having significant others, or the lack of having significant others, was associated with respondents’ valuation of hypothetical health states using TTO, more so than traditional sampling variables such as age and sex. Inadequate representativeness in terms of having significant others could bias health state preference values in valuation studies.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11136-021-03026-6.

Increasing respondent engagement in composite time trade-off tasks by imposing three minimum trade-offs to improve data quality

BACKGROUND

Web-based surveys are increasingly utilized for health valuation studies but may be more prone to lack of engagement and, therefore, poor data validity. The objective of this study was to evaluate the effect of imposed engagement (i.e., at least three trade-offs) in the composite time trade-off (cTTO) task.

METHODS

The EQ-5D-5L valuation study protocol and study design were adapted for online, unsupervised completion in two arms: base case and engagement. Validity of preferences was assessed using the prevalence of inconsistent valuations and expected patterns of TTO values. Respondent task engagement was measured using time per task. Value sets were generated using linear regression with a random intercept (RILR).

RESULTS

The base case (n = 501) and engagement arms (n = 504) clustered at different TTO values: [base case] 0, 1; [engagement] -0.5, 0.45, 0.6. Mean TTO values were lower for the engagement arm. Engagement respondents did not spend more time per TTO task: [base case] 63.3 s (SD 77.9 s); [engagement] 64.7 s (SD 73.3 s); p = 0.36. No significant difference was found between arms for prevalence of respondents with at least one inconsistent TTO value: [base case] 61.1%; [engagement] 63.5%; p = 0.43. Both value sets had significant intercepts far from 1: [base case] 0.846; [engagement] 0.783. The relative importance of the EQ-5D dimensions also differed between arms.

CONCLUSIONS

Both online arms had poor quality data. A minimum trade-off threshold did not improve engagement nor face validity of the data, indicating that modifications to the number of iterations are insufficient alone to improve data quality/validity of online TTO studies.

Comparison of online and face-to-face valuation of the EQ-5D-5L using composite time trade-off

OBJECTIVE

The aim of this study was to compare online, unsupervised and face-to-face (F2F), supervised valuation of EQ-5D-5L health states using composite time trade-off (cTTO) tasks.

METHODS

The official EuroQol experimental design and valuation protocol for the EQ-5D-5L of 86 health states were implemented in interviewer-assisted, F2F and unsupervised, online studies. Validity of preferences was assessed using prevalence of inconsistent valuations and expected patterns of TTO values. Respondent task engagement was measured using number of trade-offs and time per task. Trading patterns such as better-than-dead only was compared between modes. Value sets were generated using linear regression with a random intercept (RILR). Value set characteristics such as range of scale and dimension ranking were evaluated between modes.

RESULTS

Five hundred one online and 1,134 F2F respondents completed the surveys. Mean elicited TTO values were higher online than F2F when compared by health state severity. Compared to F2F, a larger proportion of online respondents did not assign the poorest EQ-5D-5L health state (i.e., 55555) the lowest TTO value ([Online] 41.3% [F2F] 12.2%) (p < 0.001). A higher percentage of online cTTO tasks were completed in 3 trade-offs or fewer ([Online] 15.8% [F2F] 3.7%), (p < 0.001). When modeled using the RILR, the F2F range of scale was larger than online ([Online] 0.600 [F2F] 1.307) and the respective dimension rankings differed.

CONCLUSIONS

Compared to F2F data, TTO tasks conducted online had more inconsistencies and decreased engagement, which contributed to compromised data quality. This study illustrates the challenges of conducting online valuation studies using the TTO approach.

An overview of the time trade-off method: concept, foundation, and the evaluation of distorting factors in putting a value on health

INTRODUCTION

Preference-based instruments measuring health status express the value of specific health states in a single number. One method used is time trade-off (TTO). Health-status values are key elements in calculating quality-adjusted life years (QALYs) and are pertinent for resource allocation. Since they are used in economic evaluations of healthcare, searching for a theoretical foundation of TTO in economics is justified.

AREA COVERED

This paper provides an overview of TTO, including its relation to economic theory, and discusses biases and distortions, compiled from recent and older research. Inconsistencies between TTO and random utility theory were detected; The TTO is confounded by time preferences and by respondents' life expectancies. TTO is cognitively challenging, therefore guidance during the interviews is needed, producing interview effects. TTO does not measure one thing at a time, nor are the values independent of other states that are being valued in the same task. That is, TTO does not exhibit theoretical measurement properties such as unidimensionality and the invariance principle.

EXPERT OPINION

We conclude that the TTO may be a pragmatic method of eliciting health state values, but the limitations in regard to measurement theory and practical elicitation problems makes it prone to inconsistencies and arbitrariness.

Assessing the Use of a Feedback Module to Model EQ-5D-5L Health States Values in Hong Kong

BACKGROUND

An international valuation protocol exists for obtaining societal values for each of the 3125 health states of the five-level EuroQol-five dimensions (EQ-5D-5L) questionnaire. A feedback module (FM) that can be related to theoretical models used in behavioral economics was recently included in this protocol.

OBJECTIVES

Our objective was to assess the impact of using an FM to estimate an EQ-5D-5L value set in Hong Kong.

METHODS

EQ-5D-5L health states were elicited using a composite time trade-off (C-TTO) and a discrete-choice (DC) experiment. Use of the FM according to participant characteristics and the impact of the FM on the number of inconsistent C-TTO responses were assessed. We employed a main-effects hybrid model that combined data from both elicitation techniques.

RESULTS

In total, 1014 individuals completed the survey. The sample was representative of the general Chinese Hong Kong population in terms of sex, educational attainment, marital status, and most age groups but not for employment status. The use of the FM reduced the number of C-TTO inconsistencies. Participant characteristics differed significantly between those who used and did not use the FM. The model without a constant resulted in logical consistent coefficients and was therefore selected as the model to produce the value set. The predicted EQ-5D-5L values ranged from -0.864 to 1.

CONCLUSIONS

The use of an FM to allow participants to exclude C-TTO responses reduced the number of inconsistent responses and improved the quality of the data when estimating an EQ-5D-5L value set in Hong Kong.

Employing quality control and feedback to the EQ-5D-5L valuation protocol to improve the quality of data collection

Objectives

In valuing health states using generic questionnaires such as EQ-5D, there are unrevealed issues with the quality of the data collection. The aims were to describe the problems encountered during valuation and to evaluate a quality control report and subsequent retraining of interviewers in improving this valuation.

Methods

Data from the first 266 respondents in an EQ-5D-5L valuation study were used. Interviewers were trained and answered questions regarding problems during these initial interviews. Thematic analysis was used, and individual feedback was provided. After completion of 98 interviews, a first quantitative quality control (QC) report was generated, followed by a 1-day retraining program. Subsequently individual feedback was also given on the basis of follow-up QCs. The Wilcoxon signed-rank test was used to assess improvements based on 7 indicators of quality as identified in the first QC and the QC conducted after a further 168 interviews.

Results

Interviewers encountered problems in recruiting respondents. Solutions provided were: optimization of the time of interview, the use of broader networks and the use of different scripts to explain the project’s goals to respondents. For problems in interviewing process, solutions applied were: developing the technical and personal skills of the interviewers and stimulating the respondents’ thought processes. There were also technical problems related to hardware, software and internet connections. There was an improvement in all 7 indicators of quality after the second QC.

Conclusion

Training before and during a study, and individual feedback on the basis of a quantitative QC, can increase the validity of values obtained from generic questionnaires.

How dead is dead? Qualitative findings from participants of combined traditional and lead-time time trade-off valuations

PURPOSE

The EuroQol Valuation Technology (EQ-VT) uses traditional time trade-off (tTTO) for health states better than dead and lead-time TTO (LT-TTO) for states worse than dead to elicit a value (-1.0 to +1.0) for each health state. In the Canadian EQ-5D-5L Valuation study which used the EQVT platform, we observed an unexpected peak in frequency of "0" values and few negative values, particularly in the range of 0 to -0.5. To better understand this finding, we sought to explore respondents' thought processes while valuing a health state, and their understanding of the tTTO and LT-TTO exercises.

METHODS

Qualitative semi-structured interviews were conducted with EQVT task respondents. Questions focused on valuations of health states as: (a) Same as dead in tTTO, (b) Worse than dead in tTTO but changed to same as dead in LT-TTO, (c) Worse than dead in LT-TTO, and (d) Worse than dead in LT-TTO with trading off all 10 years. Data were analyzed using content and thematic analysis.

RESULTS

Mean age of participants (N = 70) was 40 ± 18.1 years, 60% female, and 76% Caucasian. Participants provided similar reasons for valuing a health state same as or worse than dead. Many participants expressed confusion about worse than dead valuations, distinction between same as and worse than dead, and the transition from tTTO to LT-TTO. A few indicated that the addition of 10 years of full health in the LT-TTO influenced their valuations.

CONCLUSIONS

The transition from tTTO to LT-TTO in the EQVT was confusing to participants, whereby some health state valuations around this transition appeared to be arbitrary.