In Pursuit of Anchoring Vignettes That Work: Evaluating Generality Versus Specificity in Vignette Texts

Estonian National Mental Health Study: Design and methods for a registry-linked longitudinal survey

OBJECTIVES

The Estonian National Mental Health Study (EMHS) was conducted in 2021-2022 to provide population-wide data on mental health in the context of COVID-19 pandemic. The main objective of this paper is to describe the rationale, design, and methods of the EMHS and to evaluate the survey response.

METHODS

Regionally representative stratified random sample of 20,000 persons aged 15 years and older was drawn from the Estonian Population Register for the study. Persons aged 18 years and older at the time of the sampling were enrolled into three survey waves where they were invited to complete an online or postal questionnaire about mental well-being and disorders, and behavioral, cognitive, and other risk factors. Persons younger than 18 years of age were invited to fill an anonymous online questionnaire starting from wave 2. To complement and validate survey data, data on socio-demographic, health-related, and environmental variables were collected from six national administrative databases and registries. Additionally, a subsample was enrolled into a validation study using ecological momentary assessment.

RESULTS

In total, 5636 adults participated in the survey wave 1, 3751 in wave 2, and 4744 in wave 3. Adjusted response rates were 30.6%, 21.1%, and 27.6%, respectively. Women and older age groups were more likely to respond. Throughout the three survey waves, a considerable share of adult respondents screened positive for depression (27.6%, 25.1%, and 25.6% in waves 1, 2, and 3, respectively). Women and young adults aged 18 to 29 years had the highest prevalence of depression symptoms.

CONCLUSIONS

The registry-linked longitudinal EMHS dataset comprises a rich and trustworthy data source to allow in-depth analysis of mental health outcomes and their correlates among the Estonian population. The study serves as an evidence base for planning mental health policies and prevention measures for possible future crises.

Relationships Between Self-Rated Health at Three Time Points: Past, Present, Future

Background: Multiple studies have shown that people who have experienced a serious health problem such as an injury tend to overrate the quality of health they had before that event. The main objective of this study was to test whether the phenomenon of respondents overrating their past health can also be observed in people from the general population. A second aim was to test whether habitual optimism is indeed focused on events in the future.

Method: A representatively selected community sample from Leipzig, Germany (n = 2282, age range: 40–75 years) was examined. Respondents were asked to assess their current health, their past health (5 years before), and their expected future health (in 5 years) on a 0–100 scale. In addition, the study participants completed several questionnaires on specific aspects of physical and mental health.

Results: Respondents of all age groups assessed their health as having been better in the past than it was at present. Moreover, they also assessed their earlier state of health more positively than people 5 years younger did their current state. Habitual optimism was associated with respondents having more positive expectations of how healthy they will be in 5 years time (r = 0.37), but the correlation with their assessments of their current health was nearly as high (r = 0.36).

Conclusion: Highly positive scores of retrospectively assessed health among people who have experienced a health problem cannot totally be accounted for by a response to that health problem.

The importance of development standards for anchoring vignettes: an illustrative example from pediatric localized scleroderma quality of life

PURPOSE

Anchoring vignettes (AVs) are a promising measurement technique to reduce bias in patient-reported outcome (PRO) measures by helping researchers understand differences in how individuals and groups interpret response options. However, little attention has been paid to ensure quality development of AVs, and their performance has not been well assessed in pediatric populations. In this study, we explore the application of a rigorous development process for AVs based upon current standards for PROs, as well as feasibility of AVs when administered to children and adolescents.

METHODS

We developed AVs using a rigorous, patient-centered mixed methods process including three phases: (1) development, (2) a pilot study, and (3) a field test. Our proposed process included the generation of a conceptual framework based on the PRO, the Localized Scleroderma Quality of Life Instrument, and numerous vignette-specific considerations. We qualitatively explored readability and comprehension of the AVs (pilot study) and then analyzed ranking patterns within vignette sets (field test).

RESULTS

Four sets of four vignettes were developed. Revisions were suggested at each phase of development. The pilot study demonstrated that children ≥ 10 years had no trouble indicating understanding of the AVs. In the field test, although appropriate rankings of vignettes were generally demonstrated by participants, the percentage of tied rankings was higher than expected in this pediatric group.

CONCLUSIONS

This work supports the need for rigorous developmental standards for AVs, as each stage of development suggested revisions. Additionally, AVs showed initial promise for use with pediatric populations; general feasibility and understanding were supported.

Comparing South Korean and US self-rated health using anchoring vignettes

PURPOSE

This study compares the self-rated health (SRH) of South Korean and US respondents, using anchoring vignettes to adjust for reporting heterogeneity, i.e., cross-national differences in use of response categories.

METHODS

Study participants, ages 20 to 59, were recruited from online survey panels retained by Embrain in Korea (N = 1170) and the US (N = 1033). In the analyses, we first examined the two key measurement assumptions of the anchoring vignette method. Response consistency was examined by regressing SRH on vignette ratings and EQ-5D, and was satisfied when the relationship between SRH and vignette ratings was significant and positive. Vignette equivalence was assessed by examining whether respondents correctly rank-ordered vignettes by health status severity. We then compared estimates of the between-country difference in SRH from two models: a standard ordered probit regression, and a hierarchical ordered probit (Hopit) regression, which adjusted for cross-country differences in use of response categories.

RESULTS

The anchoring vignettes satisfied both measurement assumptions. Results from the ordered probit regression indicated that Korean SRH was worse than that in the US. However, results from the Hopit regression revealed that Korean SRH was actually better than that in the US, after adjusting for Korean respondents' significantly higher intercategory thresholds (demarcating between "very bad" and "bad," "bad" and "fair," etc.).

CONCLUSION

The apparently lower SRH of Korean vis-à-vis US respondents is an artifact of Koreans' higher standards for health-related response categories. After adjusting for these different standards, Korean SRH is revealed to be higher than US SRH.

Associations between self-rated health and the assessments of anchoring vignettes in cardiovascular patients

Background/Objective

Assessments of health can be biased by response shift effects. One method for detecting such effects is the use of anchoring vignettes. The aim of this study was to analyze the relationship between participants' self-assessed health state and their assessments of these vignettes.

Method

A total of 342 cardiovascular patients assessed their own state of health on a 0-100 visual analogue scale. The patients additionally assessed two vignettes featuring fictional persons suffering from specific complaints. A sample of the general population (N = 1,236) served as controls.

Results

The participants rated the health state of the vignette character featuring physical problems as being significantly better than the general population did (effect size: d = 0.53). The group difference in the assessment of the vignette featuring primarily mental health problems was lower (d = -0.17). Participants' assessments of the vignettes were positively correlated with their assessments of their own health state (r = .26 and r = .10) and with several quality of life variables.

Conclusions

Anchoring vignettes are a useful tool for detecting response shift effects.

Challenges and lessons learned from using anchoring vignettes to explore quality of life response behavior

Purpose

Asking patients to rate health-related quality of life (HRQoL) of hypothetical individuals described in anchoring vignettes has been proposed to enhance knowledge on how patients understand and respond to HRQoL questionnaires. In this article, we describe the development of anchoring vignettes and explore their utility for measuring response shift in patients’ self-reports of HRQoL.

Methods

We conducted an explorative mixed-methods study. One hundred patients with multiple sclerosis or psoriasis participated in two interviews at intervals of 3–6 months. During both interviews, patients assessed HRQoL of 16 hypothetical individuals on the SF-12 questionnaire (two vignettes for each of the eight domains of the SF-12). In addition to these quantitative ratings, we used the think-aloud method to explore changes in patients’ verbalization of their decision processes during vignette ratings.

Results

Agreement of vignette ratings at baseline and follow-up was low (ICCs < 0.55). In addition, paired sample t-tests revealed no significant directional mean changes in vignette ratings. Thus, ratings changed non-directionally, neither confirming retest reliability nor a systematic change of assessment. Furthermore, patients’ verbalization of their decision processes did not indicate whether or not the assessment strategy of individual patients had changed.

Conclusions

Patients’ ratings of anchoring vignettes fluctuate non-directionally over time. The think-aloud method appears not to be informative in exploring whether these fluctuations are due to changes in the individual decision process. Overall, vignettes might not be an appropriate approach to explore response shift, at least with regard to the specific target population and the use of the SF-12.

Electronic supplementary material

The online version of this article (10.1007/s11136-020-02488-4) contains supplementary material, which is available to authorized users.