Power analyses for measurement model misspecification and response shift detection with structural equation modeling

PURPOSE

Statistical power for response shift detection with structural equation modeling (SEM) is currently underreported. The present paper addresses this issue by providing worked-out examples and syntaxes of power calculations relevant for the statistical tests associated with the SEM approach for response shift detection.

METHODS

Power calculations and related sample-size requirements are illustrated for two modelling goals: (1) to detect misspecification in the measurement model, and (2) to detect response shift. Power analyses for hypotheses regarding (exact) overall model fit and the presence of response shift are demonstrated in a step-by-step manner. The freely available and user-friendly R-package lavaan and shiny-app 'power4SEM' are used for the calculations.

RESULTS

Using the SF-36 as an example, we illustrate the specification of null-hypothesis (H0) and alternative hypothesis (H1) models to calculate chi-square based power for the test on overall model fit, the omnibus test on response shift, and the specific test on response shift. For example, we show that a sample size of 506 is needed to reject an incorrectly specified measurement model, when the actual model has two-medium sized cross loadings. We also illustrate power calculation based on the RMSEA index for approximate fit, where H0 and H1 are defined in terms of RMSEA-values.

CONCLUSION

By providing accessible resources to perform power analyses and emphasizing the different power analyses associated with different modeling goals, we hope to facilitate the uptake of power analyses for response shift detection with SEM and thereby enhance the stringency of response shift research.

Mental health after lumbar spine surgery: cognitive appraisal processes and outcome in a longitudinal cohort study

BACKGROUND CONTEXT

A not uncommon finding following spine surgery is that many patients do not achieve mental health improvement up to population norms for their age cohort, despite improvement in pain and functioning.

PURPOSE

This study examined how patients who were categorized as depressed versus not depressed think about health-related quality of life as assessed by cognitive-appraisal processes. It examined cross-sectional and longitudinal differences over 12 months post-surgery.

DESIGN

Prospective longitudinal cohort study with data collected at pre-surgery and at ∼3- and ∼12-months post-surgery from August 2013 to August 2023.

PATIENT SAMPLE

We included 173 adults undergoing lumbar spine surgery for degenerative spinal conditions at an academic medical center. The study sample was 47% female, with a mean age of 61 (SD=15.0), and a median level of education of college graduate.

OUTCOME MEASURES

Depression was defined as a Mental Component Score (MCS)≤38 on the Rand-36, building on studies that equated MCS scores with significant depression as assessed by clinically validated depression scales. The Quality-of-Life Appraisal Profile assessed the cognitive-appraisal domains of Experience Sampling and Standards of Comparison.

METHODS

The analysis focused on two comparisons: cross-sectionally comparing those who were not depressed (n=82) to those who were depressed (n=77) at baseline; and comparing longitudinal trajectories among those depressed before surgery and improved (n=54) versus did not improve (n=23). T-tests characterized group differences in appraisal endorsement; analysis of variance evaluated appraisal items in terms of explained variance; and Pearson correlation coefficients assessed direction of association in predicting mental health.

RESULTS

There were pre-surgical and longitudinal differences in both cognitive appraisal domains. Before surgery, depressed patients were less likely than non-depressed patients to endorse emphasizing the positive; more likely to focus on worst moments, recent flare-ups, their spinal condition, and the future; and more likely to compare themselves to high aspirations (eg, perfect health). Over time, among those who were depressed before surgery, those who improved focused decreasingly on worst moments and on the time before their spinal condition, and increasingly on emphasizing the positive and balancing the positives/negatives. Appraisal explained more variance in mental health among those who did not improve as compared to those who did, at all timepoints. All appraisal items were more highly correlated with mental health among those who remained depressed as compared to those who improved, particularly over time.

CONCLUSIONS

Endorsement of cognitive appraisal processes was different for depressed versus non-depressed spine-surgery patients before surgery and distinguished those who were depressed before surgery and improved versus those who did not improve. These findings suggest that targeted interventions could be beneficial for addressing mental health concerns during the spine surgery recovery trajectory. These interventions might use appraisal measures to identify patients likely to remain depressed after surgery, and then focus on helping these patients shift their focus and standards of comparison.

Response shift in coronary artery disease

PURPOSE

Patients with coronary artery disease (CAD) experience significant angina symptoms and lifestyle changes. Revascularization procedures can result in better patient-reported outcomes (PROs) than optimal medical therapy (OMT) alone. This study evaluates the impact of response shift (RS) on changes in PROs of patients with CAD across treatment strategies.

METHODS

Data were from patients with CAD in the Alberta Provincial Project on Outcome Assessment in Coronary Heart Disease (APPROACH) registry who completed the 16-item Canadian version of the Seattle Angina Questionnaire at 2 weeks and 1 year following a coronary angiogram. Multi-group confirmatory factor analysis (MG-CFA) was used to assess measurement invariance across treatment groups at week 2. Longitudinal MG-CFA was used to test for RS according to receipt of coronary artery bypass grafting (CABG), percutaneous coronary intervention (PCI), or optimal medical therapy (OMT) alone.

RESULTS

Of the 3116 patients included in the analysis, 443 (14.2%) received CABG, 2049(65.8%) PCI, and the remainder OMT alone. The MG-CFA revealed a partial-strong invariance across the treatment groups at 2 weeks (CFI = 0.98, RMSEA [90% CI] = 0.05 [0.03, 0.06]). Recalibration RS was detected on the Angina Symptoms and Burden subscale and its magnitude in the OMT, PCI, and CABG groups were 0.32, 0.28, and 0.53, respectively. After adjusting for RS effects, the estimated target changes were largest in the CABG group and negligible in the OMT group.

CONCLUSION

Adjusting for RS is recommended in studies that use SAQ-CAN to assess changes in patients with CAD who have received revascularization versus OMT alone.

Response shift results of quantitative research using patient-reported outcome measures: a descriptive systematic review

PURPOSE

The objective of this systematic review was to describe the prevalence and magnitude of response shift effects, for different response shift methods, populations, study designs, and patient-reported outcome measures (PROM)s.

METHODS

A literature search was performed in MEDLINE, PSYCINFO, CINAHL, EMBASE, Social Science Citation Index, and Dissertations & Theses Global to identify longitudinal quantitative studies that examined response shift using PROMs, published before 2021. The magnitude of each response shift effect (effect sizes, R-squared or percentage of respondents with response shift) was ascertained based on reported statistical information or as stated in the manuscript. Prevalence and magnitudes of response shift effects were summarized at two levels of analysis (study and effect levels), for recalibration and reprioritization/reconceptualization separately, and for different response shift methods, and population, study design, and PROM characteristics. Analyses were conducted twice: (a) including all studies and samples, and (b) including only unrelated studies and independent samples.

RESULTS

Of the 150 included studies, 130 (86.7%) detected response shift effects. Of the 4868 effects investigated, 793 (16.3%) revealed response shift. Effect sizes could be determined for 105 (70.0%) of the studies for a total of 1130 effects, of which 537 (47.5%) resulted in detection of response shift. Whereas effect sizes varied widely, most median recalibration effect sizes (Cohen's d) were between 0.20 and 0.30 and median reprioritization/reconceptualization effect sizes rarely exceeded 0.15, across the characteristics. Similar results were obtained from unrelated studies.

CONCLUSION

The results draw attention to the need to focus on understanding variability in response shift results: Who experience response shifts, to what extent, and under which circumstances?

Individual differences in the long-term impact of the pandemic: moderators of COVID-related hardship, worry, and social support

PURPOSE

Understanding people's response to the pandemic needs to consider individual differences in priorities and concerns. The present study sought to understand how individual differences in cognitive-appraisal processes might moderate the impact of three COVID-specific factors-hardship, worry, and social support-on reported depression.

METHODS

This longitudinal study of the psychosocial impact of the COVID-19 pandemic included 771 people with data at three timepoints over 15.5 months. Participants were recruited from panels of chronically ill or general population samples. Depression was measured by an item response theory validated depression index created using items from existing measures that reflected similar content to the Patient Health Questionnaire-8. COVID-specific factors of hardship, worry, and social support were assessed with items compiled by the National Institutes of Health. The Quality of Life Appraisal Profilev2 Short-Form assessed cognitive appraisal processes. A series of random effects models examined whether appraisal moderated the effects of hardship, worry, and social support on depression over time.

RESULTS

Over time the association between low social support and depression was greater (p = 0.0181). Emphasizing the negative was associated with exacerbated depression, in particular for those with low social support (p = 0.0007). Focusing on demands and habituation was associated with exacerbated depression unless one experienced greater hardship (p = 0.0074). There was a stronger positive connection between recent changes and depression for those people with higher worry scores early in the pandemic as compared to later, but a stronger positive correlation for those with lower worry scores later in the pandemic (p = 0.0015). Increased endorsement of standards of comparison, emphasizing the negative, problem goals, and health goals was associated with worse depression scores (all p < 0.0001). People who were younger, disabled, or had greater difficulty paying bills also reported worse depression (p < 0.0001, 0.0001, and 0.002, respectively).

CONCLUSION

At the aggregate level, COVID-specific stressors changed over the course of the pandemic, whereas depression and social-support resources seemed stable. However, deeper analysis revealed substantial individual differences. Cognitive-appraisal processes showed considerable variability across individuals and moderated the impact of COVID-specific stressors and resources over time. Future work is needed to investigate whether coaching individuals away from maladaptive cognitive-appraisal processes can reduce depression and lead to better overall well-being.

Meaning in Life in Late-Stage Parkinson's Disease: Results from the Care of Late-Stage Parkinsonism Study (CLaSP) in Six European Countries

The Care of Late-Stage Parkinsonism (CLaSP) study is a longitudinal, multicentre, prospective cohort study to assess the needs and provision of care for people with late-stage Parkinson's disease and their caregivers in six European countries. As a cross-sectional study within the CLaSP study, 509 people with Parkinson's disease completed the "Schedule-for-Meaning-in-Life-Evaluation" (SMiLE) questionnaire. We compared the results to those of a representative sample of healthy participants (n = 856). People with late-stage Parkinson's disease reported family, partnership and spirituality as the greatest areas of importance. Overall, they had lower SMiLE indices compared to healthy participants. People with late-stage Parkinson's disease rated the importance of core meaning in life areas (namely family, social relations and health) as significantly lower than the representative cohort and they also rated satisfaction as significantly lower in most areas. In conclusion, people with late-stage Parkinson's disease do have areas where they can find meaning, such as family, partnership and spirituality. However, they indicate a lack of fulfilment of their individual MiL, reflected by low satisfaction rates in the majority of meaning in life categories. The need for spiritual support for people with Parkinson's disease indicates the important role of chaplains to help people with Parkinson's disease maintain meaning in life.

Response shift in parent-reported psychopathology in children with chronic physical illness

PURPOSE

Because physical-mental comorbidity in children is relatively common, this study tested for response shift (RS) in children with chronic physical illness using a parent-reported measure of child psychopathology.

METHODS

Data come from Multimorbidity in Children and Youth across Life-course (MY LIFE), a prospective study of n = 263 children aged 2-16 years with physical illness in Canada. Parents provided information on child psychopathology using the Ontario Child Health Study Emotional Behavioral Scales (OCHS-EBS) at baseline and 24 months. Oort's structural equation modeling was used to test for different forms of RS in parent-reported assessments between baseline and 24 months. Model fit was evaluated using root mean square error of approximation (RMSEA), comparative fit index (CFI), and standardized root mean residual (SRMR).

RESULTS

There were n = 215 (81.7%) children with complete data and were included in this analysis. Of these, n = 105 (48.8%) were female and the mean (SD) age was 9.4 (4.2) years. A two-factor measurement model provided good fit to the data [RMSEA (90% CI) = 0.05 (0.01, 0.10); CFI = 0.99; SRMR = 0.03]. Non-uniform recalibration RS was detected on the conduct disorder subscale of the OCHS-EBS. This RS effect had negligible impact on the longitudinal change in externalizing and internalizing disorders construct over time.

CONCLUSIONS

Response shift detected on the conduct disorder subscale of the OCHS-EBS, indicated that parents of children with physical illness may recalibrate their responses on child psychopathology over 24 months. Researchers and health professionals should be aware of RS when using the OCHS-EBS to assess child psychopathology over time.

Depression trajectories during the COVID-19 pandemic: a secondary analysis of the impact of cognitive-appraisal processes

PURPOSE

This study characterized depression trajectories during the COVID pandemic and investigated how appraisal and changes in appraisal over time related to these depression trajectories.

METHODS

This longitudinal study of the psychosocial impact of the COVID-19 pandemic included 771 people with data at three timepoints over 15.5 months. The depression index was validated using item-response-theory methods and receiver-operating-characteristic curve analysis. The Quality of Life (QOL) Appraisal Profile_v2 Short-Form assessed cognitive-appraisal processes. Sequence analysis characterized depression-trajectory groups, and random effects models examined appraisal main effects, appraisal-by-group, and appraisal-by-group-by-time interactions.

RESULTS

Sequence analysis generated six trajectory groups: Stably Well (n = 241), Stably Depressed (n = 299), Worsening (n = 79), Improving (n = 83), Fluctuating Pattern 1 (No-Yes-No; n = 41), and Fluctuating Pattern 2 (Yes-No-Yes; n = 28). While all groups engaged in negative appraisal processes when they were depressed, the Stably Depressed group consistently focused on negative aspects of their life. Response-shift effects were revealed such that there were differences in the appraisal-depression relationship over time for standards of comparison and recent changes for the Stably Depressed, and in health goals for those Getting Better.

CONCLUSION

The present work is, to our knowledge, the first study of response-shift effects in depression. During these first 15.5 pandemic months, group differences highlighted the connection between negative appraisals and depression, and response-shift effects in these relationships over time. Egregious life circumstances may play a lesser role for the Stably Depressed but a greater role for people who have transient periods of depression as well as for those with improving trajectories (i.e., endogenous vs. reactive depression). How one thinks about QOL is intrinsically linked to mental health, with clear clinical implications.

Response shift in hearing related quality of life after cochlear implantation - effect size and clinical significance: a then-test study

BACKGROUND

Quality of life questionnaires are often used in the assessment of rehabilitation of hearing-impaired patients with a cochlear implant. However, a prospective study with a systematic retrospective evaluation of the preoperative quality of life after surgery has not yet been conducted and may reveal a change in internal standards, such as a response shift, due to the implantation and hearing rehabilitation.

METHODS

The Nijmegen Cochlear Implant Questionnaire (NCIQ) was used for assessing hearing related quality of life. It has three general domains (physical, psychological and social) and six subdomains. Seventeen patients were tested before (t₀) and retrospectively (then-test; pre-t₁) and acutely postoperative (post-t₁) after cochlear implantation. Observed changes, then-test changes, response shifts and effect sizes were calculated. Non-parametric statistical methods were used.

RESULTS

The NCIQ total score was 52.32 ± 18.69 (mean, standard deviation) for t₀, 59.29 ± 14.06 for pre-t₁ and 67.65 ± 26.02 for post-t₁ questioning. The observed change was statistically significant in all domains but in speech production. Response shift was statistically significant in the total score and in part of the domains. The effect sizes for the response shift were moderate (> 0.5) in the total score, psychological, social general scores and subdomains.

CONCLUSIONS

In this study we found that response shift does exist in adults with severe to profound hearing loss undergoing cochlear implantation. By advising the participants to deactivate the implant for the then-test, recall bias and noise were minimized. The clinical significance of the response shift was present in the total score and in the social and psychological domains.

TRIAL REGISTRATION

This study was retrospectively registered with the German Clinical Trial Register, TRN DRKS00029467, on 07/08/2022.

"I knew who I was this morning, but I've changed a few times since then": A study combining register and self-reported QoL data in assessing how response shift may occur in an injury population

AIMS

The phenomenon of response shift may occur when people are exposed to a traumatic event and reevaluate the meaning of important aspects of their lives; a phenomenon that a traditional pre- and post-assessment cannot catch. Hence, the aim of the study was to increase the knowledge of how response shift may occur in people who have suffered an injury.

METHODS

The current study is a register-based cohort study including 2512 participants. Data were retrieved from the Swedish LifeGene project and an online questionnaire, including the EQ5D. In order to analyze how response shift may occur, a "Then-test" and Structural Equation Model (SEM) were used.

RESULTS

The results showed a clear indication of response shift through recalibration. The SEM analysis confirmed that participants significantly shifted their response between the post- and retrospective measurements. Significant differences were found for the VAS and for two of the five EQ5D dimensions: anxiety/depression and pain.

CONCLUSION

This study provides additional knowledge to how response shift can occur in an injury population. This information can guide the next generation of QoL measures and be used as guidance for designing interventions for those suffering injuries. Furthermore, it may have an impact on how to interpret evaluations of interventions.

Impact of response shift effects in the assessment of self-reported depression during treatment: Insights from a rTMS versus Venlafaxine randomized controlled trial

PURPOSE

Patient-Reported Outcomes are essential to properly assess treatment effectiveness in randomized clinical trial (RCT) for Major Depressive Disorder (MDD). MDD self-assessment may vary over time depending on change in the meaning of patients' self-evaluation of depression, i.e. Response Shift (RS). Our aim was to investigate RS and its impact on different depression domains in a clinical trial comparing rTMS versus Venlafaxine.

METHODS

The occurrence and type of RS was determined using Structural Equation Modeling applied to change over time in 3 domains (Sad Mood, Performance Impairment, Negative Self-Reference) of the short-form Beck Depression Inventory (BDI-13) in a secondary analysis of a RCT on 170 patients with MDD treated by rTMS, venlafaxine or both.

RESULTS

RS was evidenced in the venlafaxine group in the Negative Self-Reference and Sad Mood domains.

CONCLUSION

RS effects differed between treatment arms in self-reported depression domains in patients with MDD. Ignoring RS would have led to a slight underestimation of depression improvement, depending on treatment group. Further investigations of RS and advancing new methods are needed to better inform decision making based on Patient-Reported Outcomes.

Implications of the syntheses on definition, theory, and methods conducted by the Response Shift - in Sync Working Group

PURPOSE

Our aim is to advance response shift research by explicating the implications of published syntheses by the Response Shift - in Sync Working Group in an integrative way and suggesting ways for improving the quality of future response shift studies.

METHODS

Members of the Working Group further discussed the syntheses of the literature on definitions, theoretical underpinnings, operationalizations, and response shift methods. They outlined areas in need of further explication and refinement, and delineated additional implications for future research.

RESULTS

First, the proposed response shift definition was further specified and its implications for the interpretation of results explicated in relation to former, published definitions. Second, the proposed theoretical model was further explained in relation to previous theoretical models and its implications for formulating research objectives highlighted. Third, ways to explore alternative explanations per response shift method and their implications for response shift detection and explanation were delineated. The implications of the diversity of the response shift methods for response shift research were presented. Fourth, the implications of the need to enhance the quality and reporting of the response shift studies for future research were sketched.

CONCLUSION

With our work, we intend to contribute to a common language regarding response shift definitions, theory, and methods. By elucidating some of the major implications of earlier work, we hope to advance response shift research.

A change of perspective? An explorative study on why patients may not subjectively report cognitive impairments after a cardiac arrest

AIM

Cardiac arrest survivors are at risk of long-term cognitive impairment. Patients with cognitive impairments do not always have cognitive complaints and vice versa. Not reporting cognitive complaints could be caused by a lack of awareness. We hypothesized that caregivers report more cognitive failures than patients, indicating patients' lack of insight into cognitive functioning.

METHODS

This is a secondary analysis of the Activity and Life After Survival of Cardiac Arrest study on survivors of cardiac arrest and their caregivers. They were assessed at two weeks, three months, and one year after cardiac arrest. At each time point, the patient and the caregiver filled out the cognitive failure questionnaire (CFQ) regarding the patient. We analysed the correlation, intraclass correlation, and self-proxy discrepancy between patients and caregivers on the CFQ over time.

RESULTS

One-hundred-and-nineteen cardiac arrest survivors (mean age = 60, 85 % male) and their caregivers were included. The CFQ scores of the patients and caregivers were equally low. The correlation (T1 r = 0.31; T2 r = 0.40; T3 r = 0.55) and intraclass correlation (T1 r = 0.48; T2 r = 0.56; T3 r = 0.71) between patient and caregiver increased over time.

CONCLUSION

This study does not support a lack of awareness of cognitive impairments by long-term cardiac arrest survivors. Future research may focus on alternative explanations for why patients have less cognitive complaints than expected based on the frequency of cognitive impairments. Possible explanations include a response shift.

The mediating role of appraisal on health-related quality of life in adolescent and young adult cancer survivors

BACKGROUND

Adolescent and young adult (AYA) cancer survivors (first diagnosed with cancer at age 15-39) are distinct within the cancer community due to their unique challenges and diverse psycho-behavioral characteristics. This study aimed to analyze psycho-behavioral pathways and further explore the mediating role of cognitive appraisals on AYA cancer survivors' quality of life (QoL).

METHODS

Three hundred and eighty-nine AYA cancer survivors were eligible for analyses and recruited to self-administer questionnaires on QoL (the Chinese version of EORTC Quality of Life Questionnaire-C30 v3.0), resilience, coping, and appraisal on site. This study performed structural equation modeling (SEM) to examine pathways on QoL based on the Rapkin & Schwartz QoL Appraisal Model.

RESULTS

The average age of participants (47.6% female) was 32.7 ± 4.1 years. The SEM results closely fit the measured data (RMSEA = 0.053, GFI = 0.955, CFI = 0.964, SRMR = 0.052). The final model showed direct negative effects of later clinical-stage, more comorbidities, and more Acceptance-Resignation coping on QoL; indirect positive effects of better resilience on QoL through less Acceptance-Resignation coping (β = 0.286, P = 0.002). Appraisal mediated the effects of treatment and resilience on QoL (β = -0.024, P = 0.038). Further, Calm, Peaceful, and Active appraisal patterns were associated with improved Cognitive Functioning (β = 0.119, P = 0.009).

CONCLUSION

Appraisal, coping, and resilience could significantly mediate the effects of cancer and its treatment on the QoL of AYA cancer survivors. Future interventions targeting cognitive appraisals and psycho-behaviors will be helpful. Figuring out what matters to such a unique population and how they appraise a cancer diagnosis through treatment trajectories could help nurses adjust support.

Midwives' empathy and shared decision making from women's perspective - sensitivity of an assessment to compare quality of care in prenatal and obstetric care

Background

For quality-oriented evaluation of prenatal and obstetric care, it is important to systematically consider the perspective of the women receiving care in order to comprehensively assess and optimize quality in a woman-centered manner. Empathy and Shared Decision Making (SDM) are essential components of woman-centered midwifery care. The aim of the study was to analyze measurement invariance of the items of the Consultation and Relational Empathy (CARE) and Shared Decision Making-Questionnaire (SDM-Q-9) scales depending on the prenatal versus obstetric care setting.

Methods

One hundred fifty women retrospectively assessed aspects of woman-centered midwifery care in both prenatal and obstetric care setting. The birth of the child was a maximum of 12 months ago. A structural equation modelling approach was adopted to separate true effects from response shift (RS) effects depending on care setting. The latter were analyzed in terms of recalibration (changing women’s internal measurement standards), Reprioritization (changing associations of items and construct) as well as Reconceptualization (redefining the target construct).

Results

A response shift model was identified for both assessments (pregnancy/birth: CFI = .96/.96; SRMR = .046/.051). At birth, both scales indicated lower quality of care compared with prenatal care (SDM-Q-9-M/CARE-8-M:|d| = 0.190/0.392). Although no reconceptualization is required for the items of both scales, RS effects are evident for individual items. Due to recalibration and reprioritization effects, the true differences in the items are partly underestimated (SDM-Q-9-M/CARE-8-M: 3/2 items) or overestimated (4/2 items).

Conclusion

The structure of the constructs SDM and Empathy, indicating woman-centered midwifery care, are moderated by the care settings. To validly assess midwives’ empathy and shared decision making from women’s perspective, setting-dependent response shift effects have to be considered. The proven item-specific response effects contribute to a better understanding of construct characteristics in woman-centered care by midwives during pregnancy and childbirth.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12884-022-05041-y.

Performance of a Rasch-based method for group comparisons of longitudinal change and response shift at the item level in PRO data: A simulation study

The growing interest in patient perception and experience in healthcare has led to an increase in the use of patient-reported outcomes (PRO) data. However, chronically ill patients may regularly adapt to their disease and, as a consequence, might change their perception of the PRO being measured. This phenomenon named response shift (RS) may occur differently depending on clinical and individual characteristics. The RespOnse Shift ALgorithm at the Item level (ROSALI), a method for RS analysis at the item level based on Rasch models, has recently been extended to explore heterogeneity of item-level RS between two groups of patients. The performances of ROSALI in terms of RS detection at the item level and biases of estimated differences in latent variable means were assessed. A simulation study was performed to investigate four scenarios: no RS, RS in only one group, RS affecting both groups either in a similar or a different way. Performances of ROSALI were assessed using rates of false detection of RS when no RS was simulated and a set of criteria (presence of RS, correct identification of items and groups affected by RS) when RS was simulated. Rates of false detection of RS were low indicating that ROSALI satisfactorily prevents from mistakenly inferring RS. ROSALI is able to detect RS and identify the item and group(s) affected when RS affects all response categories of an item in the same way. The performances of ROSALI depend mainly on the sample size and the degree of heterogeneity of item-level RS.

Examining minimal important change of the Canadian Occupational Performance Measure for subacute rehabilitation hospital inpatients

Background

The Canadian Occupational Performance Measure (COPM) is an individualized patient-reported outcome designed to evaluate the self-perceptions of a patient’s occupational performance. Our study aimed to examine the minimal important change (MIC) in inpatients undergoing subacute rehabilitation. The MIC values were calculated using the three different anchor-based analyses with the transition index as an external criterion; the mean change method (MIC_MeanChange), the receiver operating characteristic (MIC_ROC) analysis, and the predictive modeling method adjusted for the proportion of improved patients (MIC_adjust). In this study, the MIC_adjust value was considered as the most valid statistical method. We recruited 100 inpatients with various health conditions from subacute rehabilitation hospitals. Data were collected twice: an initial assessment and a reassessment one month later. The systematic interview format (Five Ws and How) was used for both the initial and second assessments to prevent information bias (response shift).

Results

Three patients who indicated deterioration on the transition index were excluded from all analyses, and 97 patients were analyzed in this study. The MIC_adjust values were 2.20 points (95% confidence interval 1.80–2.59) for the COPM performance score and 2.06 points (95% confidence interval 1.73–2.39) for the COPM satisfaction score. The MIC_MeanChange and MIC_ROC values were considered less reasonable to interpret because the proportions of the improved patients subgroup were more than 50% (82.5%).

Conclusions

The MIC_adjust value estimates from this study can help detect whether the patients’ perceived occupational performance improved or did not change. The results support the multidisciplinary use of COPM in clinical practice and research on subacute rehabilitation inpatients.

If it's information, it's not "bias": a scoping review and proposed nomenclature for future response-shift research

BACKGROUND

The growth in response-shift methods has enabled a stronger empirical foundation to investigate response-shift phenomena in quality-of-life (QOL) research; but many of these methods utilize certain language in framing the research question(s) and interpreting results that treats response-shift effects as "bias," "noise," "nuisance," or otherwise warranting removal from the results rather than as information that matters. The present project will describe the various ways in which researchers have framed the questions for investigating response-shift issues and interpreted the findings, and will develop a nomenclature for such that highlights the important information about resilience reflected by response-shift findings.

METHODS

A scoping review was done of the QOL and response-shift literature (n = 1100 articles) from 1963 to 2020. After culling only empirical response-shift articles, raters characterized how investigators framed and interpreted study research questions (n = 164 articles).

RESULTS

Of 10 methods used, papers using four of them utilized terms like "bias" and aimed to remove response-shift effects to reveal "true change." Yet, the investigators' reflections on their own conclusions suggested that they do not truly believe that response shift is error to be removed. A structured nomenclature is proposed for discussing response-shift results in a range of research contexts and response-shift detection methods.

CONCLUSIONS

It is time for a concerted and focused effort to change the nomenclature of those methods that demonstrated this misinterpretation. Only by framing and interpreting response shift as information, not bias, can we improve our understanding and methods to help to distill outcomes with and without response-shift effects.

Evaluation of the link between the Guttman errors and response shift at the individual level

PURPOSE

Methods for response shift (RS) detection at the individual level could be of great interest when analyzing changes in patient-reported outcome data. Guttman errors (GEs), which measure discrepancies in respondents' answers compared to the average sample responses, might be useful for detecting RS at the individual level between two time points, as RS may induce an increase in the number of discrepancies over time. This study aims to establish the link between recalibration RS and the change in the number of GEs over time (denoted index [Formula: see text]) via simulations and explores the discriminating ability of this index.

METHODS

We simulated the responses of individuals affected or not affected by recalibration RS (defined as changes in the patients' standard of measurement) to determine whether simulated individuals with recalibration had a greater change in the number of GEs over time than individuals without recalibration. The effects of factors related to the sample, the questionnaire structure and recalibration were investigated. As an illustrative example, the change in the number of GEs was computed in patients suffering from eating disorders.

RESULTS

Within simulations, simulated individuals affected by recalibration had, on average, a greater change in the number of GEs over time than did individuals without RS. Some of the parameters related to the questionnaire structure and recalibration magnitude appeared to have substantial effects on the values of [Formula: see text]. Discriminating abilities appeared, however, globally low.

CONCLUSION

Some evidence of the link between recalibration and the change in GEs was found in this study. GEs could be a valuable nonparametric tool for RS detection at a more individual level, but further investigation is needed.

Scale Norming Undermines the Use of Life Satisfaction Scale Data for Welfare Analysis

Scale norming is where respondents use qualitatively different scales to answer the same question across survey waves. It makes responses challenging to compare intertemporally or interpersonally. This paper develops a formal model of the cognitive process that could give rise to scale norming in year on year responses to life satisfaction scale questions. It then uses this model to conceptually differentiate scale norming from adaptation and changes in reference points. Scale norming could make life satisfaction responses misleading with regards to the changing welfare of individuals. In particular, individuals who would say that their life is "improving" or "going well" might nonetheless give the same scale response year after year. This has negative implications for the use of scales in cost-benefit analysis and other welfarist applications. While there is already substantial empirical evidence for the existence of scale norming, its implications for welfare analysis are sometimes understated on the grounds that this evidence might simply be the product of errors of memory. The paper presents new empirical evidence for scale norming from two surveys (N1 = 278; N2 = 1050) designed such that errors of memory are an unconvincing explanation for the results.

Norm-based comparison of the quality-of-life impact of ravulizumab and eculizumab in paroxysmal nocturnal hemoglobinuria

Aims

Paroxysmal nocturnal hemoglobinuria (PNH) is a rare and life-threatening intravascular hematologic disorder with significant morbidity and premature mortality. Clinical trials (NCT02946463 and NCT03056040) comparing ravulizumab with eculizumab for PNH have supported the non-inferiority of the former and similar safety and tolerability. This secondary analysis compared PNH trial participants after 26 weeks on either treatment (n = 438) to a general-population sample (GenPop) (n = 15,386) and investigated response-shift effects.

Methods

Multivariate analysis of covariance (MANCOVA) investigated function and symptom scores on the European Organisation for Research and Treatment of Cancer (EORTC) QLQ-C30 of people with PNH as compared to GenPop, after covariate adjustment. Risk-factor groups were created based on clinical indicators known to be associated with worse PNH outcomes, and separate MANCOVAs were computed for lower- and higher-risk-factor groups. Differential item functioning (DIF) analyses examined whether item response varied systematically (1) by treatment, (2) compared to GenPop, and (3) over time, the latter two suggesting and reflecting response-shift effects, respectively. DIF analyses examined 24 items from scales with at least two items. Recalibration response shift was operationalized as uniform DIF over time, reflecting the idea that, for a given group, the difficulty of endorsing an item changes over time, after adjusting for the total subscale score. Reprioritization response shift was operationalized as non-uniform DIF over time, i.e., the relative difficulty of endorsing an item over time changes across the total domain score.

Results

Across PNH risk-factor levels, people who had been on either treatment for 26 weeks reported better-than-expected functioning and lower symptom burden compared to GenPop. Ravulizumab generally showed larger effect sizes. Results were similar for lower and higher PNH risk factors, with slightly stronger effects in the former. DIF analyses revealed no treatment DIF, but did uncover group DIF (9 items with uniform DIF, and 11 with non-uniform) and DIF over time (7 items with uniform DIF, and 3 with non-uniform).

Conclusions

This study revealed that people with PNH on ravulizumab or eculizumab for 26 weeks reported QOL levels better than those of the general population. Significant effects of DIF by group and DIF over time support recalibration and reprioritization response-shift effects. These findings suggest that the treatments enabled adaptive changes.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13023-021-02016-8.

Creating idiometric short-form measures of cognitive appraisal: balancing theory and pragmatics

BACKGROUND

The Rapkin and Schwartz appraisal theory and measure provided a path toward documenting response-shift effects and describing individual differences in ways of thinking about quality of life (QOL) that distinguished people in different circumstances. Recent work developed and validated the QOL Appraisal Profile_{version 2} (QOLAP_v2), an 85-item measure that taps response-shift-detection domains of Frame of Reference, Standards of Comparison, Sampling of Experience, and Combinatory Algorithm. Recent theoretical work proposed that appraisal measurement constitutes a new class of measurement (idiometric), distinct from psychometric and clinimetric. To validate an idiometric measure, one would document that its items reflect different circumstances and population characteristics, and explain variance in QOL. The present work sought to develop idiometric short-forms of the QOLAP_v2 item bank by examining which items were most informative, retaining the appraisal-domain structure.

METHODS

This secondary analysis (n = 1481) included chronically-ill patients and their caregivers from a longitudinal web-based survey (mean follow-up 16.6 months). Data included the QOLAP_v2, the Center for Disease Control Healthy Days Core Module, the PROMIS-10 Global Health, and demographic/medical variables. Appraisal items were measured at baseline (relevant to understanding cognitive appraisal processes); and with change scores (sensitive to response-shift effects). Multivariate analysis of covariance examined what demographic and health-status change variables were reflected by each of 85 appraisal items (in five sets), as dependent variables, and other demographic/medical variables. Multiple linear regression examined how appraisal items explained variance in global physical- and mental-health change, after covariate adjustment. A tally summarized item performance across all five sets of cross-sectional and longitudinal analyses.

RESULTS

The vast majority (i.e., 80%) of the QOLAP_v2 items performed well across the analyses presented. Using a relatively strict criterion of explaining meaningful variance across 60% of analyses, one would retain 68 items. A more lenient criterion (40%) would retain 71.

CONCLUSIONS

The present study provides heuristics to support investigators' creating 'discretionary' QOLAP_v2 short-forms to fit their study aim and amplifying individual differences in the cognitive processes underlying QOL. This approach enables adapting the measure to the study population, as per the expectation that respondent populations differ in the predominant cognitive processes used.

Investigation of measurement invariance in longitudinal health-related quality of life in preemptive or previously dialyzed kidney transplant recipients

PURPOSE

Kidney transplantation (KT) can impact patients' evaluation of health-related quality of life (HRQoL) as they adapt to their new life with a graft and its changes. Patients may adapt to KT in a different way, depending on whether they were on dialysis prior to transplantation or not (i.e. preemptive group). This may result in lack of measurement invariance between these patients' groups and/or over time (i.e. response shift, RS) which may invalidate the between-group comparison of HRQoL change scores. The aim of this study was to investigate and compare RS before and after KT between these two patients' groups. Measurement invariance was investigated between groups and over time with three measurement occasions.

METHODS

Adult patients completed the SF-36 at the last visit before KT, and 3, 6 months after. A structural equation model-based procedure was used to (i) detect and take into account measurement non-invariance between groups and RS, if appropriate, (ii) identify the period of occurrence of RS, (iii) study the heterogeneity of RS between the two groups.

RESULTS

Before KT (i.e. baseline), measurement invariance was not rejected between dialyzed (n = 196) and preemptive (n = 178) patients' groups. Between baseline and 3 months after KT, similar uniform recalibration was detected on the general health domain in both groups. Uniform recalibration was found between 3- and 6 months after KT on the vitality domain for preemptive patients only.

CONCLUSION

HRQoL, adjusted for RS, increased overall for preemptive and dialyzed kidney transplant patients after transplantation. RS may reflect differing adaptation processes following KT.

Response shift after coronary revascularization

PURPOSE

The aims of this study were to investigate (1) the extent to which response shift occurs among patients with coronary artery disease (CAD) after coronary revascularization, (2) whether the assessment of changes in health-related quality of life (HRQoL), controlled for response shift, yield more valid estimates of changes in HRQoL, as indicated by stronger associations with criterion measures of change, than without controlling for response shift, and (3) if occurrences of response shift are related to patient characteristics.

METHODS

Patients with CAD completed the SF-36 and the Seattle Angina Questionnaire (SAQ7) at baseline and 3 months after coronary revascularization. Sociodemographic, clinical and psychosocial variables were measured with the patient version of the New York Heart Association-class, Subjective Significance Questionnaire, Reconstruction of Life Events Questionnaire (RE-LIFE), and HEXACO personality inventory. Oort's Structural Equation Modeling (SEM) approach was used to investigate response shift.

RESULTS

191 patient completed questionnaires at baseline and at 3 months after treatment. The SF-36 showed recalibration and reprioritization response shift and the SAQ7 reconceptualization response shift. Controlling for these response shift effects did not result in more valid estimates of change. One significant association was found between reprioritization response shift and complete integration of having CAD into their life story, as indicated by the RE-LIFE.

CONCLUSION

Results indicate response shift in HRQoL following coronary revascularization. While we did not find an impact of response shift on the estimates of change, the SEM approach provides a more comprehensive insight into the different types of change in HRQoL following coronary revascularization.

Response shift in patient-reported outcomes: definition, theory, and a revised model

Purpose

The extant response shift definitions and theoretical response shift models, while helpful, also introduce predicaments and theoretical debates continue. To address these predicaments and stimulate empirical research, we propose a more specific formal definition of response shift and a revised theoretical model.

Methods

This work is an international collaborative effort and involved a critical assessment of the literature.

Results

Three main predicaments were identified. First, the formal definitions of response shift need further specification and clarification. Second, previous models were focused on explaining change in the construct intended to be measured rather than explaining the construct at multiple time points and neglected the importance of using at least two time points to investigate response shift. Third, extant models do not explicitly distinguish the measure from the construct. Here we define response shift as an effect occurring whenever observed change (e.g., change in patient-reported outcome measures (PROM) scores) is not fully explained by target change (i.e., change in the construct intended to be measured). The revised model distinguishes the measure (e.g., PROM) from the underlying target construct (e.g., quality of life) at two time points. The major plausible paths are delineated, and the underlying assumptions of this model are explicated.

Conclusion

It is our hope that this refined definition and model are useful in the further development of response shift theory. The model with its explicit list of assumptions and hypothesized relationships lends itself for critical, empirical examination. Future studies are needed to empirically test the assumptions and hypothesized relationships.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11136-021-02846-w.

Do patients suffering from chronic diseases retrospectively overestimate how healthy they were before they fell ill?

Background/Objective: To examine the impact of a disease on a person's subjective health state, patients are often asked to assess their current health state and to retrospectively assess how healthy they were before they fell ill. The objective of this study was to test whether patients generally overestimated the quality of their pre-disease health. Method: Six samples of patients with chronic diseases (cancer patients, cardiovascular patients, and patients diagnosed with sarcoidosis, N between 197 and 1,197) were analyzed. The patients assessed their current health states and their health states at the time before diagnosis. The retrospective scores were compared with matched data from general population studies. Results: In three of the six studies, the retrospective health ratings of the patients were significantly higher than the general population norms (effect sizes between 0.24 and 0.46), two studies yielded nonsignificant effects, and in one study there was an opposite trend. The general overestimation of pre-disease health was more pronounced in older patients as compared with younger ones, and it was more pronounced when global health/quality of life was to be assessed. Conclusions: Retrospective assessments of pre-disease health states are not appropriate for assessing disease-related changes in a person's health state.

Detecting response shift in health-related quality of life measurement among patients with hypertension using structural equation modeling

BACKGROUND

Outcomes derived from longitudinal self-reported health-related quality of life measures can be confounded by response shift. This study was aimed to detect response shift among patients with hypertension attending a community-based disease management program.

METHODS

240 consecutive consulting or follow-up patients with diagnosed hypertension were recruited. The Short Form 36-item Health Survey was self-administered at 12 community health service stations at baseline and four weeks after attending the program. The 4-step structural equation modeling approach assessed response shift.

RESULTS

Data from 203 (84.6%) patients were eligible for analyses (mean age 65.9 ± 10.8 years, 46.3% female). The results showed uniform recalibration of social functioning ([Formula: see text](1) = 22.98, P < 0.001), and non-uniform recalibration of role limitations due to physical problems ([Formula: see text](1) = 8.84, P = 0.003), and bodily pain ([Formula: see text](1) = 17.41, P < 0.001). The effects of response shift on social functioning were calculated as "small" (effect-size = 0.35), but changed the observed changes from improvement (effect-size = 0.25) to slight deterioration (effect-size = -0.10). After accounting for the response shift effect, the general physical health of participants was improved (effect-size = 0.37), while deterioration (effect-size = -0.21) in the general mental health was also found.

CONCLUSIONS

Recalibration existed among patients with hypertension attending the disease management program. The interventions in the program might act as a catalyst that induced the response shift. We conclude that response shift should be considered in hypertension research with longitudinal health-related quality of life data.

Implications of response shift for micro-, meso-, and macro-level healthcare decision-making using results of patient-reported outcome measures

PURPOSE

Results of patient-reported outcome measures (PROMs) are increasingly used to inform healthcare decision-making. Research has shown that response shift can impact PROM results. As part of an international collaboration, our goal is to provide a framework regarding the implications of response shift at the level of patient care (micro), healthcare institute (meso), and healthcare policy (macro).

METHODS

Empirical evidence of response shift that can influence patients' self-reported health and preferences provided the foundation for development of the framework. Measurement validity theory, hermeneutic philosophy, and micro-, meso-, and macro-level healthcare decision-making informed our theoretical analysis.

RESULTS

At the micro-level, patients' self-reported health needs to be interpreted via dialogue with the clinician to avoid misinterpretation of PROM data due to response shift. It is also important to consider the potential impact of response shift on study results, when these are used to support decisions. At the meso-level, individual-level data should be examined for response shift before aggregating PROM data for decision-making related to quality improvement, performance monitoring, and accreditation. At the macro-level, critical reflection on the conceptualization of health is required to know whether response shift needs to be controlled for when PROM data are used to inform healthcare coverage.

CONCLUSION

Given empirical evidence of response shift, there is a critical need for guidelines and knowledge translation to avoid potential misinterpretations of PROM results and consequential biases in decision-making. Our framework with guiding questions provides a structure for developing strategies to address potential impacts of response shift at micro-, meso-, and macro-levels.

Critical examination of current response shift methods and proposal for advancing new methods

Purpose

This work is part of an international, interdisciplinary initiative to synthesize research on response shift in results of patient-reported outcome measures. The objective is to critically examine current response shift methods. We additionally propose advancing new methods that address the limitations of extant methods.

Methods

Based on literature reviews, this critical examination comprises design-based, qualitative, individualized, and preference-based methods, latent variable models, and other statistical methods. We critically appraised their definition, operationalization, the type of response shift they can detect, whether they can adjust for and explain response shift, their assumptions, and alternative explanations. Overall limitations requiring new methods were identified.

Results

We examined 11 methods that aim to operationalize response shift, by assessing change in the meaning of one’s self-evaluation. Six of these methods distinguish between change in observed measurements (observed change) and change in the construct that was intended to be measured (target change). The methods use either (sub)group-based or individual-level analysis, or a combination. All methods have underlying assumptions to be met and alternative explanations for the inferred response shift effects. We highlighted the need to address the interpretation of the results as response shift and proposed advancing new methods handling individual variation in change over time and multiple time points.

Conclusion

No single response shift method is optimal; each method has strengths and limitations. Additionally, extra steps need to be taken to correctly interpret the results. Advancing new methods and conducting computer simulation studies that compare methods are recommended to move response shift research forward.

Supplementary Information

The online version of this article (10.1007/s11136-020-02755-4) contains supplementary material, which is available to authorized users.

Using structural equation modeling to investigate change and response shift in patient-reported outcomes: practical considerations and recommendations

BACKGROUND

Patient-reported outcomes (PROs) are of increasing importance for health-care evaluations. However, the interpretation of change in PROs may be obfuscated due to changes in the meaning of the self-evaluation, i.e., response shift. Structural equation modeling (SEM) is the most widely used statistical approach for the investigation of response shift. Yet, non-technical descriptions of SEM for response shift investigation are lacking. Moreover, application of SEM is not straightforward and requires sequential decision-making practices that have not received much attention in the literature.

AIMS

To stimulate appropriate applications and interpretations of SEM for the investigation of response shift, the current paper aims to (1) provide an accessible description of the SEM operationalizations of change that are relevant for response shift investigation; (2) discuss practical considerations in applying SEM; and (3) provide guidelines and recommendations for researchers who want to use SEM for the investigation and interpretation of change and response shift in PROs.

CONCLUSION

Appropriate applications and interpretations of SEM for the detection of response shift will help to improve our understanding of response shift phenomena and thus change in PROs. Better understanding of patients' perceived health trajectories will ultimately help to adopt more effective treatments and thus enhance patients' wellbeing.

Reliability, discriminative accuracy, and an exploration of response shift as measured using the satisfaction and Recovery Index over 12 months from musculoskeletal trauma

PURPOSE

The Satisfaction and Recovery Index (SRI) is an importance-weighted health-related satisfaction tool intended to be a patient-centric means to capture both the process and state of recovery following musculoskeletal trauma. The purpose of this study was to explore measurement invariance, responsiveness, discriminative accuracy, and potential response shift identifiable within the SRI.

METHODS

Participants were 111 adults with acute musculoskeletal trauma. Data were collected at baseline, and again at 1, 2, 3, 6, and 12 months post-injury. Other tools used were the Brief Pain Inventory (BPI) and a Global Rating of Change scale. 1-month test-retest reliability (ICC_2,1), responsiveness (standardized response mean in stable vs. changed participants), discriminative accuracy (area under the curve for differentiating between recovered and non-recovered), and response shift (change in mean importance scores over the 12-month period) were explored. All but the final analysis were compared against the BPI.

RESULTS

Test-retest reliability was excellent across all metrics (ICC_2,1 = 0.83 to 0.88). Responsiveness was greatest for the weighted SRI (SRM = 0.36) with MDC₉₅ of 13.7%. All tools showed significant ability to discriminate between participants nominating recovery vs. non-recovery (AUC≥0.69) though the BPI subscales were significantly better than the SRI. Importance ratings showed small but significant change over time in 7 of the 9 SRI items.

CONCLUSION

This study provides support for the SRI as a useful tool for evaluating recovery, though it seems more valuable for capturing the process rather than state of recovery. While response shift was small, there is enough reason to endorse retention of the importance ratings.

Response shift in results of patient-reported outcome measures: a commentary to The Response Shift-in Sync Working Group initiative

PURPOSE

The Working Group undertook a critical, comprehensive synthesis of the response shift work to date. We aimed to (1) describe the rationale for this initiative; (2) outline how the Working Group operated; (3) summarize the papers that comprise this initiative; and (4) discuss the way forward.

METHODS

Four interdisciplinary teams, consisting of response shift experts, external experts, and new investigators, prepared papers on (1) definitions and theoretical underpinnings, (2) operationalizations and response shift methods, (3) implications for healthcare decision-making, and (4) on the published magnitudes of response shift effects. Draft documents were discussed during a two-day meeting. Papers were reviewed by all members.

RESULTS

Vanier and colleagues revised the formal definition and theory of response shift, and applied these in an amended, explanatory model of response shift. Sébille and colleagues conducted a critical examination of eleven response shift methods and concluded that for each method extra steps are required to make the response shift interpretation plausible. Sawatzky and colleagues created a framework for considering the impact of response shift on healthcare decision-making at the level of the individual patient (micro), the organization (meso), and policy (macro). Sajobi and colleagues are conducting a meta-analysis of published response shift effects. Preliminary findings indicate that the mean effect sizes are often small and variable across studies that measure different outcomes and use different methods.

CONCLUSION

Future response shift research will benefit from collaboration among diverse people, formulating alternative hypotheses of response shift, and conducting the most conclusive studies aimed at testing these (falsification).

Response shift in health-related quality of life measures in the presence of formative indicators

Background

Response shift (RS) has been defined as a change in the meaning of an individual’s self-evaluation that needs to be accounted for when assessing longitudinal changes in health-related quality of life (HRQoL). RS detection through structural equation modeling is accomplished by adopting Oort’s procedure based on a measurement model in which the observed variables are defined as reflective indicators of the HRQoL latent variable; that is, the latent variable causes the variation in the reflective indicators. This study aims to propose a procedure that assesses RS when formative indicators are used in measuring HRQoL; in this last case, the latent variable is considered to be a function of some formative indicators. A secondary aim is to compare the new procedure with Oort’s procedure to highlight similarities and differences.

Methods

The data were retrieved from a consecutive series of 258 patients newly diagnosed with colorectal cancer and undergoing chemotherapy and/or surgery. The European Organisation for Research and Treatment of Cancer Quality of Life Questionnaire (EORTC QOL-C30) was administered twice, once before and once six months after treatment. Structural equation modeling was used to evaluate RS and true change with the newly proposed method (in which fatigue and pain were defined as formative indicators) and with Oort’s procedure (in which fatigue and pain were defined as reflective indicators).

Results

According to the new procedure, there was no measurement bias, and on average, patients’ quality of life improved by 3.53 points (on a scale ranging from 0 to 100) at the 6-month follow-up. With Oort’s procedure, the loading of the pain indicator was not invariant across the two time points, suggesting the presence of reprioritization, whereas the estimation of true change was very similar to the previous one: 3.87.

Conclusions

RS and true change in HRQoL can be evaluated in the presence of formative indicators. Defining a measurement model by formative or reflective indicators can lead to different results.

Response-shift effects in neuromyelitis optica spectrum disorder: estimating response-shift-adjusted scores using equating

Background

In our companion paper, random intercept models (RIMs) investigated response-shift effects in a clinical trial comparing Eculizumab to Placebo for people with neuromyelitis optica spectrum disorder (NMOSD). RIMs predicted Global Health using the EQ-5D Visual Analogue Scale item (VAS) to encompass broad criteria that people might consider. The SF36™v2 mental and physical component scores (MCS and PCS) helped us detect response shift in VAS. Here, we sought to “back-translate” the VAS into the MCS/PCS scores that would have been observed if response shift had not been present.

Methods

This secondary analysis utilized NMOSD clinical trial data evaluating the impact of Eculizumab in preventing relapses (n = 143). Analyses began by equating raw scores from the VAS, MCS, and PCS, and computing scores that removed response-shift effects. Correlation analysis and descriptive displays provided a more comprehensive examination of response-shift effects.

Results

MCS and PCS crosswalks with VAS equated the scores that include and exclude response-shift effects. These two sets of scores had low shared variance for MCS for both groups, suggesting that corresponding mental health constructs were substantially different. The shared variance contrast for physical health was distinct only for the Placebo group. The larger MCS response-shift effects were found at end of study for Placebo only and were more prominent at extremes of the MCS score distribution.

Conclusions

Our results reveal notable treatment group differences in MCS but not PCS response shifts, which can explain null results detected in previous work. The method introduced herein provides a way to provide further information about response-shift effects in clinical trial data.

Response-shift effects in neuromyelitis optica spectrum disorder: a secondary analysis of clinical trial data

Background

Researchers have long posited that response-shift effects may obfuscate treatment effects. The present work investigated possible response-shift effects in a recent clinical trial testing a new treatment for Neuromyelitis Optica Spectrum Disorder (NMOSD). This pivotal trial provided impressive support for the drug Eculizumab in preventing relapse, but less strong or null results as the indicators became more subjective or evaluative. This pattern of results suggests that response-shift effects are present.

Methods

This secondary analysis utilized data from a randomized, double-blind trial evaluating the impact of Eculizumab in preventing relapses in 143 people with NMOSD. Treatment arm and then relapse status were hypothesized ‘catalysts’ of response shift in two series of analyses. We devised a “de-constructed” version of Oort structural-equation modeling using random-effects modeling for use in small samples. This method begins by testing an omnibus response-shift hypothesis and then, pending a positive result, implements a series of random-effects models to elucidate specific response-shift effects.

Results

In the omnibus test, the ‘standard quality-of-life (QOL) model’ captured substantially less well the experience of placebo as compared to Eculizumab group. Recalibration and reconceptualization response-shift effects were detected. Detected relapse-related response shifts included recalibration, reprioritization, and reconceptualization.

Conclusions

Trial patients experienced response shifts related to treatment- and relapse-related experiences. Published trial results likely under-estimated Eculizumab vs. Placebo differences due to recalibration and reconceptualization, and relapse effects due to recalibration, reprioritization, and reconceptualization. This novel random-effects- model application builds on response-shift theory and provides a small-sample method for better estimating treatment effects in clinical trials.

Electronic supplementary material

The online version of this article (10.1007/s11136-020-02707-y) contains supplementary material, which is available to authorized users.

Capturing patient experience: does quality-of-life appraisal entail a new class of measurement?

Background

Two decades of research on quality-of-life (QOL) appraisal have demonstrated links between patient experience and health outcomes and have accounted for both intra-individual change and inter-individual differences in a wide range of research contexts. The present work investigates patterns across diagnostic and demographic groupings to demonstrate how population-specific circumstances drive the structure of QOL appraisal.

Methods

This secondary analysis (N = 6448) utilized data from six patient groups: spine surgery, multiple sclerosis, heterogeneous chronically ill, heterogeneous cancer, bladder cancer, and human immunodeficiency virus (HIV). We explored patterns of inter-item correlation across patient samples, using items from the Standards of Comparison and Sampling of Experience subsections of the QOL Appraisal Profile v1 and v2. Similar matrices were compared by demographic characteristics.

Results

Patterns of inter-item correlations for Standards of Comparison items varied sharply across disease groups and racial groups while being similar across age, gender, and education levels. Inter-item correlation matrices for Sampling of Experience items revealed marked differences among disease groups and educational and racial categories but were similar across age and gender groups.

Conclusions

Appraisal parameters showed evidence of shared and unique aspects across samples and circumstances, findings which make sense in light of sample differences in health status and demographic influences. Tools to assess patient experience and meaning may be best understood as idiometric instruments. We discuss their distinctions from psychometric and clinimetric tools at theoretical, statistical, and applied levels.

Supplementary information

Supplementary information accompanies this paper at 10.1186/s41687-020-00254-1.

Associations between self-rated health and the assessments of anchoring vignettes in cardiovascular patients

Background/Objective

Assessments of health can be biased by response shift effects. One method for detecting such effects is the use of anchoring vignettes. The aim of this study was to analyze the relationship between participants' self-assessed health state and their assessments of these vignettes.

Method

A total of 342 cardiovascular patients assessed their own state of health on a 0-100 visual analogue scale. The patients additionally assessed two vignettes featuring fictional persons suffering from specific complaints. A sample of the general population (N = 1,236) served as controls.

Results

The participants rated the health state of the vignette character featuring physical problems as being significantly better than the general population did (effect size: d = 0.53). The group difference in the assessment of the vignette featuring primarily mental health problems was lower (d = -0.17). Participants' assessments of the vignettes were positively correlated with their assessments of their own health state (r = .26 and r = .10) and with several quality of life variables.

Conclusions

Anchoring vignettes are a useful tool for detecting response shift effects.

Consequences of ignoring the response-shift and measure non-invariant items in sleep studies: an empirical data based simulation of the treatment effect of CBT-I on dysfunctional sleep beliefs

INTRODUCTION

Previous studies have shown that psychological interventions do not only improve patients' mental symptoms (i.e., true change) but may also change the internal standards patients use to evaluate their symptoms (i.e., response shifts). Although the response shifts could reflect patients' cognitive changes toward their disorders as the interventions aim to achieve, failing to differentiate them from the true change during data analyses could bias the research conclusions. Considering this issue is seldom discussed in sleep studies, this study thus examined the impacts of response-shift items in an intervention study of cognitive behavioral therapy for insomnia (CBT-I) via empirical-data based simulations.

METHOD

We used longitudinal measurement invariance tests to identify the items in an abbreviated version of the Dysfunctional Beliefs and Attitudes about Sleep Scale that are non-invariant (response shifted) against CBT-I based on data from 114 insomnia patients. The partial invariance model built accordingly was then used as a population model for simulations to examine the impacts of the response-shift items on follow-up paired t-tests.

RESULTS

Invariance tests indicate CBT-I would lift the intercept of one item in DBAS-10 and cause non-uniform calibrations in three items. The following up simulations showed that failing to exclude the intercept-lifted item from the calculations of the subscale scores would lower the probability of using paired t-test to correctly detect the treatment effect by up to 53%.

CONCLUSIONS

We recommend sleep researchers to consider the issues of response-shift when assessing sleep-related constructs in interventional studies for insomnia.

Challenges and lessons learned from using anchoring vignettes to explore quality of life response behavior

Purpose

Asking patients to rate health-related quality of life (HRQoL) of hypothetical individuals described in anchoring vignettes has been proposed to enhance knowledge on how patients understand and respond to HRQoL questionnaires. In this article, we describe the development of anchoring vignettes and explore their utility for measuring response shift in patients’ self-reports of HRQoL.

Methods

We conducted an explorative mixed-methods study. One hundred patients with multiple sclerosis or psoriasis participated in two interviews at intervals of 3–6 months. During both interviews, patients assessed HRQoL of 16 hypothetical individuals on the SF-12 questionnaire (two vignettes for each of the eight domains of the SF-12). In addition to these quantitative ratings, we used the think-aloud method to explore changes in patients’ verbalization of their decision processes during vignette ratings.

Results

Agreement of vignette ratings at baseline and follow-up was low (ICCs < 0.55). In addition, paired sample t-tests revealed no significant directional mean changes in vignette ratings. Thus, ratings changed non-directionally, neither confirming retest reliability nor a systematic change of assessment. Furthermore, patients’ verbalization of their decision processes did not indicate whether or not the assessment strategy of individual patients had changed.

Conclusions

Patients’ ratings of anchoring vignettes fluctuate non-directionally over time. The think-aloud method appears not to be informative in exploring whether these fluctuations are due to changes in the individual decision process. Overall, vignettes might not be an appropriate approach to explore response shift, at least with regard to the specific target population and the use of the SF-12.

Electronic supplementary material

The online version of this article (10.1007/s11136-020-02488-4) contains supplementary material, which is available to authorized users.

Does response shift impact interpretation of change even among scales developed using item response theory?

Background

Response-shift effects impact the interpretation of change in quality-of-life (QOL) measures developed with classical test theory (CTT) methods. This study evaluated the impact of response shift on measures developed using Item Response Theory (IRT), as compared to CTT.

Methods

Chronically ill patients and caregivers (n = 1481) participated in a web-based survey at baseline and 17 months later. Patients completed the IRT-based PROMIS-10; NeuroQOL Applied Cognition, Positive Affect & Well-Being short-forms; and the CTT-based Ryff Environmental Mastery subscale. Response-shift effects were evaluated using regression residual modeling and the QOL Appraisal Profile-v2. The sample was divided into positive and negative catalyst groups on the basis of marital, work, job-status, and comorbidity change. Regression models predicted residualized QOL change scores as a function of catalysts and appraisal changes.

Results

In this sample 859 (58%) reported a catalyst. No catalyst was associated with change in scales developed using IRT, but positive work change was associated with the CTT-based measure. Catalyst variables were associated with changes in appraisal, which in turn were related to all outcomes, particularly for global mental health after a positive work-change.

Conclusions

Appraisal processes are relevant to interpreting IRT measures, particularly for global mental health in the face of life changes.

Adaptation and biomedical transition of people living with HIV to antiretroviral treatment in Burkina Faso

This article examines the adaptation and biomedical transition of people living with HIV (PLHIV) to antiretroviral treatment (ART) in Burkina Faso. The study employs a representative sample of 3625 randomly selected patients. Subjective and objective measures of health and their predictors are compared for short-term (≤24 months) versus longer-term ART adherents (>24 months) in a cohort study. Subjective health is measured as Likert scale self-rating and objective health as CD4 count. The results show that subjective and objective health capture different aspects of adaptation to ART: 90% of the short-term and 94% of the longer-term patients give a subjective health rating of at least good, while the objective measure of good health is higher among longer-term (42%) compared to short-term patients (27%). For subjective health, short-lived pleasures are predictors among short-term adherents while economic characteristics are determinants among longer-term patients. Meanwhile, objective health is associated with factors that determine ART adherence (i.e. participating in self-help groups). To achieve a wholistic resilience management that caters to the needs of PLHIV, we need to acknowledge adaptation to the disease and to ART and design health programs, in particular in developing countries, away from one-size-fits-all solutions to stage-of-disease specific support.

Patient characteristics and valuation changes impact quality of life and satisfaction in total knee arthroplasty - results from a German prospective cohort study

Background

Evaluation of variations in pre- and postoperative patient reported outcomes (PRO) and the association between preoperative patient characteristics and health and satisfaction outcomes after total knee arthroplasty (TKA) may support shared decision-making in Germany. Since previous research on TKA health outcomes indicated valuation differences in longitudinal data, experienced-based population weights were used for the first time as an external valuation system to measure discrepancies between patient and average population valuation of HRQoL.

Methods

Baseline data (n = 203) included sociodemographic and clinical characteristics and PROs, measured by the EQ-5D-3 L and WOMAC. Six-month follow-up data (n = 161) included medical changes since hospital discharge, PROs and satisfaction. A multivariate linear regression analysis was performed to evaluate the relationship between preoperative patient characteristics and PRO scores. Patient acceptable symptom state (PASS) was calculated to provide a satisfaction threshold. Patient-reported health-related quality of life (HRQoL) valuations were compared with average experienced-based population values to detect changes in valuation.

Results

One hundred thirty-seven subjects met inclusion criteria. All PRO measures improved significantly. Preoperative WOMAC and EQ-5D VAS, housing situation, marital status, age and asthma were found to be predictors of postoperative outcomes. 73% of study participants valued their preoperative HRQoL higher than the general population valuation, indicating response shift. Preoperatively, patient-reported EQ-5D VAS was substantially higher than average experienced-based population values. Postoperatively, this difference declined sharply.

Approximately 61% of the patients reported satisfactory postoperative health, being mainly satisfied with results if postoperative WOMAC was ≥82.49 (change ≥20.25) and postoperative EQ-5D VAS was ≥75 (change ≥6).

Conclusion

On average, patients benefited from TKA. Preoperative WOMAC and EQ-5D VAS were predictors of postoperative outcomes after TKA. Particularly patients with high absolute preoperative PRO scores were more likely to remain unsatisfied. Therefore, outcome prediction can contribute to shared-decision making. Using general population valuations as a reference, this study underlined a discrepancy between population and patient valuation of HRQoL before, but not after surgery, thus indicating a potential temporary response shift before surgery.

Advancing quality-of-life research by deepening our understanding of response shift: a unifying theory of appraisal

Studies of response-shift phenomena in quality-of-life (QOL) research have grown steadily in the more than two decades of research. As a field, we have been calling a lot of different approaches "response shift" over the years, but the only approach that fully embodies the foundational (Sprangers and Schwartz, Soc Sci Med 48(11):1507-1515, 1999) theoretical model is appraisal. According to the definition proposed in this model, response shift is about individual cognitive change. This paper presents the argument that all response-shift research models need to be grounded in an understanding of cognitive appraisal. We present a reasoned argument for why appraisal is fundamental to QOL response-shift research. We describe current measurement and analytic methods for working with appraisal, and how these methods can be integrated into the current response-shift statistical 'tool box.' We propose future research directions on theory, methods, and cross-calibration of group- and individual-level methods. There are currently three tools available in multiple languages for assessing QOL appraisal processes. They have been tested and used to assess response shift in empirical studies of ~ 7000 people with chronic illness. The study findings illustrate how appraisal theory and methods can facilitate methodological investigations of and to enhance other response-shift detection methods. Future research directions are proposed to enrich QOL theory, response-shift methods, and interpretation of QOL change over time. Appraisal theory and methods are the closest approximation to a response-shift 'gold standard.' They provide the foundation for understanding response shift and point to a unified theory of QOL.

Contextualizing the lived experience of quality of life for persons with spinal cord injury: A mixed-methods application of the response shift model

Objective: The objective of this study was to gain greater insight into individuals' quality of life (QOL) definitions, appraisals, and adaptations following spinal cord injury (SCI). Design: A mixed-methods design, applying the Schwartz and Sprangers response shift (RS) model. RS is a cognitive process wherein, in response to a change in health status, individuals change internal standards, values, or conceptualization of QOL Setting: Community-dwelling participants who receive medical treatment at a major Midwestern medical system and nearby Veterans' Affairs hospital. Participants: A purposive sample of participants with SCI (N = 40) completed semi-structured interviews and accompanying quantitative measures. Interventions: Not applicable. Outcome Measures: Qualitative data were analyzed using content analysis to identify themes. Analysis of variance were performed to detect differences based on themes and QOL, well-being, and demographic and injury characteristics. Results: Four RS themes were identified, capturing the range of participant perceptions of QOL. The themes ranged from complete RS, indicating active engagement in maintaining QOL, to awareness and comparisons redefining QOL, to a relative lack of RS. Average QOL ratings differed as a function of response shift themes. PROMIS Global Health, Anxiety, and Depression also differed as a function of RS themes. Conclusion: The RS model contextualizes differences in QOL definitions, appraisals, and adaptations in a way standardized QOL measures alone do not.

Response shift effects in patients' assessments of their quality of life after cardiac rehabilitation

PURPOSE

The effect of intervention programs on health-related quality of life (HRQoL) can be underestimated due to response shift effects. This study aims to compare HRQoL between cardiac patients taking part in a rehabilitation program and the general population and to investigate changes in HRQoL in terms of response shift with two approaches.

METHODS

A sample of 282 cardiac rehabilitation inpatients (response rate: 58.9%) responded to the self-report quality of life questionnaire EORTC QLQ-C30 at baseline (during rehabilitation) and three months later (actual and retrospective judgment). Their HRQoL was compared to that of the general population. Response shift evaluation complemented the thentest with the structural equation modeling approach.

RESULTS

Compared to the general population, patients showed impaired quality of life on all scales (Hedges' g between 0.31 and 1.57). The complementation of the thentest with the structural equation modeling approach revealed response shift effects in physical, emotional, cognitive, and social functioning. No effects were found in role functioning.

CONCLUSIONS

The combination of both the thentest and the structural equation modeling approaches proved to be essential for obtaining comprehensive statistical evidence that response shift can distort measurements of change. Our results suggest that studies that use the thentest to evaluate the effectiveness of interventions should complement their analyses with the structural equation modeling approach to avoid biased effects.

Reconsidering the minimally important difference: evidence of instability over time and across groups

BACKGROUND CONTEXT

Underlying cognitive factors have been found to influence patients' symptom experience. Current evidence suggests that concomitant changes in appraisal must be taken into account to accurately interpret change as measured by standard spine patient-reported outcomes (PROs).

PURPOSE

To investigate changes in patients' minimally important differences (MID) over recovery from spinal surgery; whether and how cognitive appraisal processes are implicated in the change trajectories.

STUDY DESIGN/SETTING

Longitudinal cohort study with up to 12 months follow-up.

PATIENT SAMPLE

Surgical patients (n = 167) with a diagnosis of disc herniation or spinal stenosis.

OUTCOME MEASURES

Standard spine patient-reported PROs were used (Rand-36, Oswestry Disability Index, Numerical Rating Scale for pain, PROMIS Pain Impact).

METHODS

This study was funded by the Feldberg Chair in Spinal Research, Sunnybrook Health Sciences Centre and the authors have no conflicts of interest. MID used an anchor technique and was computed by global assessment of change (GAC) grouping. Participants were binned into groups based on their GAC response patterns at all time points: Consistently better post-surgery, consistently worse post-surgery, and bouncers, whose GAC ratings fluctuate (ie, better-then-worse-then-better; or vice versa). Individuals' longitudinal quality of life (QOL) and appraisal slope scores were computed. QOL-appraisal slopes' correlations were computed by GAC group. Fisher's Z transformation tested the hypothesis that GAC groups differed in the QOL-appraisal relationship over time.

RESULTS

Moderate to large changes are recognized as clinically important in the early stages of recovery (ie, 6 weeks post-surgery), and over time smaller and smaller changes become important. The three pattern groups emphasized and deemphasized different standards of comparison over time, with the Better group emphasizing personal goals and the Worse and Bouncers deemphasizing doctors' input. These group differences translated to differential relationships between PRO change and appraisal changes over time.

CONCLUSIONS

The MID reflects increasingly subtle change over time in PROs. Appraisal may influence how patients experience the same (MID) change over time, with better outcomes associated with emphasizing long-term goals. PRO change seems to be driven by different standards of comparison. Potential avenues for clinical intervention are discussed.

Response shift in quality of life assessment among cancer patients: A study from Iran

Background: During the course of disease, particularly of chronic diseases, changes in internal standards cause certain changes in the estimation of quality of life (QOL). These changes indicate the phenomenon of ‘response shift’. The present study aimed at assessing response shift in different scales of QOL in Iranian cancer patients.

Methods: To assess response shift through the ‘then test’ approach, we asked 211 cancer patients to complete the EORTC QLQ-C30 questionnaire at pretest (at the beginning of the study), posttest (3 months later), and then test (administered immediately after the posttest). Paired t test and Cohen’s effect size were used for comparison.

Results: Response shift was significant in all 4 scales under study, i.e. fatigue, pain, emotional functioning, and general QOL (p<0.001). Fatigue, pain, and global QOL have deteriorated significantly with then test approach and emotional function was significantly improved.

Conclusion: We observed a response shift in Iranian cancer patients in our study. Thus, in light of the multifactorial nature of QOL and the effect of the response shift bias on different aspects of QOL changes, it is of utmost importance to keep this bias in mind when interpreting the results and managing cancer patients’ treatment regimens.

Impact of reconceptualization response shift on rating of quality of life over time among people with advanced cancer

BACKGROUND

People with cancer may experience change in what constitutes quality of life (QOL) over time as a result of the cancer progression (true change) or adaptation to the experience, considered as a response shift phenomenon. As individualized measures are ideally suited to explore response shift, this study aimed to estimate the extent to which reconceptualization response shift occurred over time in a cancer population and the impact of this response shift on estimates of change on QOL measures.

METHODS

Ninety-seven people with advanced cancer completed the study measures including the Patient-Generated Index (PGI) at diagnosis (T0) and 1 year later (T1). The response shift indicator was the change in the number of areas nominated (range - 4 to + 3). Multivariate linear regression was used to estimate the effect of changing areas on change in the PGI score, single indicators of global QOL, and the EQ-5D_index adjusted for age and sex.

RESULTS

Approximately 72% of people in this sample either added or dropped areas over time. People who dropped more than two areas had higher PGI scores at T1 than T0 while people who added areas showed low PGI score.

CONCLUSION

The results are consistent with the PGI framework as areas nominated tend to focus on negative aspects of QOL.

Scoping review of response shift methods: current reporting practices and recommendations

BACKGROUND

Response shift (RS) has been defined as a change in the meaning of an individual's self-evaluation of his/her health status and quality of life. Several statistical model- and design-based methods have been developed to test for RS in longitudinal data. We reviewed the uptake of these methods in patient-reported outcomes (PRO) literature.

METHODS

CINHAHL, EMBASE, Medline, ProQuest, PsycINFO, and Web of Science were searched to identify English-language articles about RS published until 2016. Data on year and country of publication, PRO measure adopted, RS detection method, type of RS detected, and testing of underlying model assumptions were extracted from the included articles.

RESULTS

Of the 1032 articles identified, 101 (9.8%) articles were included in the study. While 54.5 of the articles reported on the Then-test, 30.7% of the articles reported on Oort's or Schmitt's structural equation modeling (SEM) procedure. Newer RS detection methods, such as relative importance analysis and random forest regression, have been used less frequently. Less than 25% reported on testing the assumptions underlying the adopted RS detection method(s).

CONCLUSIONS

Despite rapid methodological advancements in RS research, this review highlights the need for further research about RS detection methods for complex longitudinal data and standardized reporting guidelines.

Development of a practical outcome measure to account for individual differences in quality-of-life appraisal: the Brief Appraisal Inventory

BACKGROUND

The present study evaluated the distributional and structural characteristics and explanatory power of the 23-item Brief Appraisal Inventory (BAI), a more practical appraisal measure for use in clinical research and practice.

METHODS

A heterogeneous, online cohort of chronic disease patients and caregivers completed the BAI, along with demographics, comorbidities, PROMIS-10, and the Brief NEO Personality Inventory. Principal components, bivariate, and linear and logistic regression analyses addressed BAI item distributions, structure, and construct validity.

RESULTS

The study sample (n = 592) had a mean age of 43.8 (SD = 18.5), and was 79% female. The BAI items exhibited good distributions, and principal component analysis yielded five composite scores: (1) Health Worries; (2) Interpersonal and Independence concerns; (3) Accomplishing Goals and Problem-Solving; (4) Calm, Peaceful, and Active; (5) Spiritual Growth and Altruism. The construct validity of appraisal factors is supported by their zero-order correlations with demographic, health, personality, and health-related QOL measures. Comparisons of appraisal-correlates among comorbidity-burden subgroups shed light on the mediating role that appraisal may play in adapting to chronic illness. Appraisal moderated the influence of comorbidities on emotional but not physical functioning. The performance of the BAI in explaining unique variance in physical and emotional functioning is comparable to results obtained with earlier measures.

CONCLUSIONS

The BAI provides a practical, short tool for evaluating appraisal in a wide range of assessment situations. Future research might utilize the BAI in longitudinal research aimed at detecting response-shift effects over time, and in clinical settings to improve patient-provider communication about concerns related to health, health care, or QOL.