Identifying reprioritization response shift in a stroke caregiver population: a comparison of missing data methods

Implications of the syntheses on definition, theory, and methods conducted by the Response Shift - in Sync Working Group

PURPOSE

Our aim is to advance response shift research by explicating the implications of published syntheses by the Response Shift - in Sync Working Group in an integrative way and suggesting ways for improving the quality of future response shift studies.

METHODS

Members of the Working Group further discussed the syntheses of the literature on definitions, theoretical underpinnings, operationalizations, and response shift methods. They outlined areas in need of further explication and refinement, and delineated additional implications for future research.

RESULTS

First, the proposed response shift definition was further specified and its implications for the interpretation of results explicated in relation to former, published definitions. Second, the proposed theoretical model was further explained in relation to previous theoretical models and its implications for formulating research objectives highlighted. Third, ways to explore alternative explanations per response shift method and their implications for response shift detection and explanation were delineated. The implications of the diversity of the response shift methods for response shift research were presented. Fourth, the implications of the need to enhance the quality and reporting of the response shift studies for future research were sketched.

CONCLUSION

With our work, we intend to contribute to a common language regarding response shift definitions, theory, and methods. By elucidating some of the major implications of earlier work, we hope to advance response shift research.

Detection of response shift in health-related quality of life studies: a systematic review

Background

Response Shift (RS) refers to the idea that an individual may undergo changes in its health-related quality of life (HRQOL). If internal standard, values, or reconceptualization of HRQOL change over time, then answer to the same items by the same individuals may not be comparable over time. Traditional measures to evaluate RS is prone to bias and strong methodologies to study the existence of this phenomenon is required. The objective is to systematically identify, analyze, and synthesize the existing and recent evidence of statistical methods used for RS detection in HRQOL studies.

Methods

The analysis of selected studies between January 2010 and July 2020 was performed through a systematic review in MEDLINE/PubMed, Scopus, Web of Science, PsycINFO and Google Scholar databases. The search strategy used the terms “Health-Related Quality of Life” and “Response Shift” using the filters “Humans”, “Journal Article”, “English” and “2010/01/01–2020/07/31”. The search was made in August 2020.

Results

After considering the inclusion and exclusion criteria, from the total selected articles (675), 107 (15.9%) of the publications were included in the analysis. From these, 79 (71.0%) detected the existence of RS and 86 (80.4%) only used one detection method. The most used methods were Then Test (n = 41) and Oort’s Structural Equation Models (SEM) (n = 35). Other method used were Multiple Lineal Regression (n = 7), Mixed-Effect Regression (n = 6), Latent Trajectory Analysis (n = 6), Item Response Theory (n = 6), Logistics Regression (n = 5), Regression and Classification Trees (n = 4) and Relative Importance Method (n = 4). Most of these detected recalibration, including Then Test (n = 27), followed by Oort’s SEM that detected the higher combination of RS types: recalibration (n = 24), reprioritization (n = 13) and reconceptualization (n = 7).

Conclusions

There is a continuous interest of studying RS detection. Oort’s SEM becomes the most versatile method in its capability for detecting RS in all different types. Despite results from previous systematic reviews, same methods have been used during the last years. We observed the need to explore other alternative methods allowing same detection capacity with robust and highly precise methodology. The investigation on RS detection and types requires more study, therefore new opportunity grows to continue attending this phenomenon through a multidisciplinary perspective.

If it's information, it's not "bias": a scoping review and proposed nomenclature for future response-shift research

BACKGROUND

The growth in response-shift methods has enabled a stronger empirical foundation to investigate response-shift phenomena in quality-of-life (QOL) research; but many of these methods utilize certain language in framing the research question(s) and interpreting results that treats response-shift effects as "bias," "noise," "nuisance," or otherwise warranting removal from the results rather than as information that matters. The present project will describe the various ways in which researchers have framed the questions for investigating response-shift issues and interpreted the findings, and will develop a nomenclature for such that highlights the important information about resilience reflected by response-shift findings.

METHODS

A scoping review was done of the QOL and response-shift literature (n = 1100 articles) from 1963 to 2020. After culling only empirical response-shift articles, raters characterized how investigators framed and interpreted study research questions (n = 164 articles).

RESULTS

Of 10 methods used, papers using four of them utilized terms like "bias" and aimed to remove response-shift effects to reveal "true change." Yet, the investigators' reflections on their own conclusions suggested that they do not truly believe that response shift is error to be removed. A structured nomenclature is proposed for discussing response-shift results in a range of research contexts and response-shift detection methods.

CONCLUSIONS

It is time for a concerted and focused effort to change the nomenclature of those methods that demonstrated this misinterpretation. Only by framing and interpreting response shift as information, not bias, can we improve our understanding and methods to help to distill outcomes with and without response-shift effects.

Response-shift effects in neuromyelitis optica spectrum disorder: a secondary analysis of clinical trial data

Background

Researchers have long posited that response-shift effects may obfuscate treatment effects. The present work investigated possible response-shift effects in a recent clinical trial testing a new treatment for Neuromyelitis Optica Spectrum Disorder (NMOSD). This pivotal trial provided impressive support for the drug Eculizumab in preventing relapse, but less strong or null results as the indicators became more subjective or evaluative. This pattern of results suggests that response-shift effects are present.

Methods

This secondary analysis utilized data from a randomized, double-blind trial evaluating the impact of Eculizumab in preventing relapses in 143 people with NMOSD. Treatment arm and then relapse status were hypothesized ‘catalysts’ of response shift in two series of analyses. We devised a “de-constructed” version of Oort structural-equation modeling using random-effects modeling for use in small samples. This method begins by testing an omnibus response-shift hypothesis and then, pending a positive result, implements a series of random-effects models to elucidate specific response-shift effects.

Results

In the omnibus test, the ‘standard quality-of-life (QOL) model’ captured substantially less well the experience of placebo as compared to Eculizumab group. Recalibration and reconceptualization response-shift effects were detected. Detected relapse-related response shifts included recalibration, reprioritization, and reconceptualization.

Conclusions

Trial patients experienced response shifts related to treatment- and relapse-related experiences. Published trial results likely under-estimated Eculizumab vs. Placebo differences due to recalibration and reconceptualization, and relapse effects due to recalibration, reprioritization, and reconceptualization. This novel random-effects- model application builds on response-shift theory and provides a small-sample method for better estimating treatment effects in clinical trials.

Electronic supplementary material

The online version of this article (10.1007/s11136-020-02707-y) contains supplementary material, which is available to authorized users.

A systematic review of the quality of reporting of simulation studies about methods for the analysis of complex longitudinal patient-reported outcomes data

PURPOSE

This study describes the characteristics and quality of reporting for published computer simulation studies about statistical methods to analyze complex longitudinal (i.e., repeated measures) patient-reported outcomes (PROs); we included methods for longitudinal latent variable measurement and growth models and response shift.

METHODS

Scopus, PsycINFO, PubMed, EMBASE, and Social Science Citation Index were searched for English-language studies published between 1999 and 2016 using selected keywords. Extracted information included characteristics of the study purpose/objectives, simulation design, software, execution, performance, and results. The quality of reporting was evaluated using published best-practice guidelines.

SYNTHESIS

A total of 1470 articles were reviewed and 42 articles met the inclusion criteria. The majority of the included studies (73.8%) investigated an existing statistical method, primarily a latent variable model (95.2%). Most studies specified the population model, including variable distributions, mean parameters, and correlation/covariances. The number of time points and sample size(s) were reported by all studies, but justification for the selected values was rarely provided. The majority of the studies (52.4%) did not report on model non-convergence. Bias, accuracy, and model fit were commonly reported performance metrics. All studies reported results descriptively, and 26.2% also used an inferential method.

CONCLUSIONS

While methodological research on statistical analyses of complex longitudinal PRO data is informed by computer simulation studies, current reporting practices of these studies have not been consistent with best-practice guidelines. Comprehensive reporting of simulation methods and results ensures that the strengths and limitations of the investigated statistical methods are thoroughly explored.

Scoping review of response shift methods: current reporting practices and recommendations

BACKGROUND

Response shift (RS) has been defined as a change in the meaning of an individual's self-evaluation of his/her health status and quality of life. Several statistical model- and design-based methods have been developed to test for RS in longitudinal data. We reviewed the uptake of these methods in patient-reported outcomes (PRO) literature.

METHODS

CINHAHL, EMBASE, Medline, ProQuest, PsycINFO, and Web of Science were searched to identify English-language articles about RS published until 2016. Data on year and country of publication, PRO measure adopted, RS detection method, type of RS detected, and testing of underlying model assumptions were extracted from the included articles.

RESULTS

Of the 1032 articles identified, 101 (9.8%) articles were included in the study. While 54.5 of the articles reported on the Then-test, 30.7% of the articles reported on Oort's or Schmitt's structural equation modeling (SEM) procedure. Newer RS detection methods, such as relative importance analysis and random forest regression, have been used less frequently. Less than 25% reported on testing the assumptions underlying the adopted RS detection method(s).

CONCLUSIONS

Despite rapid methodological advancements in RS research, this review highlights the need for further research about RS detection methods for complex longitudinal data and standardized reporting guidelines.

Method variation in the impact of missing data on response shift detection

PURPOSE

Missing data due to attrition or item non-response can result in biased estimates and loss of power in longitudinal quality-of-life (QOL) research. The impact of missing data on response shift (RS) detection is relatively unknown. This overview article synthesizes the findings of three methods tested in this special section regarding the impact of missing data patterns on RS detection in incomplete longitudinal data.

METHODS

The RS detection methods investigated include: (1) Relative importance analysis to detect reprioritization RS in stroke caregivers; (2) Oort's structural equation modeling (SEM) to detect recalibration, reprioritization, and reconceptualization RS in cancer patients; and (3) Rasch-based item-response theory-based (IRT) models as compared to SEM models to detect recalibration and reprioritization RS in hospitalized chronic disease patients. Each method dealt with missing data differently, either with imputation (1), attrition-based multi-group analysis (2), or probabilistic analysis that is robust to missingness due to the specific objectivity property (3).

RESULTS

Relative importance analyses were sensitive to the type and amount of missing data and imputation method, with multiple imputation showing the largest RS effects. The attrition-based multi-group SEM revealed differential effects of both the changes in health-related QOL and the occurrence of response shift by attrition stratum, and enabled a more complete interpretation of findings. The IRT RS algorithm found evidence of small recalibration and reprioritization effects in General Health, whereas SEM mostly evidenced small recalibration effects. These differences may be due to differences between the two methods in handling of missing data.

CONCLUSIONS

Missing data imputation techniques result in different conclusions about the presence of reprioritization RS using the relative importance method, while the attrition-based SEM approach highlighted different recalibration and reprioritization RS effects by attrition group. The IRT analyses detected more recalibration and reprioritization RS effects than SEM, presumably due to IRT's robustness to missing data. Future research should apply simulation techniques in order to make conclusive statements about the impacts of missing data according to the type and amount of RS.