The minimal perceived change: a formal model of the responder definition according to the patient's meaning of change for patient-reported outcome data analysis and interpretation

Anchor-based minimal important difference values are often sensitive to the distribution of the change score

PURPOSE

Anchor-based studies are today the most popular approach to determine a minimal important difference value for an outcome variable. However, a variety of construction methods for such values do exist. This constitutes a challenge to the field. In order to distinguish between more or less adequate construction methods, meaningful minimal requirements can be helpful. For example, minimal important difference values should not reflect the intervention(s) the patients are exposed to in the study used for construction, as they should later allow to compare interventions. This requires that they are not sensitive to the distribution of the change score observed. This study aims at investigating to which degree established construction methods fulfil this minimal requirement.

METHODS

Six constructions methods were considered, covering very popular and recently suggested methods. The sensitivity of MID values to the distribution of the change score was investigated in a simulation study for these six construction methods.

RESULTS

Five out of six construction methods turned out to yield MID values which are sensitive to the distribution of the change score to a degree that questions their usefulness. Insensitivity can be obtained by using construction methods based solely on an estimate of the conditional distribution of the anchor variable given the change score.

CONCLUSION

In future the computation of MID values should be based on construction methods avoiding sensitivity to the distribution of the change score.

A Review of Meaningful Change Thresholds for EORTC QLQ-C30 and FACT-G Within Oncology

OBJECTIVES

This literature review provides an overview of meaningful change thresholds for the European Organisation for Research and Treatment of Cancer Quality of Life Questionnaire Core 30 (QLQ-C30) and the Functional Assessment of Cancer Therapy - General (FACT-G) used across hematological cancers and solid tumors (melanoma, lung, bladder, and prostate).

METHODS

Embase, MEDLINE, and PubMed were searched to identify relevant oncology publications from 2016 to 2021. Label claims from the US Food and Drug Administration and the European Medicines Agency for 7 recently approved drugs (pembrolizumab, atezolizumab, glasdegib, gilteritinib, tisagenlecleucel, axicabtagene ciloleucel, and daratumumab plus hyaluronidase-fihj) were reviewed.

RESULTS

Publications providing guidance on meaningful change thresholds for the QLQ-C30 displayed a growing trend away from broad "legacy" thresholds of 10 points for all QLQ-C30 scales), toward deriving "contemporary" thresholds (eg, subscale specific, population specific). Contemporary publications generally provide guidance on selecting thresholds for specific scales that account for improved or worsening thresholds (eg, QLQ-C30 subscales). This trend was not clear for FACT-G, with less new guidance available. Most clinical trials used in regulatory label submissions have used thresholds of 10 points for the QLQ-C30 subscales and 3 to 7 points for the FACT-G total score. Despite the availability of more recent guidelines, contemporary meaningful change thresholds seem slow to emerge in the published literature and regulatory labels.

CONCLUSIONS

Trialists should consider using contemporary thresholds, rather than legacy thresholds, for QLQ-C30 endpoints. Thresholds derived for a similar patient-population should be used where available. Further work is required to provide these across a broader range of cancer sites.

Development and validation of a questionnaire to assess owner and canine quality-of-life and treatment satisfaction in canine allergic dermatitis

BACKGROUND

Animal and owner quality-of-life (QoL) is pivotal in treatment decisions. Accurate measurement of owner-reported QoL and treatment satisfaction (TS) supports disease burden and treatment benefit evaluation.

OBJECTIVES

Develop and evaluate an owner-completed canine dermatitis QoL and TS questionnaire (CDQoL-TSQ) in allergic dogs.

MATERIALS AND METHODS

The CDQoL-TSQ was drafted following review of existing measures and expert input. Content validity was assessed through interviews with owners of allergic dogs. Psychometric properties of the QoL domains (Canine QoL, Owner QoL) were evaluated. Score interpretation was derived.

RESULTS

Twenty dog owners were interviewed. Item wording was amended following the first 10 interviews. Data from 211 owners were used in the psychometric evaluation. The Canine QoL domain demonstrated strong internal consistency (α = 0.89), test-retest reliability (ICC2,1  = 0.844), moderate convergent validity (r = 0.41) and moderate-high known-groups validity (effect size 0.37-0.64). The Owner QoL domain demonstrated strong internal consistency (α = 0.73), high convergent validity (r = 0.63) and moderate-high known-groups validity (0.43-0.63). Test-retest reliability approached moderate strength (ICC2,1  = 0.490). Group-level interpretation analysis showed minimal important difference of 7.0-13.6 points for dogs and 13.0-13.6 for owners. For individual dogs a change of 6.3 or 12.5 points for dogs, and 12.5 or 18.8 for owners indicates a response.

CONCLUSIONS AND CLINICAL RELEVANCE

The CDQOL-TSQ is a two-part assessment to evaluate QoL and TS in canine allergic dermatitis. The QoL questionnaire demonstrated validity and reliability, and interpretation of scores was derived, making it suitable for use in research and practice. The TS module is suitable for clinical setting use to improve owner-veterinarian communication.

Estimating anchor-based minimal important change using longitudinal confirmatory factor analysis

PURPOSE

The minimal important change (MIC) is defined as the smallest within-individual change in a patient-reported outcome measure (PROM) that patients on average perceive as important. We describe a method to estimate this value based on longitudinal confirmatory factor analysis (LCFA). The method is evaluated and compared with a recently published method based on longitudinal item response theory (LIRT) in simulated and real data. We also examined the effect of sample size on bias and precision of the estimate.

METHODS

We simulated 108 samples with various characteristics in which the true MIC was simulated as the mean of individual MICs, and estimated MICs based on LCFA and LIRT. Additionally, both MICs were estimated in existing PROMIS Pain Behavior data from 909 patients. In another set of 3888 simulated samples with sample sizes of 125, 250, 500, and 1000, we estimated LCFA-based MICs.

RESULTS

The MIC was equally well recovered with the LCFA-method as using the LIRT-method, but the LCFA analyses were more than 50 times faster. In the Pain Behavior data (with higher scores indicating more pain behavior), an LCFA-based MIC for improvement was estimated to be 2.85 points (on a simple sum scale ranging 14-42), whereas the LIRT-based MIC was estimated to be 2.60. The sample size simulations showed that smaller sample sizes decreased the precision of the LCFA-based MIC and increased the risk of model non-convergence.

CONCLUSION

The MIC can accurately be estimated using LCFA, but sample sizes need to be preferably greater than 125.

Estimating meaningful thresholds for multi-item questionnaires using item response theory

PURPOSE

Meaningful thresholds are needed to interpret patient-reported outcome measure (PROM) results. This paper introduces a new method, based on item response theory (IRT), to estimate such thresholds. The performance of the method is examined in simulated datasets and two real datasets, and compared with other methods.

METHODS

The IRT method involves fitting an IRT model to the PROM items and an anchor item indicating the criterion state of interest. The difficulty parameter of the anchor item represents the meaningful threshold on the latent trait. The latent threshold is then linked to the corresponding expected PROM score. We simulated 4500 item response datasets to a 10-item PROM, and an anchor item. The datasets varied with respect to the mean and standard deviation of the latent trait, and the reliability of the anchor item. The real datasets consisted of a depression scale with a clinical depression diagnosis as anchor variable and a pain scale with a patient acceptable symptom state (PASS) question as anchor variable.

RESULTS

The new IRT method recovered the true thresholds accurately across the simulated datasets. The other methods, except one, produced biased threshold estimates if the state prevalence was smaller or greater than 0.5. The adjusted predictive modeling method matched the new IRT method (also in the real datasets) but showed some residual bias if the prevalence was smaller than 0.3 or greater than 0.7.

CONCLUSIONS

The new IRT method perfectly recovers meaningful (interpretational) thresholds for multi-item questionnaires, provided that the data satisfy the assumptions for IRT analysis.

Establishing thresholds for meaningful within-individual change using longitudinal item response theory

Purpose

Thresholds for meaningful within-individual change (MWIC) are useful for interpreting patient-reported outcome measures (PROM). Transition ratings (TR) have been recommended as anchors to establish MWIC. Traditional statistical methods for analyzing MWIC such as mean change analysis, receiver operating characteristic (ROC) analysis, and predictive modeling ignore problems of floor/ceiling effects and measurement error in the PROM scores and the TR item. We present a novel approach to MWIC estimation for multi-item scales using longitudinal item response theory (LIRT).

Methods

A Graded Response LIRT model for baseline and follow-up PROM data was expanded to include a TR item measuring latent change. The LIRT threshold parameter for the TR established the MWIC threshold on the latent metric, from which the observed PROM score MWIC threshold was estimated. We compared the LIRT approach and traditional methods using an example data set with baseline and three follow-up assessments differing by magnitude of score improvement, variance of score improvement, and baseline-follow-up score correlation.

Results

The LIRT model provided good fit to the data. LIRT estimates of observed PROM MWIC varied between 3 and 4 points score improvement. In contrast, results from traditional methods varied from 2 to 10 points—strongly associated with proportion of self-rated improvement. Best agreement between methods was seen when approximately 50% rated their health as improved.

Conclusion

Results from traditional analyses of anchor-based MWIC are impacted by study conditions. LIRT constitutes a promising and more robust analytic approach to identifying thresholds for MWIC.

Interpreting patient reports of perceived change during treatment for depression: Findings from the Veterans Outcome Assessment survey

This study addressed ongoing questions about the meaning of patients' perceptions of change during treatment. The study used data from the Veterans Outcome Assessment survey for patients with a depressive disorder, without mental health comorbidities, treated in Department of Veterans Affairs general mental health clinics (n = 694). Perceived changes in problems/symptoms, other domains, and the quality of communication with providers were evaluated with items from the Experience of Care & Health Outcomes (ECHO) survey. Depressive symptoms were measured with the Patient Health Questionnaire-9 (PHQ-9). Linear regression models evaluated associations of perceived change at 3-months post-baseline with observed change in PHQ-9 scores, scores on other patient-reported outcome measures (PROMs), and ratings of communication with providers. Patients' reports of their clinical condition at follow-up together with ratings of communication accounted for approximately one-third of the variance in patients' perceptions of change. Adding change-scores based on baseline and follow-up scores on the PHQ-9 and other PROMs did not improve model fit. The findings suggest that patient reports of perceived change during treatment reflect their current clinical state and their experience of care more closely than actual changes in the PHQ-9 or other PROMs.

Toward a rigorous assessment of the statistical performances of methods to estimate the Minimal Important Difference of Patient-Reported Outcomes: a protocol for a large-scale simulation study

Interpreting observed changes over time in Patient-Reported Outcomes (PRO) measures is still considered a challenge. Indeed, concluding an observed change at group level is statistically significant does not necessarily equate this change is meaningful from the perspective of the patient. To help interpret within and/or between group changes in the measure over time, the estimation of the Minimal Important Difference (MID) of the instrument - the smallest value that patients consider as a perceived change - is useful. In the last 30 years, a plethora of methods and estimators have been proposed to derive this MID value using clinical data from sample of patients. MIDs for hundreds of PROs have been estimated, with frequently a substantial variability in the results depending on the method used. Nonetheless, a rigorous assessment of the statistical performances of numerous proposed methods for estimating MIDs by experimental design such as Monte-Carlo study has never been performed. The purpose of this paper is to thoroughly depict a protocol for a large-scale simulation study designed to investigate the statistical performances, especially bias against a true populational value, of the common proposed estimators for MID. This paper depicts how investigated methods and estimators were retained after the conduct of a systematic review, the design of a conceptual model that formally defines what is the true populational MID value and the translation of the conceptual model into a model allowing the simulation of responses of items to a hypothetical PRO at two times of measurement along with the response to a Patient Global Rating of Change at the second time under the constraint of a known true MID value. A statistical analysis plan is depicted in order to conclude if working hypotheses on what could be appropriate MID estimators will be verified. Strengths, assumptions, and limits of the simulation model are exposed. Finally, we show how this protocol could be the basis for fostering future methodological research on the issue of interpreting changes in PRO measures.

Present state bias in transition ratings was accurately estimated in simulated and real data

OBJECTIVE

Patient-reported transition ratings are supposed to reflect the change between a previous baseline health state and a present follow-up state, but may reflect the present state to a greater extent. This so-called 'present state bias' (PSB) potentially threatens the validity of transition ratings. Several criteria have been proposed to assess PSB. We examined how well these criteria perform and to which extent confirmatory factor analysis (CFA) for categorical data provides an accurate assessment of the degree of PSB.

STUDY DESIGN AND SETTING

We simulated a multiple samples with baseline and follow-up item responses to a hypothetical questionnaire, and transition ratings. The samples varied with respect to various distributional characteristics and the degree of PSB. The performance of criteria proposed in the literature, and a new CFA-based criterion, were evaluated by the proportion of explained variance in PSB. In addition, four real datasets were analyzed.

RESULTS

The known criteria explained 36-74% of the variance in PSB. A new CFA-based criterion, namely the ratio of the factor loadings of the transition ratings plus one, explained 81-98% of the variance in PSB across the samples.

CONCLUSION

Present state bias in transition ratings can be estimated accurately using CFA.

Triangulation of multiple meaningful change thresholds for patient-reported outcome scores

PURPOSE

The notion of what constitutes meaningful differences or changes in patient-reported outcome scores is represented by meaningful change thresholds (MCTs). Applying multiple methods to estimate MCTs inevitably results in a range of estimates; however, a single estimate or small range is sought in practice to enable consistent interpretation of scores. While current recommendations for triangulation are appropriate in principle, the vital step of moving from all estimates to a value or small range lacks clarity and is subjective in nature. This article aims to review current triangulation approaches and provide more robust recommendations than what is currently available.

METHODS

Current approaches to perform triangulation are described and discussed. Anchor-based estimates are focussed upon due to their recognition as the most valid and developed approach. Recommendations for triangulation are provided.

RESULTS

A correlation-weighted average of MCT estimates is recommended to triangulate multiple MCT estimates derived from a single study into a single value, where increased weighting is given to stronger anchor measures. The choice of method to triangulate estimates from several published studies is highly dependent on the availability of information within the publications. MCTs designed for between-group differences, within-group changes, and within-individual changes should be considered separately.

CONCLUSION

The recommendations within this article provide a reliable and transparent approach to triangulation when a single value is sought, based on meta-analytic approaches. This approach is preferable to a simple mean of estimates where all are weighted equally, or through 'eyeballing' plotted estimates which is unreliable. We encourage researchers to adopt these methods, but to remain aware of the limitations within each method and further nuances in study design that result in heterogeneity. Sensitivity analyses with a range of plausible values are encouraged; however, the recommendations provide a suitable starting value for inferences. Unresolved issues in triangulation, requiring further exploration, are highlighted.

A confirmatory factor analysis approach was found to accurately estimate the reliability of transition ratings

INTRODUCTION

Transition ratings (TRs) are single item measures which ask patients to report on their health change. They allow for a simple assessment of improvement or deterioration and are frequently used as an "anchor" to determine interpretation thresholds on a patient-reported outcome measure (PROM). Despite their widespread use, a routinely applicable method to assess their reliability is lacking. This paper introduces a method to estimate the reliability of TRs based on confirmatory factor analysis (CFA) for categorical data.

METHOD

We modelled longitudinal PROM data as independent factors representing Time 1 and Time 2 in a CFA model. PROM items taken at Time 1 (T1) loaded on the first factor, although the same items taken at Time 2 (T2) loaded on the second. The TR item loaded onto both T1 and T2 factors. Three models with various constraints on the loadings and thresholds were examined. The communality (R²) statistic was used as a measure of the TR reliability. The approach was evaluated using simulated data and exemplified in four empirical datasets.

RESULTS

The simplest CFA model without constraints on the item loadings and thresholds performed equivalently to models with constraints on loadings and thresholds over time. Further constraints on the TR item loadings to be equal and opposite over time caused biased TR reliability estimates if the T1 and T2 loadings differed in magnitude. In the four empirical datasets, reliability of TRs ranged from 0.27 to 0.48. In three examples the TR had numerically stronger loading on T2 than on T1.

DISCUSSION AND CONCLUSIONS

Results support the use of the proposed method in understanding the reliability of TRs. Empirical study results reflect the typical range of reliability that has previously been reported for single items. Methodological considerations to improve TR reliability are presented, and developments of this method, are posited.