The challenges inherent with anchor-based approaches to the interpretation of important change in clinical outcome assessments

A weighted predictive modeling method for estimating thresholds of meaningful within-individual change for patient-reported outcomes

PURPOSE

Calculating the threshold for meaningful within-individual change (MWIC) is essential for interpreting patient-reported outcomes (PRO). However, traditional methods of determining MWIC threshold yield varying estimates and lack a standardized approach. We aim to propose a novel method for more accurate MWIC threshold estimation.

METHODS

We developed a weighted predictive modeling method. The weighting involved using the rank difference between PRO score change and the anchor of each individual. A Monte Carlo simulation was conducted to compare the performance of the new method and that of existing state-of-the-art methods. Simulation parameters included distributions of PRO score changes, sample sizes, improvement proportions, and correlation strengths. Statistical performance was assessed using relative bias (rbias), coefficient of variation (CV), and relative root mean squared error (rRMSE).

RESULTS

Distribution-based methods had the largest rbias and rRMSE among all methods. Existing anchor-based methods except for the Terluin 2022 method were biased when the correlation strength was weak or when the improvement proportion was not 50%. The Terluin 2022 method requires estimating an important reliability parameter, and this method had highest CV compared to other predictive modeling methods. The new weighted method demonstrated the smallest rRMSE across most simulation settings. It also maintained relatively high accuracy under weak correlation strength or imbalanced improvement proportion. Similar results were presented under normal or skewed distributions of PRO score changes.

CONCLUSION

This novel method offers a simple and feasible alternative to existing predictive modeling methods for estimating MWIC threshold, which can facilitate the application of PRO.

Use of Minimal Important Difference for Patient-Reported Outcome Measures in Plastic Surgery: A Systematic Review

BACKGROUND

The minimal important difference (MID) is vital to consider when interpreting the clinical importance of observed changes from surgical interventions assessed by patient-reported outcome measures (PROMs). There is no accepted standard for how to calculate MIDs, and uptake in the plastic surgery literature is unknown, leading to methodologic and interpretation issues.

METHODS

Medline and Embase were searched to identify all plastic surgery randomized controlled trials (RCTs) using PROMs as outcomes and MID estimation studies for PROMs used by RCTs. Included studies were assessed for uptake and application of MIDs, and MID estimation methodology and values were categorized.

RESULTS

A total of 554 RCTs using PROMs as outcomes were identified. Of these, 419 RCTs had the possibility of incorporating a previously published MID. The uptake rate of MIDs was 11.5% ( n = 48 of 419). The most common ways MIDs were applied were to calculate sample size (37.5%) or to determine whether results were clinically important (35.4%). A total of 99 studies estimating MID values for the most common PROMs in plastic surgery, based on our review, were analyzed. The most common estimation methodologies were receiver operating characteristic curve analysis (49%), change difference (31%), and SD (25%).

CONCLUSIONS

This review highlights limited uptake and application of MIDs in plastic surgery. The authors propose 4 major barriers: (1) no repository of published MIDs for PROMs used in plastic surgery exists; (2) available MIDs are not specific to plastic surgery populations; (3) high heterogeneity in MID estimation methodology was present; and (4) there are wide ranges in MID values, with no superior choice identified.

(Re-)Defining "Successful Aging" as the Endpoint in Clinical Trials? Current Methods, Challenges, and Emerging Solutions

BACKGROUND AND OBJECTIVES

"Successful aging" as defined in Rowe & Kahn's model aims to include the major domains of aging under a single overarching concept. From here the question arises how "successful aging" itself can be implemented as an endpoint in clinical research in a way that it is compliant to methodological regulatory framework and recommendations as formulated, for example, by the United States Food and Drug Admnistration (FDA). This article discusses from an applied perspective approaches and examples of how "successful aging" as a multidomain concept can be put into measurement practice.

RESEARCH DESIGN AND METHODS

Narrative literature review.

RESULTS

Existing methods of merging insights from, for example, functional, social, and cognitive outcomes can fail to perform under conditions of extreme heterogeneity, as often present in samples of older adults. In research on rare diseases, the methodological approach of a Multidomain Responder Index (MDRI) has been proposed to handle heterogeneity. MDRIs may also provide a solution to combine aging outcomes from different domains into a single "successful aging" endpoint. However, strict measurement criteria will need to be fulfilled to find acceptance in decision making and validated meaningful change thresholds are a critical prerequisite.

DISCUSSION AND IMPLICATIONS

Gerontology can spearhead methodological approaches of handling multidimensional measurement under heterogeneity by validating a "successful aging" MDRI suitable for use as an endpoint in clinical research.

Interstitial Lung Disease Patients' Global Impressions of Symptoms, Severity Ratings, and Meaningfulness of Changes Over Time

Rationale: In interstitial lung disease (ILD), symptoms drive impairments in health-related quality of life. Patient-reported outcome measures (PROMs) can assess whether interventions change symptom severity. The meaningfulness of change in a PROM score is estimated by anchoring it to a related variable for which meaningful change has been previously established. Patient global impressions of severity (PGISs) are single-item PROMs that may make trustworthy anchors, but, for ILD, the meaningfulness of change in PGIS items for shortness of breath (SOB), cough, and fatigue/low energy are unknown. Objectives: To improve understanding of how patients with ILD rate and categorize symptoms, how differing levels of symptom severity affect lived experiences, and how patients derive and apply meaningfulness to change in symptoms. Methods: We used one-on-one interviews and an electronic survey to collect data from patients with various forms of ILD. Interviews were conducted to provide richness and context to survey responses. We conducted certain analyses with respondents stratified by supplemental oxygen use. Results: Interviewees (N = 18) confirmed SOB, cough, and fatigue/low energy as the most bothersome symptoms of ILD. Among 298 survey respondents, on a PGIS for SOB with a 0-4 numeric rating scale, on average, those who used supplemental O2 had more severe SOB than nonusers, and most respondents considered a 2-point change meaningful for worsening (45.5%) or improvement (47.2%). On a PGIS with a five-option ordinal response scale, for SOB, most considered a 1-category change meaningful for worsening (49.8%) and a 2-category change meaningful for improvement (42.3); for cough frequency, most respondents considered a 1-category change on the five-option ordinal response scale meaningful for worsening (48.2%) or improvement (45.0%). Survey responses for SOB at the present time versus 3 months earlier (patient global impressions of change) were biased toward the present state. Conclusions: PGISs can be used as anchors for meaningful change analyses of PROMs that assess SOB or cough in patients with ILD. Patient global impressions of change demonstrate present-state bias and should not be used. Patients' descriptions paint a vivid picture of lived experience with varying levels of symptom severity and can help contextualize change scores.

Dyspnea (breathlessness) in amyotrophic lateral sclerosis/motor neuron disease: prevalence, progression, severity, and correlates

OBJECTIVE

Dyspnea, or breathlessness, is an important symptom in amyotrophic lateral sclerosis/motor neuron disease (ALS/MND). We examined the measurement properties of the Dyspnea-12.

METHODS

Rasch analysis enabled conversion of raw Dyspnea-12 scores to interval level metric equivalents. Converted data were used to perform trajectory modeling; those following different trajectories were compared for demographic, clinical, symptom, and functioning characteristics. Logistic regression examined differences between distinct trajectories.

RESULTS

In 1022 people, at baseline, mean metric Dyspnea-12 was 7.6 (SD 9.3). 49.8% had dyspnea, severe in 12.6%. Trajectory analysis over 28 months revealed three breathlessness trajectories: group 1 reported none at baseline/follow-up (42.7%); group 2 significantly increased over time (9.4%); group 3 had a much higher level at baseline which rose over follow-up (47.9%). Group 3 had worse outcomes on all symptoms, functioning and quality of life; compared to group 1, their odds of: respiratory onset sixfold greater; King's stage ≥3 2.9 greater; increased odds of being bothered by choking, head drop, fasciculations, and muscle cramps; fatigue and anxiety also elevated (p < .01).

CONCLUSION

Dyspnea is a cardinal symptom in ALS/MND and can be quickly measured using the Dyspnea-12. Raw scores can easily be converted to interval level measurement, for valid change scores and trajectory modeling. Dyspnea trajectories reveal different patterns, showing that clinical services must provide monitoring which is customized to individual patient need. Almost half of this large population had worsening dyspnea, confirming the importance of respiratory monitoring and interventions being integrated into routine ALS care.

Responsiveness and Minimal Clinically Important Difference of the Canadian Occupational Performance Measure Among Patients With Frozen Shoulder

IMPORTANCE

Patient-reported outcome measures provide insights into intervention effects on patients. The Canadian Occupational Performance Measure (COPM) emphasizes identifying priorities in daily activity engagement and evaluating an individual's perception of changes over time.

OBJECTIVE

To assess the responsiveness of the COPM and the minimal clinically important difference (MCID) among patients with frozen shoulders.

DESIGN

Prospective, single-blind, randomized controlled trial.

SETTING

Two physical medicine and rehabilitation clinics.

PARTICIPANTS

Ninety-four patients with frozen shoulders enrolled in a previous study.

OUTCOMES AND MEASURES

Baseline and 3-mo evaluations of the COPM and other measures. Responsiveness was assessed using effect size (ES) and standardized response mean (SRM). The MCID values were determined through a distribution-based approach, which used the 0.5 standard deviation and ES methods, and an anchor-based approach, which used the receiver operating characteristic curve method.

RESULTS

The ES and SRM results indicated that the COPM had high responsiveness. The distribution-based MCID values for COPM Performance and COPM Satisfaction were 1.17 and 1.44, respectively. The anchor-based MCID values were 2.5 (area under the curve [AUC] = 0.78, 95% confidence interval [CI] [0.64-0.91]) and 2.1 (AUC = 0.76, 95% CI [0.60-0.91]), respectively.

CONCLUSIONS AND RELEVANCE

The findings suggest that the COPM is a responsive outcome measure for patients with frozen shoulder. The established MCID values for the COPM can be valuable for interpreting changes in patient performance and satisfaction, thus aiding clinical interventions and research planning. Plain-Language Summary: This is the first study to review the effectiveness of the Canadian Occupational Performance Measure (COPM) to determine the success of occupational therapy interventions for people with a frozen shoulder. The findings suggest that the COPM is an effective and valuable tool for clients with a frozen shoulder to understand their experiences and treatment priorities and to detect meaningful changes in their performance and satisfaction after an occupational therapy intervention.

Clinical meaningfulness and psychometric robustness of the MG Symptoms PRO scales in clinical trials in adults with myasthenia gravis

Objectives

The objective of this research was to generate psychometric evidence supporting the myasthenia gravis (MG) symptoms patient-reported outcome (PRO) scales as a fit-for-purpose measure of severity of core symptoms of MG and provide information allowing their meaningful interpretation using data from a phase 3 study in MG.

Methods

Data from the MycarinG study, a phase 3 study of rozanolixizumab in patients with generalized MG who experience moderate to severe symptoms (ClinicalTrials.gov Identifier: NCT03971422) were analyzed with both classical test theory (CTT) and Rasch measurement theory (RMT). Meaningful within-individual change and group-level meaningful change were estimated for three MG Symptoms PRO scales using anchor- and distribution-based methods. Anchor-based methods used patient global impression of severity (PGIS) and change (PGIC) in MG symptoms as anchors.

Results

Good measurement properties of the MG Symptoms PRO scales were shown in the sample of 200 participants: good to excellent reliability (test-retest and internal consistency reliability) and validity (associations between items and scores within the MG Symptoms PRO scales and between the MG Symptoms PRO scores and other clinical outcomes-MG ADL, QMG score, MGC score, and MGFA classes-were as expected); and the items showed good coverage of the continuum and fit to the Rasch model. Triangulation of the anchor- and distribution-based method results led to the definition of clinically meaningful within-patient improvement in scores for Muscle Weakness Fatigability (-16.67), Physical Fatigue (-20.00), and Bulbar Muscle Weakness (-20.00), with associated ranges. Benchmarks are also proposed for the interpretation of group-level results.

Conclusion

The strong psychometric performance of the MG Symptoms PRO scales and the information generated to guide its interpretation supports its use in clinical trials for demonstrating the clinical benefits of new treatments targeting core symptoms of MG (muscle weakness fatigability, physical fatigue, bulbar muscle weakness, respiratory muscle weakness, and ocular muscle weakness).

Anchor-based minimal important difference values are often sensitive to the distribution of the change score

PURPOSE

Anchor-based studies are today the most popular approach to determine a minimal important difference value for an outcome variable. However, a variety of construction methods for such values do exist. This constitutes a challenge to the field. In order to distinguish between more or less adequate construction methods, meaningful minimal requirements can be helpful. For example, minimal important difference values should not reflect the intervention(s) the patients are exposed to in the study used for construction, as they should later allow to compare interventions. This requires that they are not sensitive to the distribution of the change score observed. This study aims at investigating to which degree established construction methods fulfil this minimal requirement.

METHODS

Six constructions methods were considered, covering very popular and recently suggested methods. The sensitivity of MID values to the distribution of the change score was investigated in a simulation study for these six construction methods.

RESULTS

Five out of six construction methods turned out to yield MID values which are sensitive to the distribution of the change score to a degree that questions their usefulness. Insensitivity can be obtained by using construction methods based solely on an estimate of the conditional distribution of the anchor variable given the change score.

CONCLUSION

In future the computation of MID values should be based on construction methods avoiding sensitivity to the distribution of the change score.

Meaning of Family Reported Outcome Measure (FROM-16) severity score bands: a cross-sectional online study in the UK

OBJECTIVE

To assign clinical meanings to the Family Reported Outcome Measure (FROM-16) scores through the development of score bands using the anchor-based approach.

DESIGN AND SETTING

A cross-sectional online study recruited participants through UK-based patient support groups, research support platforms (HealthWise Wales, Autism Research Centre-Cambridge University database, Join Dementia Research) and through social service departments in Wales.

PARTICIPANTS

Family members/partners (aged ≥18 years) of patients with different health conditions.

INTERVENTION

Family members/partners of patients completed the FROM-16 questionnaire and a Global Question (GQ).

MAIN OUTCOME MEASURE

Various FROM-16 band sets were devised as a result of mapping of mean, median and mode of the GQ scores to FROM-16 total score, and receiver operating characteristic-area under the curve cut-off values. The band set with the best agreement with GQ based on weighted kappa was selected.

RESULTS

A total of 4413 family members/partners (male=1533, 34.7%; female=2858, 64.8%; Prefer not to say=16, 0.4%; other=6, 0.14%) of people with a health condition (male=1994, 45.2%; female=2400, 54.4%; Prefer not to say=12, 0.3%; other=7, 0.16%) completed the online survey: mean FROM-16 score=15.02 (range 0-32, SD=8.08), mean GQ score=2.32 (range 0-4, SD=1.08). The proposed FROM-16 score bandings are: 0-1=no effect on the quality of life of family member; 2-8=small effect on family member; 9-16=moderate effect on family member; 17-25=very large effect on family member; 26-32=extremely large effect on family member (weighted kappa=0.60).

CONCLUSION

The FROM-16 score descriptor bands provide new information to clinicians about interpreting scores and score changes, allowing better-informed treatment decisions for patients and their families. The score banding of FROM-16, along with a short administration time, demonstrates its potential to support holistic clinical practice.