Using Item Response Theory to Identify Responders to Treatment: Examples with the Patient-Reported Outcomes Measurement Information System (PROMIS®) Physical Function Scale and Emotional Distress Composite

Comparing patient-reported outcomes measurement information system® (PROMIS®)-16 domain scores with the PROMIS-29 and 5-item PROMIS cognitive function scores

PURPOSE

This study evaluates the interpretability of Patient-Reported Outcomes Measurement Information System® (PROMIS®)-16 profile domain scores (physical function, ability to participate in social roles and activities, anxiety, depression, sleep disturbance, pain interference, cognitive function - abilities, and fatigue) compared to the PROMIS-29 scores and a 5-item PROMIS cognitive function score. The study aims to provide insights into using these measures in clinical and research settings.

METHODS

Analyses were conducted using data from 4130 adults from a nationally representative, probability-based internet panel between September and October 2022. A subset of 1256 individuals with back pain was followed up at six months. We compared the PROMIS-16 profile with the corresponding domain scores from the PROMIS-29 and a custom five-item cognitive function measure. We evaluated (1) reliability through inter-item correlations within each domain and (2) criterion validity by comparing PROMIS-16 profile with the corresponding longer PROMIS measures: (a) standardized mean differences in domain scores, (b) correlations, and (c) concordance of change (i.e., got worse, stayed the same, got better) among those with back pain from baseline to six months later using the reliable change index. We report the Kappa coefficient of agreement and the frequency and percentage of participants with concordant classifications.

RESULTS

Inter-item correlations for the PROMIS-16 domains ranged from 0.65 in cognitive function to 0.92 in pain interference. Standardized mean differences between PROMIS-16 and the scores for the corresponding longer PROMIS domains were minimal (< 0.2). Correlations among the corresponding domain scores ranged from 0.82 for sleep disturbance to 0.98 for pain interference. The percentage of concordance in change groups ranged from 63% for sleep disturbance to 88% for pain interference. Except for sleep disturbance, the change groups derived from the PROMIS-16 showed moderate to substantial agreement with scores estimated from the longer PROMIS measures (Kappa coefficients ≥ 0.41).

CONCLUSION

The PROMIS-16 domain scores perform similarly to the longer PROMIS measures and can be interpreted in the same way. This similarity indicates that PROMIS-16 can be useful for research as a brief health-related quality-of-life profile measure.

Utilizing a learning health system to capture real-world patient data: Application of the reliable change index to evaluate and improve the outcome of a pain rehabilitation program

BACKGROUND AND OBJECTIVES

The learning healthcare system (LHS) has been developed to integrate patients' clinical data into clinical decisions and improve treatment outcomes. Having little guidance on this integration process, we aim to explain (a) an applicable analytic tool for clinicians to evaluate the clinical outcomes at a group and an individual level and (b) our quality improvement (QI) project, analyzing the outcomes of a new outpatient pain rehabilitation program ("Back-in-Action": BIA) and applying the analysis results to modify our clinical practice.

METHODS

Through our LHS (CHOIR; https://choir.stanford.edu), we administered the Pain Catastrophizing Scale (PCS), Chronic Pain Acceptance Questionnaire (CPAQ), and Patient-Reported Outcomes Measures (PROMIS)® before and after BIA. After searching for appropriate analytic tools, we decided to use the Reliable Change Index (RCI) to determine if an observed change in the direction of better (improvement) or worse (deterioration) would be beyond or within the measurement error (no change).

RESULTS

Our RCI calculations revealed that at least a 9-point decrease in the PCS scores and 10-point increase in the CPAQ scores would indicate reliable improvement. RCIs for the PROMIS measures ranged from 5 to 8 T-score points (i.e., 0.5-0.8 SD). When evaluating change scores of the PCS, CPAQ, and PROMIS measures, we found that 94% of patients showed improvement in at least one domain after BIA and 6% showed no reliable improvement.

CONCLUSIONS

Our QI project revealed RCI as a useful tool to evaluate treatment outcomes at a group and an individual level, and RCI could be incorporated into the LHS to generate a progress report automatically for clinicians. We further explained how clinicians could use RCI results to modify a clinical practice, to improve the outcomes of a pain program, and to develop individualized care plans. Lastly, we suggested future research areas to improve the LHS application in pain practice.

Performance of the Physical Functioning Activities of Daily Living Scale in the 2020 Medicare Health Outcomes Survey

OBJECTIVE

Assessing functional limitations for adults at high risk of frailty yields valuable information for identifying those in need of therapy. We evaluate a self-report measure used to assess physical function among Medicare recipients in the United States.

DESIGN

Secondary analysis of the 2020 Medicare Health Outcomes Survey.

SETTING

A random sample of adult enrollees of 510 managed care plans.

PARTICIPANTS

287,476 adults (37% completion rate): 58% women; 16% were <65 years old (entitled via disability), 50% 65-74, and 34% 75 or older; 77% White, 14% Black, and 8% another race; 19% had

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURE

We evaluate item distributions, dimensionality, monotonicity of response options, reliability, and validity of the 8-item Physical Functioning Activities of Daily Living (PFADL) scale.

RESULTS

Most reported they could do 6 basic activities of daily living without difficulty. More limitations were reported for the other 2 PFADL items: 32% were not limited at all in climbing several flights of stairs and 40% in moderate activities. Product-moment correlations among the 8 items ranged from r=0.19 between the easiest-to-do (eating) and most difficult-to-do (climbing several flights of stairs) items to r=0.73 between bathing and dressing. The coefficient alpha and omega for the 8-item scale were both 0.86. Item slopes ranged from 2.6 (climbing several flights of stairs and eating) to 4.8 (dressing). Item characteristic curves revealed that response options were most likely to be selected in the appropriate order along the physical functioning continuum. The PFADL had at least 0.80 reliability between about -3 SDs below the mean to the mean. It was negatively correlated with comorbid condition count, disability days, problems with balance or walking, falling, and obesity.

CONCLUSIONS

The PFADL is useful for assessing average or below physical function in Medicare recipients.

Collapse

Key Words

• Aged
• Functional status
• Medicare
• Rehabilitation
• Self-assessment

Collapse

MESH Headings

• Adult
• Humans
• Female
• Aged
• United States
• Male
• Activities of Daily Living
• Reproducibility of Results
• Medicare
• Surveys and Questionnaires
• Outcome Assessment, Health Care
• Disability Evaluation

Collapse

Grants

• P30 AG021684 NIA NIH HHS

Collapse

Affiliation(s)

Ron D Hays UCLA Division of General Internal Medicine, Department of Medicine, Los Angeles, CA.
Marc N Elliott RAND Corporation, Santa Monica, CA

Collapse

Estimating individual health-related quality of life changes in low back pain patients

BACKGROUND

There is a need to evaluate different options for estimating individual change in health-related quality of life for patients with low back pain.

METHODS

Secondary analysis of data collected at baseline and 6 weeks later in a randomized trial of 749 adults with low back pain receiving usual medical care (UMC) or UMC plus chiropractic care at a small hospital at a military training site or two large military medical centers. The mean age was 31; 76% were male and 67% were White. The study participants completed the Patient-Reported Outcomes Measurement Information System (PROMIS®)-29 v 1.0 physical function, pain interference, pain intensity, fatigue, sleep disturbance, depression, anxiety, satisfaction with participation in social roles, physical summary, and mental health summary scores (T-scored with mean = 50 and standard deviation (SD) = 10 in the U.S. general population).

RESULTS

Reliability estimates at the baseline ranged from 0.700 to 0.969. Six-week test-retest intraclass correlation estimates were substantially lower than these estimates: the median test-retest intraclass correlation for the two-way mixed-effects model was 0. 532. Restricting the test-retest reliability estimates to the subset who reported they were about the same as at baseline on a retrospective rating of change item increased the median test-retest reliability to 0.686. The amount of individual change that was statistically significant varied by how reliability was estimated, and which SD was used. The smallest change needed was found when internal consistency reliability and the SD at baseline were used. When these values were used, the amount of change needed to be statistically significant (p < .05) at the individual level ranged from 3.33 (mental health summary scale) to 12.30 (pain intensity item) T-score points.

CONCLUSIONS

We recommend that in research studies estimates of the magnitude of individual change needed for statistical significance be provided for multiple reliability and standard deviation estimates. Whenever possible, patients should be classified based on whether they 1) improved significantly and perceived they got better, 2) improved significantly but did not perceive they were better, 3) did not improve significantly but felt they got better, or 4) did not improve significantly or report getting better.

Does Scoring Method Impact Estimation of Significant Individual Changes Assessed by Patient-Reported Outcome Measures? Comparing Classical Test Theory Versus Item Response Theory

OBJECTIVES

This study aimed to examine the ability of classical test theory (CTT) and item response theory (IRT) scores assessed by Patient-Reported Outcomes Measurement Information System® (PROMIS®) measures to identify significant individual changes in the setting of clinical studies, using both simulated and empirical data.

METHODS

We used simulated data to compare the estimation of significant individual changes between CTT and IRT scores across different conditions and a clinical trial data set to verify the simulation results. We calculated reliable change indexes to estimate significant individual changes.

RESULTS

For small true change, IRT scores showed a slightly higher rate of classifying change groups than CTT scores and were comparable with CTT scores for a shorter test length. Additionally, IRT scores were found to have a prominent advantage in the classification rates of change groups for medium to high true change over CTT scores. Such an advantage became prominent in a longer test length. The empirical data analysis results using an anchor-based approach further supported the above findings that IRT scores can more accurately classify participants into change groups than CTT scores.

CONCLUSIONS

Given that IRT scores perform better, or at least comparably, in most conditions, we recommend using IRT scores to estimate significant individual changes and identify responders to treatment. This study provides evidence-based guidance in detecting individual changes based on CTT and IRT scores under various measurement conditions and leads to recommendations for identifying responders to treatment for participants in clinical trials.

Identifying meaningful change on PROMIS short forms in cancer patients: a comparison of item response theory and classic test theory frameworks

BACKGROUND

This study compares classical test theory and item response theory frameworks to determine reliable change. Reliable change followed by anchoring to the change in categorically distinct responses on a criterion measure is a useful method to detect meaningful change on a target measure.

METHODS

Adult cancer patients were recruited from five cancer centers. Baseline and follow-up assessments at 6 weeks were administered. We investigated short forms derived from PROMIS® item banks on anxiety, depression, fatigue, pain intensity, pain interference, and sleep disturbance. We detected reliable change using reliable change index (RCI). We derived the T-scores corresponding to the RCI calculated under IRT and CTT frameworks using PROMIS® short forms. For changes that were reliable, meaningful change was identified using patient-reported change in PRO-CTCAE by at least one level. For both CTT and IRT approaches, we applied one-sided tests to detect reliable improvement or worsening using RCI. We compared the percentages of patients with reliable change and reliable/meaningful change.

RESULTS

The amount of change in T score corresponding to RCICTT of 1.65 ranged from 5.1 to 9.2 depending on domains. The amount of change corresponding to RCIIRT of 1.65 varied across the score range, and the minimum change ranged from 3.0 to 8.2 depending on domains. Across domains, the RCICTT and RCIIRT classified 80% to 98% of the patients consistently. When there was disagreement, the RCIIRT tended to identify more patients as having reliably changed compared to RCICTT if scores at both timepoints were in the range of 43 to 78 in anxiety, 45 to 70 in depression, 38 to 80 in fatigue, 35 to 78 in sleep disturbance, and 48 to 74 in pain interference, due to smaller standard errors in these ranges using the IRT method. The CTT method found more changes compared to IRT for the pain intensity domain that was shorter in length. Using RCICTT, 22% to 66% had reliable change in either direction depending on domains, and among these patients, 62% to 83% had meaningful change. Using RCIIRT, 37% to 68% had reliable change in either direction, and among these patients, 62% to 81% had meaningful change.

CONCLUSION

Applying the two-step criteria demonstrated in this study, we determined how much change is needed to declare reliable change at different levels of baseline scores. We offer reference values for percentage of patients who meaningfully change for investigators using the PROMIS instruments in oncology.

Validity and reliability of Patient-Reported Outcomes Measurement Information System (PROMIS®) using Computerized Adaptive Testing (CAT) in patients with advanced chronic kidney disease

BACKGROUND

The Patient-Reported Outcomes Measurement Information System (PROMIS®) has been recommended for computerized adaptive testing (CAT) of health-related quality of life (HRQOL). This study compared the content, validity and reliability of seven PROMIS CATs to the 12-item Short-Form Health Survey (SF-12) in patients with advanced chronic kidney disease (CKD).

METHODS

Adult CKD patients with an eGFR < 30 ml/min.1.73m2 not receiving dialysis treatment completed seven PROMIS CATs (assessing physical function, pain interference, fatigue, sleep disturbance, anxiety, depression, and ability to participate in social roles and activities), the SF-12 and, additionally, the PROMIS Pain Intensity single item and Dialysis Symptom Index (DSI) at inclusion and 2-weeks. A content comparison was performed between PROMIS CATs and SF-12. Construct validity of PROMIS CATs was assessed using Pearson's correlations. Test-retest reliability of all patient-reported outcome measures (PROMs) was assessed by calculating the intra-class correlation coefficient (ICC) and minimal detectable change (MDC).

RESULTS

In total, 207 patients participated in the study. A median of 45 items (10 minutes) was completed for PROMIS CATs. All PROMIS CATs showed evidence for sufficient construct validity. PROMIS CATs, most SF-12 domains and summary scores, and DSI showed sufficient test-retest reliability (ICC ≥ 0.70). PROMIS CATs had a lower MDC compared to the SF-12 (5.7-7.4 compared to 11.2-21.7 across domains, respectively).

CONCLUSION

PROMIS CATs showed sufficient construct validity and test-retest reliability in patients with advanced CKD. PROMIS CATs required more items but showed better reliability than the SF-12. Future research is needed to investigate the feasibility of PROMIS CATs for routine nephrology care.

Assessing the Significance of Individual Change in 2 Samples of Patients in Treatment for Low Back Pain Using 5 Different Statistical Indicators

OBJECTIVE

The purpose of this study was to estimate the significance of individual change using 5 statistical indicators in 2 samples of patients treated for low back pain.

METHODS

This secondary analysis used observational and clinical trial data from 2 samples of patients with low back pain to compare 5 ways of estimating significant individual change on the Impact Stratification Score (ISS) administered at the following 2 time points: 3 months apart in an observational study of 1680 patients undergoing chiropractic care, and 6 weeks apart in a randomized trial of 750 active-duty military personnel with low back pain. The following 5 methods were compared: (1) standard deviation index; (2) standard error of measurement (SEM); (3) standard error of estimate (SEE); (4) standard error of prediction (SEP); and (5) the reliable change index (RCI). The ISS is the sum of the Patient-Reported Outcomes Measurement Information System (PROMIS)-29 v2.1 physical function, pain interference, and pain intensity scores and is scored to have a possible range of 8 (least impact) to 50 (greatest impact).

RESULTS

The amount of change on the ISS needed for significant individual change in both samples was 5 for the SEM and for the SEE and 7 for the SEP and RCI.

CONCLUSIONS

The results of the current study provide some preliminary support for use of the SEP or the RCI to identify significant individual change and provide estimated thresholds of individual change that can be used for the ISS. The SEP and RCI estimates of significant change were consistent with retrospective ratings of change of at least moderately better in prior research. These 2 were less likely than other methods to classify people with low back pain as responders who have not actually gotten better (false positive). In contrast, the SEM and SEE were less likely to miss real change (false negative).

Reliable change and the reliable change index: still useful after all these years?

In 1984 Jacobson and colleagues introduced the concept of reliable change, viz the amount of change on a measure that an individual needed to show to determine that it exceeded the extent of change likely due to measurement error alone. Establishing reliable change was a pre-requisite for determining clinical significance. This paper summarizes the rationale for determining reliable change as providing an individual-focused, idiographic alternative to the dominant nomothetic approach to clinical outcome research based on group mean data and statistical significance. The conventional computational steps for calculating an individual’s standardized difference (reliable change) score and the minimum raw change score on the measure (a reliable change index) required to classify individuals as reliably positively changed, indeterminate, or reliably deteriorated are described. Two methods for graphically representing reliable change are presented, and a range of possible uses in both research and practice settings are summarized. A number of issues and debates concerning the calculation of reliable change are reviewed. It is concluded that the concept of reliable change remains useful for both cognitive behavioural researchers and practitioners, but that there are options regarding methods of computation. In any use of reliable change, the rationale for selecting among method options and the exact computations used need clear and careful description so that we can continuously judge the utility and appropriateness of the use of reliable change and enhance its value to the field.

Key learning aims

(1)

Guidelines for Designing and Evaluating Feasibility Pilot Studies

BACKGROUND

Pilot studies test the feasibility of methods and procedures to be used in larger-scale studies. Although numerous articles describe guidelines for the conduct of pilot studies, few have included specific feasibility indicators or strategies for evaluating multiple aspects of feasibility. In addition, using pilot studies to estimate effect sizes to plan sample sizes for subsequent randomized controlled trials has been challenged; however, there has been little consensus on alternative strategies.

METHODS

In Section 1, specific indicators (recruitment, retention, intervention fidelity, acceptability, adherence, and engagement) are presented for feasibility assessment of data collection methods and intervention implementation. Section 1 also highlights the importance of examining feasibility when adapting an intervention tested in mainstream populations to a new more diverse group. In Section 2, statistical and design issues are presented, including sample sizes for pilot studies, estimates of minimally important differences, design effects, confidence intervals (CI) and nonparametric statistics. An in-depth treatment of the limits of effect size estimation as well as process variables is presented. Tables showing CI around parameters are provided. With small samples, effect size, completion and adherence rate estimates will have large CI.

CONCLUSION

This commentary offers examples of indicators for evaluating feasibility, and of the limits of effect size estimation in pilot studies. As demonstrated, most pilot studies should not be used to estimate effect sizes, provide power calculations for statistical tests or perform exploratory analyses of efficacy. It is hoped that these guidelines will be useful to those planning pilot/feasibility studies before a larger-scale study.

Assessing the Accuracy of Errors of Measurement. Implications for Assessing Reliable Change in Clinical settings

Item response theory (IRT) models are non-linear latent variable models for discrete measures, whereas factor analysis (FA) is a latent variable model for continuous measures. In FA, the standard error (SE) of individuals' scores is common for all individuals. In IRT, the SE depends on the individual's score, and the SE function is to be provided. The empirical standard deviation of the scores across discrete ranges should also be computed to inform the extent to which IRT SEs overestimate or underestimate the variability of the scores. Within the target range of scores the test was designed to measure, one should expect IRT SEs to be smaller and more precise than FA SEs, and therefore preferable to assess clinical change. Outside the target range, IRT SEs may be too large and more imprecise than FA SEs, and FA more precise to assess change. As a result, whether FA or IRT characterize reliable change more accurately in a sample will depend on the proportion of individuals within or outside the IRT target score range. An application is provided to illustrate these concepts.

Between-group minimally important change versus individual treatment responders

PURPOSE

Estimates of the minimally important change (MIC) can be used to evaluate whether group-level differences are large enough to be important. But responders to treatment have been based upon group-level MIC thresholds, resulting in inaccurate classification of change over time. This article reviews options and provides suggestions about individual-level statistics to assess whether individuals have improved, stayed the same, or declined.

METHODS

Review of MIC estimation and an example of misapplication of MIC group-level estimates to assess individual change. Secondary data analysis to show how perceptions about meaningful change can be used along with significance of individual change.

RESULTS

MIC thresholds yield over-optimistic conclusions about responders to treatment because they classify those who have not changed as responders.

CONCLUSIONS

Future studies need to evaluate the significance of individual change using appropriate individual-level statistics such as the reliable change index or the equivalent coefficient of repeatability. Supplementing individual statistical significance with retrospective assessments of change is desirable.