1
|
Zeng C, Hays RD, Rodriguez A, Hanmer J, Herman PM, Edelen MO. Comparing patient-reported outcomes measurement information system ® (PROMIS ®)-16 domain scores with the PROMIS-29 and 5-item PROMIS cognitive function scores. Qual Life Res 2024:10.1007/s11136-024-03747-4. [PMID: 39143447 DOI: 10.1007/s11136-024-03747-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 07/23/2024] [Indexed: 08/16/2024]
Abstract
PURPOSE This study evaluates the interpretability of Patient-Reported Outcomes Measurement Information System® (PROMIS®)-16 profile domain scores (physical function, ability to participate in social roles and activities, anxiety, depression, sleep disturbance, pain interference, cognitive function - abilities, and fatigue) compared to the PROMIS-29 scores and a 5-item PROMIS cognitive function score. The study aims to provide insights into using these measures in clinical and research settings. METHODS Analyses were conducted using data from 4130 adults from a nationally representative, probability-based internet panel between September and October 2022. A subset of 1256 individuals with back pain was followed up at six months. We compared the PROMIS-16 profile with the corresponding domain scores from the PROMIS-29 and a custom five-item cognitive function measure. We evaluated (1) reliability through inter-item correlations within each domain and (2) criterion validity by comparing PROMIS-16 profile with the corresponding longer PROMIS measures: (a) standardized mean differences in domain scores, (b) correlations, and (c) concordance of change (i.e., got worse, stayed the same, got better) among those with back pain from baseline to six months later using the reliable change index. We report the Kappa coefficient of agreement and the frequency and percentage of participants with concordant classifications. RESULTS Inter-item correlations for the PROMIS-16 domains ranged from 0.65 in cognitive function to 0.92 in pain interference. Standardized mean differences between PROMIS-16 and the scores for the corresponding longer PROMIS domains were minimal (< 0.2). Correlations among the corresponding domain scores ranged from 0.82 for sleep disturbance to 0.98 for pain interference. The percentage of concordance in change groups ranged from 63% for sleep disturbance to 88% for pain interference. Except for sleep disturbance, the change groups derived from the PROMIS-16 showed moderate to substantial agreement with scores estimated from the longer PROMIS measures (Kappa coefficients ≥ 0.41). CONCLUSION The PROMIS-16 domain scores perform similarly to the longer PROMIS measures and can be interpreted in the same way. This similarity indicates that PROMIS-16 can be useful for research as a brief health-related quality-of-life profile measure.
Collapse
Affiliation(s)
- Chengbo Zeng
- Patient Reported Outcomes, Value and Experience (PROVE) Center, Department of Surgery, Brigham and Women's Hospital, Boston, MA, USA
| | - Ron D Hays
- Division of General Internal Medicine and Health Services Research, UCLA Department of Medicine, Los Angeles, CA, USA
| | - Anthony Rodriguez
- Behavioral and Policy Sciences, RAND Corporation, 20 Park Plaza #920, Boston, MA, USA
| | - Janel Hanmer
- Division of General Internal Medicine, University of Pittsburgh, Pittsburgh, PA, USA
| | - Patricia M Herman
- Behavioral and Policy Sciences, RAND Corporation, 1776 Main Street, Santa Monica, CA, USA
| | - Maria Orlando Edelen
- Patient Reported Outcomes, Value and Experience (PROVE) Center, Department of Surgery, Brigham and Women's Hospital, Boston, MA, USA.
- Behavioral and Policy Sciences, RAND Corporation, 20 Park Plaza #920, Boston, MA, USA.
| |
Collapse
|
2
|
You DS, Chong JL, Mackey SC, Poupore-King H. Utilizing a learning health system to capture real-world patient data: Application of the reliable change index to evaluate and improve the outcome of a pain rehabilitation program. Pain Pract 2024; 24:856-865. [PMID: 38465804 DOI: 10.1111/papr.13364] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/12/2024]
Abstract
BACKGROUND AND OBJECTIVES The learning healthcare system (LHS) has been developed to integrate patients' clinical data into clinical decisions and improve treatment outcomes. Having little guidance on this integration process, we aim to explain (a) an applicable analytic tool for clinicians to evaluate the clinical outcomes at a group and an individual level and (b) our quality improvement (QI) project, analyzing the outcomes of a new outpatient pain rehabilitation program ("Back-in-Action": BIA) and applying the analysis results to modify our clinical practice. METHODS Through our LHS (CHOIR; https://choir.stanford.edu), we administered the Pain Catastrophizing Scale (PCS), Chronic Pain Acceptance Questionnaire (CPAQ), and Patient-Reported Outcomes Measures (PROMIS)® before and after BIA. After searching for appropriate analytic tools, we decided to use the Reliable Change Index (RCI) to determine if an observed change in the direction of better (improvement) or worse (deterioration) would be beyond or within the measurement error (no change). RESULTS Our RCI calculations revealed that at least a 9-point decrease in the PCS scores and 10-point increase in the CPAQ scores would indicate reliable improvement. RCIs for the PROMIS measures ranged from 5 to 8 T-score points (i.e., 0.5-0.8 SD). When evaluating change scores of the PCS, CPAQ, and PROMIS measures, we found that 94% of patients showed improvement in at least one domain after BIA and 6% showed no reliable improvement. CONCLUSIONS Our QI project revealed RCI as a useful tool to evaluate treatment outcomes at a group and an individual level, and RCI could be incorporated into the LHS to generate a progress report automatically for clinicians. We further explained how clinicians could use RCI results to modify a clinical practice, to improve the outcomes of a pain program, and to develop individualized care plans. Lastly, we suggested future research areas to improve the LHS application in pain practice.
Collapse
Affiliation(s)
- Dokyoung S You
- Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University School of Medicine, Palo Alto, California, USA
| | - Jeanette L Chong
- Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University School of Medicine, Palo Alto, California, USA
| | - Sean C Mackey
- Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University School of Medicine, Palo Alto, California, USA
| | - Heather Poupore-King
- Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University School of Medicine, Palo Alto, California, USA
| |
Collapse
|
3
|
Hays RD, Elliott MN. Performance of the Physical Functioning Activities of Daily Living Scale in the 2020 Medicare Health Outcomes Survey. Arch Phys Med Rehabil 2024; 105:696-703. [PMID: 37995776 PMCID: PMC11097997 DOI: 10.1016/j.apmr.2023.11.005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/12/2023] [Revised: 09/06/2023] [Accepted: 11/09/2023] [Indexed: 11/25/2023]
Abstract
OBJECTIVE Assessing functional limitations for adults at high risk of frailty yields valuable information for identifying those in need of therapy. We evaluate a self-report measure used to assess physical function among Medicare recipients in the United States. DESIGN Secondary analysis of the 2020 Medicare Health Outcomes Survey. SETTING A random sample of adult enrollees of 510 managed care plans. PARTICIPANTS 287,476 adults (37% completion rate): 58% women; 16% were <65 years old (entitled via disability), 50% 65-74, and 34% 75 or older; 77% White, 14% Black, and 8% another race; 19% had INTERVENTIONS Not applicable. MAIN OUTCOME MEASURE We evaluate item distributions, dimensionality, monotonicity of response options, reliability, and validity of the 8-item Physical Functioning Activities of Daily Living (PFADL) scale. RESULTS Most reported they could do 6 basic activities of daily living without difficulty. More limitations were reported for the other 2 PFADL items: 32% were not limited at all in climbing several flights of stairs and 40% in moderate activities. Product-moment correlations among the 8 items ranged from r=0.19 between the easiest-to-do (eating) and most difficult-to-do (climbing several flights of stairs) items to r=0.73 between bathing and dressing. The coefficient alpha and omega for the 8-item scale were both 0.86. Item slopes ranged from 2.6 (climbing several flights of stairs and eating) to 4.8 (dressing). Item characteristic curves revealed that response options were most likely to be selected in the appropriate order along the physical functioning continuum. The PFADL had at least 0.80 reliability between about -3 SDs below the mean to the mean. It was negatively correlated with comorbid condition count, disability days, problems with balance or walking, falling, and obesity. CONCLUSIONS The PFADL is useful for assessing average or below physical function in Medicare recipients.
Collapse
Affiliation(s)
- Ron D Hays
- UCLA Division of General Internal Medicine, Department of Medicine, Los Angeles, CA.
| | | |
Collapse
|
4
|
Hays RD, Reise SP, Herman PM. Estimating individual health-related quality of life changes in low back pain patients. BMC Musculoskelet Disord 2023; 24:961. [PMID: 38082389 PMCID: PMC10712133 DOI: 10.1186/s12891-023-07093-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/27/2023] [Accepted: 12/04/2023] [Indexed: 12/18/2023] Open
Abstract
BACKGROUND There is a need to evaluate different options for estimating individual change in health-related quality of life for patients with low back pain. METHODS Secondary analysis of data collected at baseline and 6 weeks later in a randomized trial of 749 adults with low back pain receiving usual medical care (UMC) or UMC plus chiropractic care at a small hospital at a military training site or two large military medical centers. The mean age was 31; 76% were male and 67% were White. The study participants completed the Patient-Reported Outcomes Measurement Information System (PROMIS®)-29 v 1.0 physical function, pain interference, pain intensity, fatigue, sleep disturbance, depression, anxiety, satisfaction with participation in social roles, physical summary, and mental health summary scores (T-scored with mean = 50 and standard deviation (SD) = 10 in the U.S. general population). RESULTS Reliability estimates at the baseline ranged from 0.700 to 0.969. Six-week test-retest intraclass correlation estimates were substantially lower than these estimates: the median test-retest intraclass correlation for the two-way mixed-effects model was 0. 532. Restricting the test-retest reliability estimates to the subset who reported they were about the same as at baseline on a retrospective rating of change item increased the median test-retest reliability to 0.686. The amount of individual change that was statistically significant varied by how reliability was estimated, and which SD was used. The smallest change needed was found when internal consistency reliability and the SD at baseline were used. When these values were used, the amount of change needed to be statistically significant (p < .05) at the individual level ranged from 3.33 (mental health summary scale) to 12.30 (pain intensity item) T-score points. CONCLUSIONS We recommend that in research studies estimates of the magnitude of individual change needed for statistical significance be provided for multiple reliability and standard deviation estimates. Whenever possible, patients should be classified based on whether they 1) improved significantly and perceived they got better, 2) improved significantly but did not perceive they were better, 3) did not improve significantly but felt they got better, or 4) did not improve significantly or report getting better.
Collapse
Affiliation(s)
- Ron D Hays
- Division of General Internal Medicine & Health Services Research, UCLA Department of Medicine, 1100 Glendon Avenue, Los Angeles, CA, 90024, USA.
| | | | | |
Collapse
|
5
|
Tang X, Schalet BD, Peipert JD, Cella D. Does Scoring Method Impact Estimation of Significant Individual Changes Assessed by Patient-Reported Outcome Measures? Comparing Classical Test Theory Versus Item Response Theory. VALUE IN HEALTH : THE JOURNAL OF THE INTERNATIONAL SOCIETY FOR PHARMACOECONOMICS AND OUTCOMES RESEARCH 2023; 26:1518-1524. [PMID: 37315768 DOI: 10.1016/j.jval.2023.06.002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/20/2022] [Revised: 05/25/2023] [Accepted: 06/01/2023] [Indexed: 06/16/2023]
Abstract
OBJECTIVES This study aimed to examine the ability of classical test theory (CTT) and item response theory (IRT) scores assessed by Patient-Reported Outcomes Measurement Information System® (PROMIS®) measures to identify significant individual changes in the setting of clinical studies, using both simulated and empirical data. METHODS We used simulated data to compare the estimation of significant individual changes between CTT and IRT scores across different conditions and a clinical trial data set to verify the simulation results. We calculated reliable change indexes to estimate significant individual changes. RESULTS For small true change, IRT scores showed a slightly higher rate of classifying change groups than CTT scores and were comparable with CTT scores for a shorter test length. Additionally, IRT scores were found to have a prominent advantage in the classification rates of change groups for medium to high true change over CTT scores. Such an advantage became prominent in a longer test length. The empirical data analysis results using an anchor-based approach further supported the above findings that IRT scores can more accurately classify participants into change groups than CTT scores. CONCLUSIONS Given that IRT scores perform better, or at least comparably, in most conditions, we recommend using IRT scores to estimate significant individual changes and identify responders to treatment. This study provides evidence-based guidance in detecting individual changes based on CTT and IRT scores under various measurement conditions and leads to recommendations for identifying responders to treatment for participants in clinical trials.
Collapse
Affiliation(s)
- Xiaodan Tang
- Department of Medical Social Sciences, Northwestern University Feinberg School of Medicine, Chicago, IL, USA.
| | - Benjamin David Schalet
- Department of Epidemiology and Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - John Devin Peipert
- Department of Medical Social Sciences, Northwestern University Feinberg School of Medicine, Chicago, IL, USA
| | - David Cella
- Department of Medical Social Sciences, Northwestern University Feinberg School of Medicine, Chicago, IL, USA
| |
Collapse
|
6
|
Lee MK, Peipert JD, Cella D, Yost KJ, Eton DT, Novotny PJ, Sloan JA, Dueck AC. Identifying meaningful change on PROMIS short forms in cancer patients: a comparison of item response theory and classic test theory frameworks. Qual Life Res 2023; 32:1355-1367. [PMID: 36152109 PMCID: PMC10123030 DOI: 10.1007/s11136-022-03255-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 09/07/2022] [Indexed: 11/29/2022]
Abstract
BACKGROUND This study compares classical test theory and item response theory frameworks to determine reliable change. Reliable change followed by anchoring to the change in categorically distinct responses on a criterion measure is a useful method to detect meaningful change on a target measure. METHODS Adult cancer patients were recruited from five cancer centers. Baseline and follow-up assessments at 6 weeks were administered. We investigated short forms derived from PROMIS® item banks on anxiety, depression, fatigue, pain intensity, pain interference, and sleep disturbance. We detected reliable change using reliable change index (RCI). We derived the T-scores corresponding to the RCI calculated under IRT and CTT frameworks using PROMIS® short forms. For changes that were reliable, meaningful change was identified using patient-reported change in PRO-CTCAE by at least one level. For both CTT and IRT approaches, we applied one-sided tests to detect reliable improvement or worsening using RCI. We compared the percentages of patients with reliable change and reliable/meaningful change. RESULTS The amount of change in T score corresponding to RCICTT of 1.65 ranged from 5.1 to 9.2 depending on domains. The amount of change corresponding to RCIIRT of 1.65 varied across the score range, and the minimum change ranged from 3.0 to 8.2 depending on domains. Across domains, the RCICTT and RCIIRT classified 80% to 98% of the patients consistently. When there was disagreement, the RCIIRT tended to identify more patients as having reliably changed compared to RCICTT if scores at both timepoints were in the range of 43 to 78 in anxiety, 45 to 70 in depression, 38 to 80 in fatigue, 35 to 78 in sleep disturbance, and 48 to 74 in pain interference, due to smaller standard errors in these ranges using the IRT method. The CTT method found more changes compared to IRT for the pain intensity domain that was shorter in length. Using RCICTT, 22% to 66% had reliable change in either direction depending on domains, and among these patients, 62% to 83% had meaningful change. Using RCIIRT, 37% to 68% had reliable change in either direction, and among these patients, 62% to 81% had meaningful change. CONCLUSION Applying the two-step criteria demonstrated in this study, we determined how much change is needed to declare reliable change at different levels of baseline scores. We offer reference values for percentage of patients who meaningfully change for investigators using the PROMIS instruments in oncology.
Collapse
Affiliation(s)
- Minji K Lee
- Department of Quantitative Health Sciences, Mayo Clinic, 200 First Street SW, Rochester, MN, 55905, USA.
| | - John D Peipert
- Department of Medical Social Sciences, Northwestern University Feinberg School of Medicine, 625 Michigan Ave, 27th Floor, Chicago, IL, 60611, USA
| | - David Cella
- Department of Medical Social Sciences, Northwestern University Feinberg School of Medicine, 625 Michigan Ave, 27th Floor, Chicago, IL, 60611, USA
| | - Kathleen J Yost
- Department of Quantitative Health Sciences, Mayo Clinic, 200 First Street SW, Rochester, MN, 55905, USA
| | - David T Eton
- Department of Quantitative Health Sciences, Mayo Clinic, 200 First Street SW, Rochester, MN, 55905, USA
| | - Paul J Novotny
- Department of Quantitative Health Sciences, Mayo Clinic, 200 First Street SW, Rochester, MN, 55905, USA
| | - Jeff A Sloan
- Department of Quantitative Health Sciences, Mayo Clinic, 200 First Street SW, Rochester, MN, 55905, USA
| | - Amylou C Dueck
- Department of Quantitative Health Sciences, Mayo Clinic, 13400 E. Shea Blvd., Scottsdale, AZ, 85259, USA
| |
Collapse
|
7
|
van der Willik EM, van Breda F, van Jaarsveld BC, van de Putte M, Jetten IW, Dekker FW, Meuleman Y, van Ittersum FJ, Terwee CB. Validity and reliability of Patient-Reported Outcomes Measurement Information System (PROMIS®) using Computerized Adaptive Testing (CAT) in patients with advanced chronic kidney disease. Nephrol Dial Transplant 2022; 38:1158-1169. [PMID: 35913734 PMCID: PMC10157750 DOI: 10.1093/ndt/gfac231] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/07/2022] [Indexed: 11/14/2022] Open
Abstract
BACKGROUND The Patient-Reported Outcomes Measurement Information System (PROMIS®) has been recommended for computerized adaptive testing (CAT) of health-related quality of life (HRQOL). This study compared the content, validity and reliability of seven PROMIS CATs to the 12-item Short-Form Health Survey (SF-12) in patients with advanced chronic kidney disease (CKD). METHODS Adult CKD patients with an eGFR < 30 ml/min.1.73m2 not receiving dialysis treatment completed seven PROMIS CATs (assessing physical function, pain interference, fatigue, sleep disturbance, anxiety, depression, and ability to participate in social roles and activities), the SF-12 and, additionally, the PROMIS Pain Intensity single item and Dialysis Symptom Index (DSI) at inclusion and 2-weeks. A content comparison was performed between PROMIS CATs and SF-12. Construct validity of PROMIS CATs was assessed using Pearson's correlations. Test-retest reliability of all patient-reported outcome measures (PROMs) was assessed by calculating the intra-class correlation coefficient (ICC) and minimal detectable change (MDC). RESULTS In total, 207 patients participated in the study. A median of 45 items (10 minutes) was completed for PROMIS CATs. All PROMIS CATs showed evidence for sufficient construct validity. PROMIS CATs, most SF-12 domains and summary scores, and DSI showed sufficient test-retest reliability (ICC ≥ 0.70). PROMIS CATs had a lower MDC compared to the SF-12 (5.7-7.4 compared to 11.2-21.7 across domains, respectively). CONCLUSION PROMIS CATs showed sufficient construct validity and test-retest reliability in patients with advanced CKD. PROMIS CATs required more items but showed better reliability than the SF-12. Future research is needed to investigate the feasibility of PROMIS CATs for routine nephrology care.
Collapse
Affiliation(s)
- Esmee M van der Willik
- Department of Epidemiology and Data Science, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam Public Health research institute, Amsterdam, The Netherlands.,Department of Clinical Epidemiology, Leiden University Medical Center, Leiden, The Netherlands
| | - Fenna van Breda
- Department of Nephrology, Amsterdam University Medical Centers, Amsterdam, The Netherlands
| | - Brigit C van Jaarsveld
- Department of Nephrology, Amsterdam University Medical Centers, Amsterdam, The Netherlands
| | - Marlon van de Putte
- Department of Nephrology, Amsterdam University Medical Centers, Amsterdam, The Netherlands
| | - Isabelle W Jetten
- Department of Nephrology, Amsterdam University Medical Centers, Amsterdam, The Netherlands
| | - Friedo W Dekker
- Department of Clinical Epidemiology, Leiden University Medical Center, Leiden, The Netherlands
| | - Yvette Meuleman
- Department of Clinical Epidemiology, Leiden University Medical Center, Leiden, The Netherlands
| | - Frans J van Ittersum
- Department of Nephrology, Amsterdam University Medical Centers, Amsterdam, The Netherlands
| | - Caroline B Terwee
- Department of Epidemiology and Data Science, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam Public Health research institute, Amsterdam, The Netherlands
| |
Collapse
|
8
|
Hays RD, Slaughter ME, Spritzer KL, Herman PM. Assessing the Significance of Individual Change in 2 Samples of Patients in Treatment for Low Back Pain Using 5 Different Statistical Indicators. J Manipulative Physiol Ther 2022; 44:699-706. [PMID: 35753878 DOI: 10.1016/j.jmpt.2022.03.002] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/10/2021] [Revised: 03/07/2022] [Accepted: 03/07/2022] [Indexed: 10/17/2022]
Abstract
OBJECTIVE The purpose of this study was to estimate the significance of individual change using 5 statistical indicators in 2 samples of patients treated for low back pain. METHODS This secondary analysis used observational and clinical trial data from 2 samples of patients with low back pain to compare 5 ways of estimating significant individual change on the Impact Stratification Score (ISS) administered at the following 2 time points: 3 months apart in an observational study of 1680 patients undergoing chiropractic care, and 6 weeks apart in a randomized trial of 750 active-duty military personnel with low back pain. The following 5 methods were compared: (1) standard deviation index; (2) standard error of measurement (SEM); (3) standard error of estimate (SEE); (4) standard error of prediction (SEP); and (5) the reliable change index (RCI). The ISS is the sum of the Patient-Reported Outcomes Measurement Information System (PROMIS)-29 v2.1 physical function, pain interference, and pain intensity scores and is scored to have a possible range of 8 (least impact) to 50 (greatest impact). RESULTS The amount of change on the ISS needed for significant individual change in both samples was 5 for the SEM and for the SEE and 7 for the SEP and RCI. CONCLUSIONS The results of the current study provide some preliminary support for use of the SEP or the RCI to identify significant individual change and provide estimated thresholds of individual change that can be used for the ISS. The SEP and RCI estimates of significant change were consistent with retrospective ratings of change of at least moderately better in prior research. These 2 were less likely than other methods to classify people with low back pain as responders who have not actually gotten better (false positive). In contrast, the SEM and SEE were less likely to miss real change (false negative).
Collapse
Affiliation(s)
- Ron D Hays
- Department of Medicine, Division of General Internal Medicine & Health Services Research, University of California Los Angeles, Los Angeles, California.
| | | | - Karen L Spritzer
- Department of Medicine, Division of General Internal Medicine & Health Services Research, University of California Los Angeles, Los Angeles, California
| | | |
Collapse
|
9
|
Reliable change and the reliable change index: still useful after all these years? COGNITIVE BEHAVIOUR THERAPIST 2022. [DOI: 10.1017/s1754470x22000484] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022]
Abstract
Abstract
In 1984 Jacobson and colleagues introduced the concept of reliable change, viz the amount of change on a measure that an individual needed to show to determine that it exceeded the extent of change likely due to measurement error alone. Establishing reliable change was a pre-requisite for determining clinical significance. This paper summarizes the rationale for determining reliable change as providing an individual-focused, idiographic alternative to the dominant nomothetic approach to clinical outcome research based on group mean data and statistical significance. The conventional computational steps for calculating an individual’s standardized difference (reliable change) score and the minimum raw change score on the measure (a reliable change index) required to classify individuals as reliably positively changed, indeterminate, or reliably deteriorated are described. Two methods for graphically representing reliable change are presented, and a range of possible uses in both research and practice settings are summarized. A number of issues and debates concerning the calculation of reliable change are reviewed. It is concluded that the concept of reliable change remains useful for both cognitive behavioural researchers and practitioners, but that there are options regarding methods of computation. In any use of reliable change, the rationale for selecting among method options and the exact computations used need clear and careful description so that we can continuously judge the utility and appropriateness of the use of reliable change and enhance its value to the field.
Key learning aims
(1)
Recognizing why the concept of reliable change and the reliable change index is still important.
(2)
Understanding the conventional formulas for calculating reliable change and the reliable change index (the Jacobson-Truax (JT) method).
(3)
Seeing key ways that both researchers and practitioners can use reliable change to improve both research and practice.
(4)
Understanding how several issues and debates that have arisen concerning the estimation of reliable change (e.g. how to accommodate practice effects) have progressed.
(5)
Recognizing that there are a range of ways that reliable change may be estimated, and the need to provide full details of the method used in any particular instance of its use.
Collapse
|
10
|
Abstract
BACKGROUND Pilot studies test the feasibility of methods and procedures to be used in larger-scale studies. Although numerous articles describe guidelines for the conduct of pilot studies, few have included specific feasibility indicators or strategies for evaluating multiple aspects of feasibility. In addition, using pilot studies to estimate effect sizes to plan sample sizes for subsequent randomized controlled trials has been challenged; however, there has been little consensus on alternative strategies. METHODS In Section 1, specific indicators (recruitment, retention, intervention fidelity, acceptability, adherence, and engagement) are presented for feasibility assessment of data collection methods and intervention implementation. Section 1 also highlights the importance of examining feasibility when adapting an intervention tested in mainstream populations to a new more diverse group. In Section 2, statistical and design issues are presented, including sample sizes for pilot studies, estimates of minimally important differences, design effects, confidence intervals (CI) and nonparametric statistics. An in-depth treatment of the limits of effect size estimation as well as process variables is presented. Tables showing CI around parameters are provided. With small samples, effect size, completion and adherence rate estimates will have large CI. CONCLUSION This commentary offers examples of indicators for evaluating feasibility, and of the limits of effect size estimation in pilot studies. As demonstrated, most pilot studies should not be used to estimate effect sizes, provide power calculations for statistical tests or perform exploratory analyses of efficacy. It is hoped that these guidelines will be useful to those planning pilot/feasibility studies before a larger-scale study.
Collapse
Affiliation(s)
- Jeanne A. Teresi
- Columbia University Stroud Center at New York State
Psychiatric Institute, 1051 Riverside Drive, Box 42, Room 2714, New York, New York,
10032-3702, USA
- Research Division, Hebrew Home at Riverdale, 5901 Palisade
Avenue, Riverdale New York 10471
| | - Xiaoying Yu
- Office of Biostatistics, Department of Preventive Medicine
and Population Health, University of Texas Medical Branch at Galveston, 301
University Boulevard, Galveston, Texas, 77555-1147
| | - Anita L. Stewart
- University of California, San Francisco, Institute for
Health & Aging, 490 Illinois St., 12 floor, Box 0646, San
Francisco, CA 94143
| | - Ron D. Hays
- University of California, Los Angeles; Division of General
Internal Medicine and Health Services Research, 1100 Glendon Avenue, Suite 850, Los
Angeles, California 90024
| |
Collapse
|
11
|
Maydeu-Olivares A. Assessing the Accuracy of Errors of Measurement. Implications for Assessing Reliable Change in Clinical settings. PSYCHOMETRIKA 2021; 86:793-799. [PMID: 34453659 DOI: 10.1007/s11336-021-09806-w] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/10/2021] [Revised: 08/10/2021] [Indexed: 06/13/2023]
Abstract
Item response theory (IRT) models are non-linear latent variable models for discrete measures, whereas factor analysis (FA) is a latent variable model for continuous measures. In FA, the standard error (SE) of individuals' scores is common for all individuals. In IRT, the SE depends on the individual's score, and the SE function is to be provided. The empirical standard deviation of the scores across discrete ranges should also be computed to inform the extent to which IRT SEs overestimate or underestimate the variability of the scores. Within the target range of scores the test was designed to measure, one should expect IRT SEs to be smaller and more precise than FA SEs, and therefore preferable to assess clinical change. Outside the target range, IRT SEs may be too large and more imprecise than FA SEs, and FA more precise to assess change. As a result, whether FA or IRT characterize reliable change more accurately in a sample will depend on the proportion of individuals within or outside the IRT target score range. An application is provided to illustrate these concepts.
Collapse
Affiliation(s)
- Alberto Maydeu-Olivares
- Department of Psychology, University of South Carolina, Barnwell College, 1512 Pendleton St., Columbia, SC, 29208, USA.
- University of Barcelona, Barcelona, Spain.
| |
Collapse
|
12
|
Reeve BB, Hays RD. Guest Editors' Introduction to the Invited Special Section. PSYCHOMETRIKA 2021; 86:671-673. [PMID: 34390454 DOI: 10.1007/s11336-021-09795-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Affiliation(s)
- Bryce B Reeve
- Duke University School of Medicine, Durham, NC, USA.
| | | |
Collapse
|
13
|
Abstract
PURPOSE Estimates of the minimally important change (MIC) can be used to evaluate whether group-level differences are large enough to be important. But responders to treatment have been based upon group-level MIC thresholds, resulting in inaccurate classification of change over time. This article reviews options and provides suggestions about individual-level statistics to assess whether individuals have improved, stayed the same, or declined. METHODS Review of MIC estimation and an example of misapplication of MIC group-level estimates to assess individual change. Secondary data analysis to show how perceptions about meaningful change can be used along with significance of individual change. RESULTS MIC thresholds yield over-optimistic conclusions about responders to treatment because they classify those who have not changed as responders. CONCLUSIONS Future studies need to evaluate the significance of individual change using appropriate individual-level statistics such as the reliable change index or the equivalent coefficient of repeatability. Supplementing individual statistical significance with retrospective assessments of change is desirable.
Collapse
|