Utilizing a learning health system to capture real-world patient data: Application of the reliable change index to evaluate and improve the outcome of a pain rehabilitation program

BACKGROUND AND OBJECTIVES

The learning healthcare system (LHS) has been developed to integrate patients' clinical data into clinical decisions and improve treatment outcomes. Having little guidance on this integration process, we aim to explain (a) an applicable analytic tool for clinicians to evaluate the clinical outcomes at a group and an individual level and (b) our quality improvement (QI) project, analyzing the outcomes of a new outpatient pain rehabilitation program ("Back-in-Action": BIA) and applying the analysis results to modify our clinical practice.

METHODS

Through our LHS (CHOIR; https://choir.stanford.edu), we administered the Pain Catastrophizing Scale (PCS), Chronic Pain Acceptance Questionnaire (CPAQ), and Patient-Reported Outcomes Measures (PROMIS)® before and after BIA. After searching for appropriate analytic tools, we decided to use the Reliable Change Index (RCI) to determine if an observed change in the direction of better (improvement) or worse (deterioration) would be beyond or within the measurement error (no change).

RESULTS

Our RCI calculations revealed that at least a 9-point decrease in the PCS scores and 10-point increase in the CPAQ scores would indicate reliable improvement. RCIs for the PROMIS measures ranged from 5 to 8 T-score points (i.e., 0.5-0.8 SD). When evaluating change scores of the PCS, CPAQ, and PROMIS measures, we found that 94% of patients showed improvement in at least one domain after BIA and 6% showed no reliable improvement.

CONCLUSIONS

Our QI project revealed RCI as a useful tool to evaluate treatment outcomes at a group and an individual level, and RCI could be incorporated into the LHS to generate a progress report automatically for clinicians. We further explained how clinicians could use RCI results to modify a clinical practice, to improve the outcomes of a pain program, and to develop individualized care plans. Lastly, we suggested future research areas to improve the LHS application in pain practice.

Test-retest reliability and reliable change index of the Philips IntelliSpace Cognition digital test battery

OBJECTIVE

This article provides the test-retest reliability and Reliable Change Indices (RCIs) of the Philips IntelliSpace Cognition (ISC) platform, which contains digitized versions of well-established neuropsychological tests.

METHOD

147 participants (ages 19 to 88) completed a digital cognitive test battery on the ISC platform or paper-pencil versions of the same test battery during two separate visits. Intraclass correlation coefficients (ICC) were calculated separately for the ISC and analog test versions to compare reliabilities between administration modalities. RCIs were calculated for the digital tests using the practice-adjusted RCI and standardized regression-based (SRB) method.

RESULTS

Test-retest reliabilities for the ISC tests ranged from moderate to excellent and were comparable to the test-retest reliabilities for the paper-pencil tests. Baseline test performance, retest interval, age, and education predicted test performance at visit 2 with baseline test performance being the strongest predictor for all outcome measures. For most outcome measures, both methods for the calculation of RCIs show agreement on whether or not a reliable change was observed.

CONCLUSIONS

RCIs for the digital tests enable clinicians to determine whether a measured change between assessments is due to real improvement or decline. Together, this contributes to the growing evidence for the clinical utility of the ISC platform.

Health Economic Evaluation of an Online-Based Motivational Program to Reduce Problematic Media Use and Promote Treatment Motivation for Internet Use Disorder-Results of the OMPRIS Study

Internet Use Disorders (IUD) have a relevant effect on national economies. In the randomized, controlled, multicenter, prospective, and single-blinded OMPRIS study (pre-registration number DRKS00019925; Innovation Fund of the Joint Federal Committee of Germany, grant number 01VSF18043), a four-week online program to reduce media addiction symptoms, was evaluated for cost-effectiveness. The intervention group (IG) was compared to a waiting control group (WCG) from German statutory health insurance (SHI) and a societal perspective. Resource use, namely indirect and direct (non) medical costs, was assessed by a standardized questionnaire at baseline and after the intervention. Additionally, intervention costs were calculated. Determining the Reliable Change Index (RCI) based on the primary outcome, assessed by the "Scale for the Assessment of Internet and Computer Game Addiction" (AICA-S), individuals with and without reliable change (RC) were distinguished. The incremental cost-effectiveness ratio was calculated using the difference-in-difference approach. There were 169 (IG n = 81, WCG n = 88) persons included in the analysis. The mean age was 31.9 (SD 12.1) years. A total of 75.1% were male, and 1.8% diverse. A total of 65% (IG) and 27% (WCG) had an RC. The cost per person with RC was about EUR 860 (SHI) and EUR 1110 (society). The intervention leads to an improvement of media addiction symptoms at moderate additional costs.

Detecting Change in Needs-Supplies Fit Through Reliable Change Methodology

Studying change is a critical part of psychology and science in general. Studies often treat fit as static and use between-person designs to assess change. Accordingly, potentially insightful within-person information is frequently overlooked. The current study aimed to establish the utility of reliable change methodology within the domain of organizational psychology, using needs-supplies fit as a guiding framework. When employee needs can be tracked with a fair degree of clarity, organizations can devise better strategies to routinely address discrepancies between desired employee needs and organizational offerings. This longitudinal study used secondary data from 258 secondary school teachers. The Needs-Supplies Fit Scale was administered. The study's hypothesis was assessed through reliable change methodology. When considering all the participants that experienced at least some change across time intervals (n = 148), 23.65% (n = 35) of this change was meaningful. This declined to 17.33% meaningful change when factoring in the entire sample, including those who did not experience any change. When organizations are aware of the ever-evolving needs of employees, quicker action can be taken to avoid impending person-environment misfit. This study contributes to existing within-person studies that showcased the malleability of needs-supplies fit and emphasize the value of placing a more prominent focus on the individual.

Evaluating 'living well' with mild-to-moderate dementia: Co-production and validation of the IDEAL My Life Questionnaire

OBJECTIVES

We aimed to co-produce and validate an accessible, evidence-based questionnaire measuring 'living well' with dementia that reflects the experience of people with mild-to-moderate dementia.

METHODS

Nine people with dementia formed a co-production group. An initial series of workshops generated the format of the questionnaire and a longlist of items. Preliminary testing with 53 IDEAL cohort participants yielded a shortlist of items. These were tested with 136 IDEAL cohort participants during a further round of data collection and assessed for reliability and validity. The co-production group contributed to decisions throughout and agreed the final version.

RESULTS

An initial list of 230 items was reduced to 41 for initial testing, 12 for full testing, and 10 for the final version. The 10-item version had good internal consistency and test-retest reliability, and a single factor structure. Analyses showed significant large positive correlations with scores on measures of quality of life, well-being, and satisfaction with life, and expected patterns of association including a significant large negative association with depression scores and no association with cognitive test scores.

CONCLUSIONS

The co-produced My Life Questionnaire is an accessible and valid measure of 'living well' with dementia suitable for use in a range of contexts.

Does Scoring Method Impact Estimation of Significant Individual Changes Assessed by Patient-Reported Outcome Measures? Comparing Classical Test Theory Versus Item Response Theory

OBJECTIVES

This study aimed to examine the ability of classical test theory (CTT) and item response theory (IRT) scores assessed by Patient-Reported Outcomes Measurement Information System® (PROMIS®) measures to identify significant individual changes in the setting of clinical studies, using both simulated and empirical data.

METHODS

We used simulated data to compare the estimation of significant individual changes between CTT and IRT scores across different conditions and a clinical trial data set to verify the simulation results. We calculated reliable change indexes to estimate significant individual changes.

RESULTS

For small true change, IRT scores showed a slightly higher rate of classifying change groups than CTT scores and were comparable with CTT scores for a shorter test length. Additionally, IRT scores were found to have a prominent advantage in the classification rates of change groups for medium to high true change over CTT scores. Such an advantage became prominent in a longer test length. The empirical data analysis results using an anchor-based approach further supported the above findings that IRT scores can more accurately classify participants into change groups than CTT scores.

CONCLUSIONS

Given that IRT scores perform better, or at least comparably, in most conditions, we recommend using IRT scores to estimate significant individual changes and identify responders to treatment. This study provides evidence-based guidance in detecting individual changes based on CTT and IRT scores under various measurement conditions and leads to recommendations for identifying responders to treatment for participants in clinical trials.

Cognitive function and breast cancer molecular subtype before and after chemotherapy

Chemotherapy-related cognitive impairment has been reported in patients with breast cancer and received growing attention due to increased survival rate. However, cognitive outcome according to pathological tumor features, especially human epidermal growth factor receptor (HER2) status, has not been clearly elucidated. Despite its potential link with cognitive status through neuroinflammatory response, existing research is sparse and limited to cross-sectional studies. In this observational cohort study, 52 breast cancer patients received a series of neuropsychological examinations before and after chemotherapy. Patients' performances were compared with normative data, and analyzed with Reliable Change Indices and mixed-model analysis of covariance. Results showed that there was a higher percentage of HER2+ patients than HER2- patients who showed defective attention and processing speed before chemotherapy, and that there were more patients with HER2+ status showing cognitive decline on tests of attention and executive functions following chemotherapy. Group-wise analyses confirmed the foregoing pattern and further revealed that patients with HER2+ status also tended to deteriorate more in verbal memory after chemotherapy. These findings indicate that HER2 overexpression may serve as prognostic factors that help explain the heterogeneous cognitive outcome in breast cancer survivors. Further studies are needed to replicate this finding and delineate the underlying mechanisms.

Determining the reliable feature change in longitudinal radiomics studies: A methodological approach using the reliable change index

PURPOSE

Determination of reliable change of radiomics feature over time is essential and vital in delta-radiomics, but has not yet been rigorously examined. This study attempts to propose a methodological approach using reliable change index (RCI), a statistical metric to determine the reliability of quantitative biomarker changes by accounting for the baseline measurement standard error, in delta-radiomics. The use of RCI was demonstrated with the MRI data acquired from a group of prostate cancer (PCa) patients treated by 1.5 T MRI-guided radiotherapy (MRgRT).

METHODS

Fifty consecutive PCa patients who underwent five-fractionated MRgRT were retrospectively included, and 1023 radiomics features were extracted from the clinical target volume (CTV) and planning target volume (PTV). The two MRI datasets acquired at the first fraction (MRI11 and MRI21) were used to calculate the baseline feature reliability against image acquisition using intraclass correlation coefficient (ICC). The RCI was constructed based on the baseline feature measurement standard deviation, ICC, and feature value differences at two time points between the fifth (MRI51) and the first fraction MRI (MRI11). The reliable change of features was determined in each patient only if the calculated RCI was over 1.96 or smaller than -1.96. The feature changes between MRI51 and MRI11 were correlated to two patient-reported quality-of-life clinical endpoints of urinary domain summary score (UDSS) and bowel domain summary score (BDSS) in 35 patients using the Spearman correlation test. Only the significant correlations between a feature that was reliably changed in ≥7 patients (20%) by RCI and an endpoint were considered as true significant correlations.

RESULTS

The 352 (34.4%) and 386 (37.7%) features among all 1023 features were determined by RCI to be reliably changed in more than five (10%) patients in the CTV and PTV, respectively. Nineteen features were found reliably changed in the CTV and 31 features in the PTV, respectively, in 10 (20%) or more patients. These features were not necessarily associated with significantly different longitudinal feature values (group p-value < 0.05). Most reliably changed features in more than 10 patients had excellent or good baseline test-retest reliability ICC, while none showed poor reliability. The RCI method ruled out the features to be reliably changed when substantial feature measurement bias was presented. After applying the RCI criterion, only four and five true significant correlations were confirmed with UDSS and BDSS in the CTV, respectively, with low true significance correlation rates of 10.8% (4/37) and 17.9% (5/28). No true significant correlations were found in the PTV.

CONCLUSIONS

The RCI method was proposed for delta-radiomics and demonstrated using PCa MRgRT data. The RCI has advantages over some other statistical metrics commonly used in the previous delta-radiomics studies, and is useful to reliably identify the longitudinal radiomics feature change on an individual basis. This proposed RCI method should be helpful for the development of essential feature selection methodology in delta-radiomics.

The development of a reliable change index and cutoff score for the SCORE-15

The Systemic Clinical Outcome and Routine Evaluation version 15 (SCORE-15) is a measure used to assess family-level change in family therapy. The SCORE-15 has been demonstrated to be a reliable and valid measure, with high clinical utility; however, the SCORE-15 lacks the ability to determine whether the change in family functioning during the course of therapy is clinically significant. This study aimed to establish a reliable change index (RCI) and clinical cutoff score so that researchers and clinicians can determine clinically significant change in family therapy. US samples of 71 clinical participants and 244 community participants completed the SCORE-15. Results indicated a cutoff score of 40.37 and an RCI of 9.52. Consequently, family members who improve their SCORE-15 score during the course of therapy by at least 9 points and who cross the threshold of 40 during the course of therapy are considered to have experienced clinically significant change.

Logistic versus linear regression-based reliable change index: A simulation study with implications for clinical studies with different sample sizes

The linear regression-based reliable change index (RCI) is widely used to identify memory impairments through longitudinal assessment. However, the minimum sample size required for estimates to be reliable has never been specified. Using data from 920 participants from the Alzheimer's Disease Neuroimaging Initiative data as true parameters, we run 12,000 simulations for samples of size 10-1,000 and analyzed the percentage of times the estimates are significant, their coverage rate, and the accuracy of the models including both the true-positive rate and the true-negative rate. We compared the linear RCI with a logistic RCI for discrete, bounded scores. We found that the logistic RCI is more accurate than the linear RCI overall, with the linear RCI approximating the logistic RCI for samples of size 200 or greater. We provide an R package to compute the logistic RCI, which can be downloaded from the Comprehensive R Archive Network (CRAN) at https://cran.r-project.org/web/packages/LogisticRCI/, and the code to reproduce all results in this article at https://github.com/rafamoral/LogisticRCIpaper/. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Measuring Clinically Meaningful Change in Outcomes for Youth With Chronic Pain Following Graded Exposure Treatment

INTRODUCTION

Functional improvement is a critical outcome for individuals living with chronic pain. Graded exposure treatment (GET) has been associated with statistically significant improvements in functional outcomes for youth with chronic pain by targeting pain-related fear and avoidance.

OBJECTIVE

The aim of the present study was to explore clinically meaningful change in outcomes in adolescents with chronic pain following participation in a GET, and to then classify patients as treatment responders versus nonresponders.

MATERIALS AND METHODS

Participants included 27 youth (Mage=13.5) with chronic pain enrolled in a recently published single-arm randomized baseline trial of GET Living. Reliable change at the individual level was assessed using the Reliable Change Index (RCI). Adolescents were classified as treatment responders if they achieved a reliable change in outcomes across time points and also demonstrated a change in clinical severity range in the expected direction (ie, from severe to moderate).

RESULTS

Reliable and clinically significant improvements in pain-related fear and avoidance, functional disability, and school functioning were demonstrated at discharge, with improvements maintained at 3-month and 6-month follow-up. Among core outcomes, 48% (n=13) of patients were classified as treatment responders in one or more outcomes at discharge, with this increasing to 76% (n=19) at 3-month follow-up.

DISCUSSION

Examining reliable and clinically meaningful change (vs. statistical significance alone) provides a way to examine treatment response to an intervention and to enhance the interpretability of findings, helping to bridge the gap between clinical trials and clinical practice by providing guidelines for interpretation.

A Comparison of Cross-Sectional and Longitudinal Methods of Defining Objective Subtle Cognitive Decline in Preclinical Alzheimer's Disease Based on Cogstate One Card Learning Accuracy Performance

BACKGROUND

Longitudinal, but not cross-sectional, cognitive testing is one option proposed to define transitional cognitive decline for individuals on the Alzheimer's disease continuum.

OBJECTIVE

Compare diagnostic accuracy of cross-sectional subtle objective cognitive impairment (sOBJ) and longitudinal objective decline (ΔOBJ) over 30 months for identifying 1) cognitively unimpaired participants with preclinical Alzheimer's disease defined by elevated brain amyloid and tau (A+T+) and 2) incident mild cognitive impairment (MCI) based on Cogstate One Card Learning (OCL) accuracy performance.

METHODS

Mayo Clinic Study of Aging cognitively unimpaired participants aged 50 + with amyloid and tau PET scans (n = 311) comprised the biomarker-defined sample. A case-control sample of participants aged 65 + remaining cognitively unimpaired for at least 30 months included 64 who subsequently developed MCI (incident MCI cases) and 184 controls, risk-set matched by age, sex, education, and visit number. sOBJ was assessed by OCL z-scores. ΔOBJ was assessed using within subjects' standard deviation and annualized change from linear regression or linear mixed effects (LME) models. Concordance measures Area Under the ROC Curve (AUC) or C-statistic and odds ratios (OR) from conditional logistic regression models were derived. sOBJ and ΔOBJ were modeled jointly to compare methods.

RESULTS

sOBJ and ΔOBJ-LME methods differentiated A+T+ from A-T- (AUC = 0.64, 0.69) and controls from incident MCI (C-statistic = 0.59, 0.69) better than chance; other ΔOBJ methods did not. ΔOBJ-LME improved prediction of future MCI over baseline sOBJ (p = 0.003) but not over 30-month sOBJ (p = 0.09).

CONCLUSION

Longitudinal decline did not offer substantial benefit over cross-sectional assessment in detecting preclinical Alzheimer's disease or incident MCI.

Methodological considerations of assessing meaningful/reliable change in computerized neurocognitive testing following sport-related concussion

OBJECTIVE

Traditional reliable change index (RCI) methods may be psychometrically limited due to their inability to account for particularly high or low baseline performance and regression to the mean following serial testing. The current study sought to examine differences between RCI and standardized regression-based (RBz) methods in a cohort of adolescent athletes engaged in sport-related concussion recovery.

METHODS

Consultation records and results of computerized testing data via the Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT) battery were retrospectively reviewed among 240 adolescent athletes involved in return-to-play protocols following a sport-related concussion.

RESULTS

High concordance rates were found between RCI and RBz metrics across ImPACT composites, consistent with previous literature. However, especially for those with lower baseline performance, RBz scores tended to suggest cognitive performance not meeting or exceeding baseline scores despite RCI metrics being appropriate across speed-based ImPACT composites. In contrast, results revealed high rates of RCI scores suggesting continued cognitive difficulties despite RBz metrics being within normal limits, especially for adolescents with higher baseline performance.

CONCLUSIONS

Results suggest value in interpreting RBz values in addition to RCI values as these allow for clinical interpretation more sensitive to statistical confounds, including baseline performance and regression to the mean.

In-Treatment Changes in Quality of Life-Related Variables in Therapeutic Communities for Cocaine Abusers: Are These Changes Associated with Clinical Outcomes?

BACKGROUND

Few studies have explored changes in quality of life during the first three months of admission to a therapeutic community for addictions. The objective of this study was to determine the relationship between these changes and treatment outcomes at discharge.

METHODS

We undertook a prospective longitudinal study of 142 cocaine-dependent patients treated at a therapeutics community. All of these patients reached the 3-month evaluation and were subsequently followed until discharge. All participants completed the following measures: Health-Related Quality of Life for Drug Abusers Test; Beck Depression Inventory; State-Trait Anxiety Inventory; Opiate Treatment Index; Dual Diagnosis and Discharge Checklist.

RESULTS

At the 3-month assessment, scores on the Health-Related Quality of Life for Drug Abusers Test had increased significantly (Cohen's d: 0.92), while scores on the Opiate Treatment Index (Cohen's d: 0.86) and Beck Depression Inventory (Cohen's d: 0.20) scales both decreased significantly. A higher proportion of the patients considered to have achieved "clinically relevant" treatment outcomes at discharge versus those without clinically relevant outcomes were considered "recovered" according to the Reliable Change Index.

CONCLUSIONS

An improvement in quality of life-related variables from baseline to the 3-month assessment was associated with better outcomes at discharge from the therapeutic community. The findings of this study may help us to optimise therapeutic interventions.

Individual Differences in the Effects of Physical Activity on Cognitive Function in People with Mild to Moderate Dementia

The aim of this study was to investigate whether the effect of physical activity on cognitive function in persons with dementia is moderated by patient characteristics as Apolipoprotein E and dementia type. We included 101 individuals with dementia and calculated the reliable change index to determine the change in global cognition, executive function, episodic memory, working memory, and processing speed before and after a 12-week exercise training. We found a higher treatment-related benefit in episodic memory in persons with non-Alzheimer’s disease compared to persons with Alzheimer’s disease, and in executive function in individuals with better baseline cognitive function.

Long-Term Effects of Home-Based Family Therapy for Non-responding Adolescents With Psychiatric Disorders. A 3-Year Follow-Up

Objective

Home-based treatment of families with low socio-economic status and multiple psychosocial problems (multi-problem families, MPFs) is gaining importance in clinical social epidemiology and health services research. The sustainability of the treatment is of special importance in order to breach transgenerational effects.

Methods

We examined outcome, effect size, and clinical significance of home-based treatment for 84 multi-problem families in a naturalistic setting. 48 of the families were available for a follow-up after 3 years. The baseline characteristics of these family systems included low collaboration, an increased family adversity index, minors with high rates of child psychiatric disorders, a high prevalence of comorbidity, low relational family functioning, and adolescents who refused any form of treatment or had unilaterally terminated different forms of treatment before. The home-based family therapy consisted of one or two face-to-face counseling sessions per week over an average of 28.8 months (SD = 19.2). The symptoms and competence of the adolescents, the caregivers, and the family structure were assessed with 13 variables.

Results

All variables showed significant improvement rates (pre- vs. post- treatment) with medium to high effect sizes (mean of Cohen's d = 1.04, range = 0.34 - 2.18). All variables showed a sustained or even further improvement at follow-up.

Conclusion

This study provides evidence of statistically (p), practically (d), and clinically (RCI) significant changes in symptom and competence-related variables among adolescents and caregivers in MPFs with sustainable long-term effects in the 3-year follow-up period.

Assessing Cognitive Change and Quality of Life 12 Months After Epilepsy Surgery-Development and Application of Reliable Change Indices and Standardized Regression-Based Change Norms for a Neuropsychological Test Battery in the German Language

Objective

The establishment of patient-centered measures capable of empirically determining meaningful cognitive change after surgery can significantly improve the medical care of epilepsy patients. Thus, this study aimed to develop reliable change indices (RCIs) and standardized regression-based (SRB) change norms for a comprehensive neuropsychological test battery in the German language.

Methods

Forty-seven consecutive patients with temporal lobe epilepsy underwent neuropsychological assessments, both before and 12 months after surgery. Practice-effect-adjusted RCIs and SRB change norms for each test score were computed. To assess their usefulness, the presented methods were applied to a clinical sample, and binary logistic regression analyses were conducted to model the odds of achieving improvement in quality of life (QOL) after surgery.

Results

The determined RCIs at 90% confidence intervals and the SRB equations for each test score included in the test battery are provided. Cohen’s kappa analyses revealed a moderate mean agreement between the two measures, varying from slight to almost perfect agreement across test scores. Using these measures, a negative association between improvement in QOL and decline in verbal memory functions after surgery was detected (adjusted odds ratio = 0.09, p = 0.006).

Significance

To the best of our knowledge, this study is the first to develop RCIs and SRB change norms necessary for the objective determination of neuropsychological change in a comprehensive test battery in the German language, facilitating the individual monitoring of improvement and decline in each patients’ cognitive functioning and psychosocial situations after epilepsy surgery. The application of the described measures revealed a strong negative association between improvement in QOL and decline in verbal memory functions after surgery.

Reliable change in neuropsychological test scores is associated with brain atrophy in older adults

The reliable change index (RCI) is a commonly used method for interpreting change in neuropsychological test scores over time. However, the RCI is a psychometric method that, to date, has not been validated against neuroanatomical changes. Longitudinal neuroimaging and neuropsychological data from baseline and one-year follow-up visits were retrieved from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database. The RCI was used to identify participants showing reliable decline on memory (ADNI-Mem; N = 450) and executive functioning (ADNI-EF; N = 456) factor scores. For each factor score, two groups (reliable change vs. no reliable change) were matched on potential baseline confounding variables. Longitudinal neuroanatomical data were analysed using tensor-based morphometry. Analysis revealed that reliable change on ADNI-Mem was associated with atrophy in the medial temporal cortex, limbic cortex, temporal lobe and some regions of the parietal lobe. Similar atrophy patterns were found for reliable change on ADNI-EF, except that atrophy extended to the frontal lobe and the atrophy was more extensive and of higher magnitude. The current study not only validates clinical usage of the RCI with neuroanatomical evidence of associated underlying brain change but also suggests patterns of likely brain atrophy when reliable cognitive decline is detected.

An Exploratory Study of Executive Function Development in Children with Autism, after Receiving Early Intensive Behavioral Training

Objective: To examine the development of executive functions, in preschool children with autism spectrum disorders (ASD), receiving early intensive behavioral training (EIBI). Method: Executive functions (EF) were assessed with The Behavior Rating Inventory of Executive Function - Preschool Version (BRIEF-P), by parents and preschool teachers at the time of diagnostic assessment and after 15 months of EIBI intervention. Ten children with ASD (M = 2.9 years, nine males) participated in the study. Reliable Change Index scores were computed for each of the participants in order to investigate any significant change in BRIEF-P T-scores. Results: Three children showed a significant improvement in EF, based on parent ratings. Four children showed a significant improvement in EF based on preschool teacher ratings. Conclusion: Findings indicating a reliable improvement in one third of preschool children with ASD receiving EIBI are encouraging but need to be replicated in larger scale controlled studies.

Cognitive outcomes in meningioma patients undergoing surgery: individual changes over time and predictors of late cognitive functioning

BACKGROUND

Meningioma patients are known to face cognitive deficits before and after surgery. We examined individual changes in cognitive performance over time and identified preoperative predictors of cognitive functioning 12 months after surgery in a large sample of meningioma patients.

METHODS

Patients underwent neuropsychological assessment (NPA) using CNS Vital Signs 1 day before (T0) and 3 (T3) and 12 (T12) months after surgery. Patients' sociodemographically corrected scores on 7 cognitive domains were compared with performance of a normative sample using one-sample z tests and chi-square tests of independence. Reliable change indices with correction for practice effects were calculated for individual patients. Linear mixed effects models were used to identify preoperative predictors of performance at T12.

RESULTS

At T0, 261 patients were assessed, and 229 and 82 patients were retested at T3 and T12, respectively. Patients showed impaired cognitive performance before and after surgery, and although performance improved on the group level, cognitive scores remained significantly lower than in the normative sample up to T12. On the individual level, performance remained stable in the majority of patients. Better preoperative performance, younger age, male sex, and higher educational level predicted better late cognitive performance.

CONCLUSIONS

Meningioma patients face serious and persistent pre- and postsurgical cognitive deficits. A preoperative NPA together with sociodemographic characteristics may provide valuable information on the late cognitive outcome of individual meningioma patients. These results can help to inform patients and clinicians on late cognitive outcomes at an early stage, and emphasizes the importance of presurgical NPA and timely cognitive rehabilitation.

Critical consideration of assessment methods for clinically significant changes of mental distress after psycho-oncological interventions

OBJECTIVES

Considering the heterogeneity of cancer entities and the associated disease progression, personalized care of patients is increasingly emphasized in psycho-oncology. This individualization makes the use of measurements of individual clinically significant change important when studying the efficacy and effectiveness of psycho-oncological care. Two conceptualizations for the measurement of clinical significance are critically contrasted in this study: the Reliable Change Index (RCI) and the Minimal Important Difference (MID) method.

METHODS

In total, 2,121 cancer patients participated in the study and a subsample of 708 patients was reassessed about 4 months later. Psychological distress was measured using the Hospital Anxiety and Depression Scale. We evaluated two measures of clinical significance (RCI, MID) by comparing the respective numbers of improved, unimproved, and deteriorated patients.

RESULTS

Individually significant changes were observed with both methods; however, determined rates of improvement differed substantially: MID (66.67%) and RCI (48.23%). Most importantly, according to MID, 17.93% of patients were identified as being improved, although their respective improvements were not statistically significant and thus unreliable.

CONCLUSIONS

The benefits of RCI outweigh MID, and therefore, the RCI is recommended as a measure to assess change.

Clinically reliable cognitive decline in relapsing remitting multiple sclerosis: Is it the tip of the iceberg?

OBJECTIVES

Cognitive impairment is common in multiple sclerosis, but the brain MRI correlates in relapsing remitting multiple sclerosis remain controversial. The current study aimed to investigate whether cognitive decline can be predicted by global and/or regional brain atrophy.

METHODS

Sixty-three patients with relapsing remitting multiple sclerosis (36 men, mean age 39.9 ± 9.4 years old, mean EDSS 1.4 ± 1.2, mean disease duration 4.9 ± 4.3 years) and 46 healthy controls (21 men, mean age 37.5 ± 10.8 years old) were included. Demographic data were obtained, and a longitudinal neuropsychological and global and regional MRI brain volume assessment was performed.

RESULTS

The patients performed worse than controls in most neuropsychological tests at baseline. The percentage of patients with clinically meaningful cognitive decline ranged from only 0% to 7.9%. Statistically significant volume reduction was found for all MRI measures except for the left accumbens nucleus. Whole or regional brain atrophy ranged from -0.02% to -0.25%, with subcortical structures showing the largest atrophy rates. A total of 22.2% to 93.7% patients showed atrophy across the brain structures assessed volumetrically.

DISCUSSION

It was regional rather than whole-brain changes that significantly predicted cognitive decline for the patients in the tasks that tested processing speed, visuo-spatial and inhibition skills. The overall data suggest that the progression of cognitive impairment in relapsing remitting multiple sclerosis as captured by conventional neuropsychological testing is the tip of the iceberg of neurodegenerative sequelae in the disease. Also, regional volumetric changes are better than whole-brain atrophy at predicting cognitive dysfunction.

[Formula: see text] Reliable change in pediatric brain tumor: A preliminary investigation

Children treated for brain tumor show evidence of declines in general intellectual abilities (i.e., IQ). Group-level data indicate subtle declines over time on average, but no study has utilized a clinical criterion to identify and describe a reliable change in survivors of pediatric brain tumor (PBT). In this study, we discuss the utility of reliable change index (RCI) methodology to supplement group-level analysis (e.g., repeated measures ANOVA). This pilot sample consisted of 22 children (M age = 10.47 years) treated for PBT who completed initial and follow-up assessments (M interval = 23.58 months). Cognitive data included composite scores from the WISC-IV. An RCI z-score was calculated for each participant on each composite score based on two different test-retest reliability coefficients. As a group, survivors of PBT did not demonstrate a statistically significant change from initial to follow-up on any WISC-IV composite score. When RCI was calculated based on reliability coefficients with shorter test-retest intervals provided by the test publisher, 77% of survivors demonstrated a reliable change in performance on at least one measure. The frequency of RCI decreases in working memory was significantly higher than expected. In contrast, only 32% of survivors showed reliable changes on at least one measure when RCI was based on a reliability coefficient derived from a clinical sample with a longer retest interval. This study demonstrates that highly divergent results may be obtained with RCI and the importance of the source of reliability estimates.

Depression, quality of life, activities of daily living, and subjective memory after deep brain stimulation in Parkinson disease-A reliable change index analysis

OBJECTIVES

In the field of Parkinson disease (PD) research, many studies have shown that deep brain stimulation (DBS) can soften side effects, which arise during long-term medical therapy. This study focuses on the changes in depressive symptoms, quality of life (with the subdivisions physical and mental health), activities of daily living, and subjective memory functioning in PD patients testing the baseline and the outcome 1 year after DBS.

METHODS

For the first time, the reliable change index (RCI) methodology was applied to compare PD-DBS patients (n = 22) with best medically treated PD patients (PD-BMT; n = 28), subjects with mild cognitive impairment (MCI, n = 43) and healthy controls (n = 25) in the above-mentioned domains. The used questionnaires included the revised Beck Depression Inventory (BDI-II), the Short Form (36) Health Survey (SF-36), the Bayer Activities of Daily Living Scale (B-ADL), and the Forgetfulness Assessment Inventory (FAI).

RESULTS

The reliable change indices show high constant or improved results of the PD-DBS patients in the domains subjective memory (85.7%-100.0%), activities of daily living (60.0%-90.0%), physical health summary (77.8%), depressive symptoms (61.9%), and mental health summary (50.0%) in comparison with the PD-BMT, MCI, and control group.

CONCLUSIONS

DBS is an established alternative to best medical treatment of PD. The comparisons between the PD-DBS and PD-BMT groups do suggest that the domains mental health, depressive symptoms, and physical health benefit most, while the domains activities of daily living and subjective memory functioning are rather constant. Nevertheless, further research is needed to identify mechanisms and predictors that lead to improvement in individual cases.

Cognitive functioning in patients with nonfunctioning pituitary adenoma before and after endoscopic endonasal transsphenoidal surgery

OBJECTIVE

Patients with nonfunctioning pituitary adenomas (NFPAs) can suffer from cognitive dysfunction. However, the literature on longitudinal cognitive follow-up of patients undergoing endoscopic endonasal transsphenoidal surgery (EETS) is limited. This study was performed to investigate perioperative cognitive status and course in patients with NFPAs.

METHODS

Patients underwent computerized neuropsychological assessment 1 day before (n = 45) and 3 months after (n = 36) EETS. Performance in 7 domains was measured with a computerized test battery (CNS Vital Signs) and standardized using data from a healthy control group. The authors conducted analyses of cognitive performance at both time points and changes pre- to post-ETSS on a group and an individual level. Linear multiple regression analyses were employed to investigate predictors of cognitive performance.

RESULTS

On average, patients scored significantly lower in 6 of 7 cognitive domains before and after surgery than controls. Impairment proportions were significantly higher among patients (56% before surgery, 63% after surgery) than among controls. Patients showed no change over time in group-level (mean) performance, but 28% of individual patients exhibited cognitive improvement and 28% exhibited cognitive decline after surgery. Hormonal deficiency showed a positive correlation with verbal memory before surgery. Postoperative performances in all cognitive domains were predicted by preoperative performances.

CONCLUSIONS

Cognitive impairment was present before and after EETS in over half of NFPA patients. Individual patients showed diverse postoperative cognitive courses. Monitoring of cognitive functioning in clinical trajectories and further identification of disease-related and psychological predictors of cognition are warranted.

Longitudinal Estimation of the Clinically Significant Change in the Treatment of Major Depression Disorder

Background: Although major depressive disorder is usually treated with antidepressants, only 50-70% of the patients respond to this treatment. This study applied Jacobson and Truax's (1991) methodology (reliable change index, RCI) to a sample of depressive patients being treated with one of two antidepressants to evaluate their functioning and the effect of certain variables such as severity and age. Method: Seventy-three depressive patients medicated with Escitalopram (n = 37) or Duloxetine (n = 36) were assessed using the Hamilton depression rating scale over a 24-week period. Results: They indicate that the RCI stabilizes in an absolute way starting in week 16, and it is not until week 24 that all of the patients become part of the functional population. We found limited statistical significance with respect to the RCI and the external variables. Conclusion: Our study suggests the need to accompany the traditional statistical methodology with some other clinical estimation systems capable of going beyond a simple subtraction between pre and posttreatment values. Hence, it is concluded that RCI estimations could be stronger and more stable than the classical statistical techniques.

Demoralization in Patients With Substance Use and Co-Occurring Psychiatric Disorders

OBJECTIVE

In recent years, treatment of substance use disorder has rekindled emphasis on recovery which, being a gradual process, starts with remoralization. In this study, we examine the level of demoralization throughout the treatment process for patients with comorbid substance dependence and psychiatric disorders.

METHODS

217 patients with co-occurring disorders and 179 community-based individuals participated in this study. Demoralization was measured twice over one month as inpatient treatment happened.

RESULTS

In contrast with the community sample, we found high levels of demoralization in the clinical cohort, with 86% of patients having demoralization scores above threshold. During the first month there was a statistically significant reduction in demoralization scores. However, clinically relevant change appeared limited, with only 3% of patients moving from dysfunctional to functional status in this naturalistic setting without targeted intervention aimed at remoralization.

CONCLUSIONS

Although the level of demoralization is significantly improved during the first month of treatment, patients still remain strongly demoralized. Clinically relevant improvement is limited. It could be worthwhile to set up targeted interventions aimed at remoralization. Furthermore, we advocate for the assessment of demoralization in the clinical setting to monitor patients' treatment outcomes.

Eating behaviors among the participants of an inpatient weight loss treatment

Background and purpose

Eating behaviors play a crucial role in the development and maintenance of excess weight. The aim of the study was to explore the predictors and changes in eating behaviors among overweight and obese patients.

Methods

The sample of the 6-month prospective survey consisted of patients who participated in the inpatient weight loss treatment program in the Lipidological Department of the Szent Imre Hospital (baseline: N=339, 19% men; follow-up: N=175, 16% men). The mean age was 50.2 years (SD=13.47), the mean BMI was 38.6 (SD=7.58) at baseline. Measures: self-reported anthropometric data, Three-Factor Eating Questionnaire Revised 21-Items, CES-D Depression Scale.

Results

According to the results of Multiple Indicators and Multiple Causes analysis, older age predicted greater cognitive restraint (b=0.12, p=0.047). Women were more prone to emotional eating than men (b=0.21, p<0.001). Higher levels of education predicted greater uncontrolled eating (b=0.16, p=0.007) and emotional eating (b=0.12, p=0.039). Depression showed a positive relationship with emotional eating (b=0.19, p=0.001), and mediated the relationship between gender and emotional eating (b=0.04, p=0.009), and BMI and emotional eating (b=0.03, p=0.015). Those whose weight loss was at least 5% showed a greater improvement in the eating behaviors than those whose weight loss was below 5% (cognitive restraint: t(168)=-4.765, p<0.001, uncontrolled eating: t(168)=-2.442, p=0.016, and emotional eating: Z=-2.011, p=0.044).

Conclusion

Results reveal certain determinants of eating behaviors that enhance or obstruct successful long term weight loss and highlight the role of eating behavior changes in weight loss. These mark intervention points for the optimization of results achievable by weight loss treatments.

Comparison of Classical Test Theory and Item Response Theory in Individual Change Assessment

Clinical psychologists are advised to assess clinical and statistical significance when assessing change in individual patients. Individual change assessment can be conducted using either the methodologies of classical test theory (CTT) or item response theory (IRT). Researchers have been optimistic about the possible advantages of using IRT rather than CTT in change assessment. However, little empirical evidence is available to support the alleged superiority of IRT in the context of individual change assessment. In this study, the authors compared the CTT and IRT methods with respect to their Type I error and detection rates. Preliminary results revealed that IRT is indeed superior to CTT in individual change detection, provided that the tests consist of at least 20 items. For shorter tests, however, CTT is generally better at correctly detecting change in individuals. The results and their implications are discussed.

Independent validation of the Pain Management Plan in a multi-disciplinary pain team setting

CONTEXT/BACKGROUND

The Pain Management Plan (PP) is a brief cognitive behavioural therapy (CBT) self-management programme for people living with persistent pain that can be individually facilitated or provided in a group setting. Evidence of PP efficacy has been reported previously by the pain centres involved in its development.

OBJECTIVES

To provide a fully independent evaluation of the PP and compare these with the findings reported by Cole et al.

METHODS

The PP programme was delivered by the County Durham Pain Team (Co. Durham PT) as outlined in training sessions led by Cole et al. Pre- and post-quantitative/patient experience measures were repeated with reliable and clinical significant change determined and compared to the original evaluation.

RESULTS

Of the 69 participants who completed the programme, 33% achieved reliable change and 20% clinical significant change using the Pain Self-Efficacy Questionnaire (PSEQ). Across the Brief Pain Inventory (BPI) interference domains between 11% and 22% of participants achieved clinical significant change. There were high levels of positive patient feedback with 25% of participants scoring 100% satisfaction. The mean participant satisfaction across the population was 88%.

CONCLUSION

The results from this evaluation validate those reported by Cole et al. It demonstrates clinically significant improvement in pain and health functioning and high patient appreciation results. Both evaluations emphasise the potential of this programme as an early intervention delivered within a stratified care pain pathway. This approach could optimise the use of finite resources and improve wider access to pain management.

Test-retest reliability analysis of the Cambridge Neuropsychological Automated Tests for the assessment of dementia in older people living in retirement homes

The validity of the Cambridge Neuropsychological Automated Tests has been widely studied, but their reliability has not. This study aimed to estimate the test-retest reliability of these tests in a sample of 34 older adults, aged 69 to 90 years old, without neuropsychiatric diagnoses and living in retirement homes in the district of Lisbon, Portugal. The battery was administered twice, with a 4-week interval between sessions. The Paired Associates Learning (PAL), Spatial Working Memory (SWM), Rapid Visual Information Processing, and Reaction Time tests revealed measures with high-to-adequate test-retest correlations (.71-.89), although several PAL and SWM measures showed susceptibility to practice effects. Two estimated standardized regression-based methods were found to be more efficient at correcting for practice effects than a method of fixed correction. We also found weak test-retest correlations (.56-.68) for several measures. These results suggest that some, but not all, measures are suitable for cognitive assessment and monitoring in this population.

Comparison of indices of clinically meaningful change in child and adolescent mental health services: difference scores, reliable change, crossing clinical thresholds and 'added value' - an exploration using parent rated scores on the SDQ

BACKGROUND

Establishing what constitutes clinically significant change is important both for reviewing the function of services and for reflecting on individual clinical practice. A range of methods for assessing change exist, but it remains unclear which are best to use and under which circumstances.

METHOD

This paper reviews four indices of change [difference scores (DS), crossing clinical threshold (CCT), reliable change index (RCI) and added value scores (AVS)] drawing on outcome data for 9764 young people from child and adolescent mental health services across England.

RESULTS

Looking at DS, the t-test for time one to time two scores indicated a significant difference between baseline and follow up scores, with a standardised effect size of d = 0.40. AVS analysis resulted in a smaller effect size of 0.12. Analysis of those crossing the clinical threshold showed 21.2% of cases were classified as recovered, while 5.5% were classified as deteriorated. RCI identified 16.5% of cases as showing reliable improvement and 2.3% of cases as showing reliable deterioration. Across RCI and CCT 80.5% of the pairings were exact (i.e., identified in the same category using each method).

CONCLUSIONS

Findings indicate that the level of agreement across approaches is at least moderate; however, the estimated extent of change varied to some extent based on the index used. Each index may be appropriate for different contexts: CCT and RCI may be best suited to use for individual case review; whereas DS and AVS may be more appropriate for case-mix adjusted national reporting.

Personality Change at Mid-Life is Associated with Changes in Self-Rated Health: Evidence from the Hawaii Personality and Health Cohort

Personality traits change across the lifespan, and trait change, in addition to trait level, may be related to health. Longitudinal data from the Hawaii Personality and Health Cohort were used to investigate associations between changes in traits and self-rated health (SRH). Participants (N = 733, M_age = 44.4) completed measures of the Big Five personality traits and SRH twice approximately 3 years apart. Personality trait changes were associated with SRH change. Additionally, increases on Agreeableness, Conscientiousness, and Openness, and decreases on Neuroticism, predicted increases in SRH, even when controlling for gender and education. Relating correlated trait change at mid-life, when traits reach peak stability, to a consequential health outcome such as SRH change, demonstrates the value of treating both traits and health indicators as dynamic variables.