Reliable change assessment in sport concussion research: a comment on the proposal and reviews of Collie et al

Sport as Laboratory: Lessons Learned From Baseline and Postconcussion Assessment Research

Knowledge and awareness of sport concussion has been forwarded by research modeled on the neuropsychological testing paradigm associated with Barth’s “sport as laboratory” assessment model. The purpose of this paper is to elucidate lessons learned from that research. Key considerations for planning and implementing large-scale studies of concussion in sport while making adequate provision for the clinical needs of concussed athletes are reviewed. Toward that end, logistical, methodological, and ethical considerations are discussed within the context of research conducted in a university setting. Topics addressed include culture of sport and risk; research planning and design; communication with strategic partners; defining injury; choosing a test battery; data management, outcomes, and analyses; dissemination of results; and finally, clinical and ethical implications that may arise during the research enterprise. The paper concludes with a summary of the main lessons learned and directions for future research.

Choice of reliable change model can alter decisions regarding neuropsychological impairment after sports-related concussion

OBJECTIVE

Impaired neuropsychological test performance after concussion has been used to guide restraint from play, in particular using reliable change indices (RCI). It remains unclear which of the RCI is most appropriate.

DESIGN

Athletes were assessed prospectively before and after cerebral concussion and compared with control athletes.

SETTING

Athletes were assessed in a clinical office environment after referral from a Sports Physician.

PARTICIPANTS

One hundred ninety-four Australian rugby league athletes were assessed preseason (time 1).

INTERVENTIONS

Twenty-seven concussed athletes were assessed 2 days after trauma (time 2) and compared with 26 distribution-matched volunteer uninjured controls.

MAIN OUTCOME MEASURES

Cognitive performance was assessed on 5 neuropsychological measures of speed of information processing, psychomotor speed, and response inhibition. Four previously reported RCI models used in sports concussion were contrasted, as described by Barr and McCrea (2001) and Maassen et al (2006).

RESULTS

Reliable change index models were marginally comparable in classifying the control sample. In the concussed sample, no one model seemed to be consistently more or less sensitive. Moreover, the same model could be most sensitive for one individual and least sensitive for another, even on the same test.

CONCLUSIONS

Reliable change index models can yield different outcomes regarding whether an athlete has experienced cognitive impairment after concussion. Reliable change index model sensitivity to impairment depends on multiple test and situational factors, including test-retest reliability, differences in test and retest variances, and the individual's relative position on initial testing. In the absence of consensus, the clinician must use highly reliable measures with suitably matched controls if using an RCI.

How Do You Know Whether Your Patient Is Getting Better (or Worse)? A User's Guide

It is important to know when improvement or deterioration in cognitive function occurs. Until fairly recently neuropsychologists have made these judgments clinically, with little resort to empirical methods. In addition to the issue of whether a change in performance is reliable, there is also the consideration of whether it is clinically significant. This article briefly discusses these concepts, reviews the most common methods for determining reliability in change in test scores, considers their use in the broader clinical context, and illustrates their application with reference to an actual database.

Assessing reliable change using the repeatable battery for the assessment of neuropsychological status (RBANS) for patients with Parkinson's Disease undergoing deep brain stimulation (DBS) surgery

Parkinson's disease (PD) is progressive neurological disease characterized by resting tremor, rigidity, akinesia, postural instability and cognitive changes. The symptoms of PD are debilitating and often become unsatisfactorily treated by medication. Deep brain stimulation (DBS) is an effective treatment to significantly reduce the cardinal motor symptoms of PD. However, the neuropsychological effects of this treatment are less clear. This study examined pre- to post-DBS scores on the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) using Reliable Change Indices (RCIs) derived from 20 patients with PD who were medically managed, and then compared to 20 patients with PD treated with DBS and medication. When using group statistical analyses and false discovery rate correction, no significant differences between or within groups were evident at baseline or at follow-up. However, when using the RCIs more patients in the DBS group exhibited reliable change in RBANS scores than did the Med Tx group. Although preliminary, these RCIs provide clinicians and researchers a foundational tool for assessing the effects of interventions (e.g., DBS) independent of the effects of PD and measurement error when using the RBANS.

The assessment of changes in cognitive functioning: age-, education-, and gender-specific reliable change indices for older adults tested on the CERAD-NP battery: results of the German Study on Ageing, Cognition, and Dementia in Primary Care Patients (AgeCoDe)

OBJECTIVES

The Consortium to Establish a Registry for Alzheimer's Disease-Neuropsychological (CERAD-NP) battery represents a commonly used neuropsychological instrument to measure cognitive functioning in the elderly. This study provides normative data for changes in cognitive function that normally occur in cognitively healthy individuals to interpret changes in CERAD-NP test scores over longer time periods.

DESIGN

Longitudinal cohort study with three assessments at 1.5-year intervals over a period of 3 years.

SETTING

: Primary care medical record registry sample.

PARTICIPANTS

As part of the German Study on Ageing, Cognition, and Dementia in Primary Care Patients, a sample of 1,450 cognitively healthy general practitioner patients, age 75 years and older, was assessed.

MEASUREMENTS

Age-, education-, and gender-specific Reliable Change Indices (RCIs) were computed for a 90% confidence interval for selected subtests of the CERAD-NP battery.

RESULTS

Across different age, education, and gender subgroups, changes from at least six to nine points in Verbal Fluency, four to eight points in Word List Memory, two to four points in Word List Recall, and one to four points in Word List Recognition indicated significant (i.e. reliable) changes in CERAD-NP test scores at the 90% confidence level. Furthermore, the calculation of RCIs for individual patients is demonstrated.

CONCLUSIONS

Smaller changes in CERAD-NP test scores can be interpreted with only high uncertainty because of probable measurement error, practice effects, and normal age-related cognitive decline. This study, for the first time, provides age-, education-, and gender-specific CERAD-NP reference values on the basis of RCI methods for the interpretation of cognitive changes in older-age groups.

Reliable change of the sensory organization test

OBJECTIVE

To establish the sensitivity and specificity of the NeuroCom Sensory Organization Test (SOT) and provide practitioners with cut-scores for clinical decision making using estimates of reliable change.

DESIGN

Retrospective cohort study.

SETTING

Research laboratory.

PATIENTS

Healthy (n = 66) and concussed (n = 63) young adult participants.

INTERVENTIONS

Postural control assessments on the NeuroCom SOT were completed twice (baseline and follow-up) for both groups. Postconcussion assessments were administered within 24 hours of injury diagnosis.

MAIN OUTCOME MEASUREMENTS

The reliable change technique was used to calculated cut-scores for each SOT variable (composite balance; somatosensory, visual, and vestibular ratios) at the 95%, 90%, 85%, 80%, 75%, and 70% confidence interval levels.

RESULTS

When cut-scores were applied to the post-concussion evaluations, sensitivity and specificity varied with SOT variable and confidence interval. An evaluation for change on one or more SOT variable resulted in the highest combined sensitivity (57%) and specificity (80%) at the 75% confidence interval.

CONCLUSIONS

Use of reliable change scores to detect significant changes in performance on the SOT resulted in decreased sensitivity and improved specificity compared to a previous report. These findings indicate that some concussed athletes may not show large changes in postconcussion postural control and this postural control evaluation should not be used in exclusion of other assessment techniques. The postural control assessment should be combined with other evaluative measures to gain the highest sensitivity to concussive injuries.

Measuring cognitive change in older adults: reliable change indices for the Mini-Mental State Examination

BACKGROUND

In clinical and research settings, the Mini-Mental State Examination (MMSE) is commonly used to measure cognitive change over time. The interpretation of changes in MMSE is often difficult. They do not necessarily result from true clinical change. Their interpretation requires comparison with normative data for change. However, MMSE change norms are lacking for long intervals.

OBJECTIVE

To examine what is a reliable change in MMSE for long follow-up periods commonly used in clinic. To provide normative data for change.

METHODS

A sample of 119 cognitively normal individuals, aged 75 years and over, who participated in the Leipzig Longitudinal Study of the Aged (LEILA 75+). All participants were tested six times at 1.5 year intervals with the MMSE over a mean period of 7.1 years. Reliable change indices were computed for a common confidence interval (90%).

RESULTS

In repeated assessments with 1.5 year intervals, a change in MMSE of at least 2-4 points indicated a reliable change at the 90% confidence level.

CONCLUSION

Small changes in MMSE can be interpreted only with great uncertainty. They have a reasonable probability of being caused by measurement error, regression to the mean or practice.

Reliable change on the Dementia Rating Scale

A central role for neuropsychological evaluations is the measurement of change in cognitive functioning over time. However, change scores obtained from repeated neuropsychological assessments may be affected by normal variability because of measurement error and practice effects caused by repeated measurements. The current study uses reliable change estimates to establish normative rates of change on the Dementia Rating Scale from baseline to first follow-up testing among 1080 cognitively normal adults aged 65 and older. Results showed that a 6-point decline by European Americans or a 9-point decline by African American adults within a 9-15 month test-retest interval represents reliable change. Within a 16-24-month test-retest interval, a 7-point decline among European Americans or an 8-point decline among African American adults represents reliable change. In addition, preliminary cross-validation was performed in a clinical comparison sample of another 22 older adults. The findings are discussed in the context of potential clinical and research applications.

Measuring cognitive change in older adults: reliable change indices for the SIDAM

BACKGROUND

In clinical settings, neuropsychological tests and screening instruments are often used to measure cognitive change over time. However, the interpretation of changes in test scores is often difficult. For most instruments there is no information how much change occurs normally in cognitively healthy individuals.

AIM

To examine what is a reliable change for a widely used screening instrument for cognitive impairment and dementia.

METHODS

A Sample of 119 cognitively normal elderly individuals aged 75 and over participated in the Leipzig Longitudinal Study of the Aged (LEILA 75+). All participants have been tested six times at 1.5 year intervals with the test part of the SIDAM over a mean period of 7.1 years. Reliable change indices (RCI) were computed for a common confidence interval (90%). It is demonstrated how to compute RCI for individual patients.

RESULTS

In repeated assessments with 1.5 year intervals, a change in SIDAM score of at least 4 to 7 points (dependent on which of the six assessments were compared) indicates a reliable change at the 90% confidence level. Smaller changes can be interpreted only with high uncertainty.

CONCLUSION

The interpretation of changes in test scores in older adults should account for potential practice effect, normal decline and regression to the mean.

Reliable change assessment with practice effects in sport concussion research: a comment on Hinton-Bayre

In his comments on our previous article, Hinton-Bayre advocates the use of the regression based approach in most cases of determining reliable change. This article comments on Hinton-Bayre's argument, discusses cases where the regression method might not be the preferred method, and presents adjustments that make the method more generally preferable.