Are Reliable Change (RC) calculations appropriate for determining the extent of cognitive change in concussed athletes?

An MDS 3.0 Distressed Behavior in Dementia Indicator (DBDI): A Clinical Tool to Capture Change

BACKGROUND/OBJECTIVES

Persons with dementia frequently demonstrate distress behaviors in dementia (DBD), associated with poorer outcomes. This study aimed to create a measure of DBD from routinely administered Minimum Data Set (MDS 3.0) behavior section items that demonstrated sensitivity to change, for evaluation of intervention efforts for VA Community Living Center (CLCs) residents exhibiting DBD.

SETTING

72 VA nursing home settings, or Community Living Centers (CLCs).

PARTICIPANTS

CLC residents with DBD (n = 302) were enrolled in an interdisciplinary behavioral intervention between 2013 and 2017.

DESIGN

A factor analysis of MDS behavior section items from assessments closest to baseline was conducted. Internal consistency, hypothesized associations between MDS factors and clinical measures, and sensitivity to detect change over time was explored.

MEASUREMENTS

Residents were assessed at baseline and post-intervention using the MDS behavior section items and a validated clinical measure of DBD.

RESULTS

The Distress Behavior in Dementia Indicator (DBDI) was created as a consistent factor with internal consistency, and was related to a validated measure as predicted at baseline and post-intervention. Sensitivity to change was demonstrated by using change score correlations (r = 0.40-0.50), effect size (d = 0.63), and reliable change indices.

CONCLUSION

The DBDI is recommended for routine use in CLCs to evaluate impact of intervention effectiveness and provide quality improvement feedback.

Comparison of livestock-associated and community-associated Staphylococcus aureus pathogenicity in a mouse model of skin and soft tissue infection

Industrial hog operation (IHO) workers are at increased risk of carrying Staphylococcus aureus in their nares, particularly strains that are livestock-associated (LA) and multidrug-resistant. The pathogenicity of LA-S. aureus strains remains unclear, with some prior studies suggesting reduced transmission and virulence in humans compared to community-associated methicillin-resistant (CA-MRSA) S. aureus. The objective of this study was to determine the degree to which LA-S. aureus strains contracted by IHO workers cause disease relative to a representative CA-MRSA strain in a mouse model of skin and soft tissue infection (SSTI). Mice infected with CC398 LA-S. aureus strains (IHW398-1 and IHW398-2) developed larger lesion sizes with higher bacterial burden than mice infected with CA-MRSA (SF8300) (p < 0.05). The greatest lesion size and bacterial burden was seen with a CC398 strain that produced a recurrent SSTI in an IHO worker. The LA-S. aureus infected mice had decreased IL-1β protein levels compared with CA-MRSA-infected mice (p < 0.05), suggesting a suboptimal host response to LA-S. aureus SSTIs. WGSA revealed heterogeneity in virulence factor and antimicrobial resistance genes carried by LA-S. aureus and CA-MRSA strains. The observed pathogenicity suggest that more attention should be placed on preventing the spread of LA-S. aureus into human populations.

Evaluating the Statistical Significance of Health-Related Quality-Of-Life Change in Individual Patients

Assessing individual change is feasible and potentially useful in clinical practice. This article provides an overview of the evaluation of statistically significant change in health-related quality of life (HRQOL) for individual patients. We review the standard error of measurement, standard error of prediction, and reliable change indices using a sample of 54 patients receiving care at the UCLA Center for East-West Medicine. The largest amount of change necessary for statistical significance was found for the reliable change index and the smallest change was needed for the standard error of measurement. The amount of change required for statistical significance was intermediate for the standard error of prediction. The median kappa for classifying change (declined, stayed the same, improved) by different indices was .82, indicating a high level of agreement. Future research is needed to determine if one index is most appropriate for evaluating the significance of individual change.

Some Thoughts about the Suitability of the Reliable Change Index (RCI) for Analysis of Ordinal Scale Data

The reliable change index (RCI) was introduced approximately 30 decades ago in order to provide an empirical, statistically grounded technique for determining whether improvement after a therapeutic intervention was real or due to measurement error. Since the definitions of the properties and limitations of scales of measurement described by Stevens in 1947, there has been vigorous controversy about whether it is permissible to analyse ordinal data with parametric statistics. Specifically, are parameters and statistics such as means and standard deviations meaningful in the context of ordinal data? These are important concerns because many of the scales used to measure outcomes in behavioural research and clinical settings yield ordinal-scale measures. Given that the standard deviation is used in the computation of the RCI, the question as to whether or not the RCI is reliable when used with ordinal-scale data is explored. Data from the SPRS-2 was used to calculate minimum reliable difference criteria in terms of both (ordinal) Total Raw Scores (MRDRS) and logit scores (MRDLS) derived from Rasch analysis. Test–retest differences across the Total Raw Score range were evaluated using each criterion. At both extremes of the range, small changes in Total Raw Score not deemed to be reliable according to the MRDRScriterion were classified as reliable according to the MRDLScriterion. Conversely, test–retest changes in the centre of the range deemed to be reliable according to the MRDRScriterion were classified as unreliable according to the MRDLScriterion. It is suggested that while MRDRScan determinenumericallyreliable differences, MRDLScan determine reliable differences that are meaningful in terms of the underlying construct being measured.

Meaningful Change in Cognition in Multiple Sclerosis: Method Matters

Objective:

To determine if different methods of evaluating cognitive change over time yield measurably different outcomes.

Methods:

Twelve cognitively impaired patients with clinically definite Multiple sclerosis (10 relapsing-remitting, 2 secondary progressive) underwent neuropsychological testing (baseline, 6, 12 months). Data was analysed using: t-tests evaluating group differences on individual tests, group differences in composite scores, reliable change analyses at the level of the individual, and comparisons regarding number of tests failed at each time point.

Results:

Group t-tests on individual tests yielded no change. When tests were grouped according to theoretical constructs, analyses revealed change in processing speed. Reliable change estimates revealed that 16% of the sample deteriorated. When change was measured with respect to the number of domains affected at each time point, 58% of the sample deteriorated on at least one subtest.

Conclusions:

Methodology has a significant impact on interpretation of longitudinal data. In the same group of subjects, traditional group analyses documented no change in individual test scores or change on a single composite score. Analyses of individual results documented change from 16 to 58% of the sample. Advantages and disadvantages of each method were discussed. Findings have implications for interpretation of longitudinal studies.

Effect of sport-related concussion on clinically measured simple reaction time

BACKGROUND

Reaction time (RT) is a valuable component of the sport concussion assessment battery. RT is typically measured using computers running specialised software, which limits its applicability in some athletic settings and populations. To address this, we developed a simple clinical test of RT (RTclin) that involves grasping a falling measuring stick.

PURPOSE

To determine the effect of concussion on RTclin and its sensitivity and specificity for concussion.

MATERIALS AND METHODS

Concussed athletes (n=28) and non-concussed control team-mates (n=28) completed RTclin assessments at baseline and within 48 h of injury. Repeated measures analysis of variance compared mean baseline and follow-up RTclin values between groups. Sensitivity and specificity were calculated over a range of reliable change confidence levels.

RESULTS

RTclin differed significantly between groups (p<0.001): there was significant prolongation from baseline to postinjury in the concussed group (p=0.003), with a trend towards improvement in the control group (p=0.058). Sensitivity and specificity were maximised when a critical change value of 0 ms was applied (ie, any increase in RTclin from baseline was interpreted as abnormal), which corresponded to a sensitivity of 75%, specificity of 68% and a 65% reliable change confidence level.

CONCLUSIONS

RTclin appears sensitive to the effects of concussion and distinguished concussed and non-concussed athletes with similar sensitivity and specificity to other commonly used concussion assessment tools. Given its simplicity, low cost and minimal time requirement, RTclin should be considered a viable component of the sports medicine provider's multifaceted concussion assessment battery.

How Do You Know Whether Your Patient Is Getting Better (or Worse)? A User's Guide

It is important to know when improvement or deterioration in cognitive function occurs. Until fairly recently neuropsychologists have made these judgments clinically, with little resort to empirical methods. In addition to the issue of whether a change in performance is reliable, there is also the consideration of whether it is clinically significant. This article briefly discusses these concepts, reviews the most common methods for determining reliability in change in test scores, considers their use in the broader clinical context, and illustrates their application with reference to an actual database.

A computerized test battery sensitive to mild and severe brain injury

OBJECTIVE

Computerized neurocognitive testing (CNT) appears to be suited to measure relatively mild degrees of neurocognitive impairment in circumstances where speed, efficiency, and low cost are important. Computerized tests are used in the evaluation and management of patients who have had mild brain injuries; the objective is to determine if computerized testing is equally reliable and valid in the evaluation of patients who have had more severe brain injuries.

DESIGN

A cross-sectional, naturalistic study of brain injury patients compared with normal controls.

SETTING

An outpatient neuropsychiatry clinic.

PARTICIPANTS

141 patients, aged 18-65 years, who had sustained traumatic brain injuries (TBIs): 13 patients with postconcussion syndrome; 15 who had recovered from mild brain injuries; 85 patients who had had severe brain injuries, but who had recovered, and were living independently; and 28 severe brain injury patients who were unable to live without assistance; compared with 145 normal controls.

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURES

The CNS Vital Signs (CNS VS) battery is a PC-based system that includes tests of verbal and visual memory, psychomotor speed, complex attention, reaction time, and cognitive flexibility.

RESULTS

Performance on the CNS VS battery was related to severity of brain injury and degree of recovery. Tests of psychomotor speed and cognitive flexibility were the most relevant to TBI status. Patients who had recovered from mild brain injuries scored almost as well as normal controls. The Neurocognition Index (NCI), a summary score based on performance on all the tests in the battery, was 100 for normal controls and 98 for recovered mild brain injury patients. Postconcussive patients scored 82 on the MCI, and severe brain injury patients scored 66 on the NCI if they were living independently and 47 if they were not.

CONCLUSIONS

Computerized tests like CNS VS allow clinicians the advantage of precise neurocognitive measurement in the service of diagnosis and appropriate treatment. CNTs are never going to replace the flexibility or comprehensiveness of conventional neuropsychological testing, but they have a role to play in circumstances where a full test battery is not feasible, such as screening and serial assessment.

Reliable change of the sensory organization test

OBJECTIVE

To establish the sensitivity and specificity of the NeuroCom Sensory Organization Test (SOT) and provide practitioners with cut-scores for clinical decision making using estimates of reliable change.

DESIGN

Retrospective cohort study.

SETTING

Research laboratory.

PATIENTS

Healthy (n = 66) and concussed (n = 63) young adult participants.

INTERVENTIONS

Postural control assessments on the NeuroCom SOT were completed twice (baseline and follow-up) for both groups. Postconcussion assessments were administered within 24 hours of injury diagnosis.

MAIN OUTCOME MEASUREMENTS

The reliable change technique was used to calculated cut-scores for each SOT variable (composite balance; somatosensory, visual, and vestibular ratios) at the 95%, 90%, 85%, 80%, 75%, and 70% confidence interval levels.

RESULTS

When cut-scores were applied to the post-concussion evaluations, sensitivity and specificity varied with SOT variable and confidence interval. An evaluation for change on one or more SOT variable resulted in the highest combined sensitivity (57%) and specificity (80%) at the 75% confidence interval.

CONCLUSIONS

Use of reliable change scores to detect significant changes in performance on the SOT resulted in decreased sensitivity and improved specificity compared to a previous report. These findings indicate that some concussed athletes may not show large changes in postconcussion postural control and this postural control evaluation should not be used in exclusion of other assessment techniques. The postural control assessment should be combined with other evaluative measures to gain the highest sensitivity to concussive injuries.

Evaluation of a treatment programme for alcohol-related aggression

BACKGROUND

The development of effective treatments for alcohol-related aggression and violence is important in binge drinking cultures, as in parts of the UK.Aim The aim was to evaluate the progress and experience of 10 participants in Control of Violence for Angry Impulsive Drinkers (COVAID) using a single case methodology.

METHOD

Participants completed 10 individual weekly sessions with trained facilitators following the COVAID manual. Change scores on psychometric questionnaires were examined by calculating clinical significance and reliability of change. Self-reports of alcohol consumption and aggression were examined. Follow-up data on convictions were collected. Participants were asked their opinions about COVAID.

RESULTS

Scores on the Alcohol-Related Aggression Questionnaire (ARAQ) improved for nine participants; change was both clinically significant and reliable in five cases. Nine participants improved on the Controlled Drinking Self-Efficacy Scale (CDSES), with seven showing clinically significant improvement. Six participants reported a reduction in alcohol consumption from the first to the second half of the programme. At a mean of 29 weeks post-treatment, none of the participants had been reconvicted for a violent offence. Participants reported finding COVAID useful and interesting.

CONCLUSION

Overall, our findings support the possibility that COVAID may assist in reducing alcohol-related violence and violent offending.

Neurocognitive performance of concussed athletes when symptom free

CONTEXT

Concussed athletes may underreport concussion-related symptoms in order to expedite return to play. Whether neurocognitive impairments persist once concussion-related symptoms resolve has yet to be determined. Reliance on athlete-reported, postconcussion symptoms when making return-to-play decisions may expose athletes to subsequent injury if complete recovery has not occurred.

OBJECTIVE

To evaluate the presence of neurocognitive decrements in concussed athletes no longer reporting concussion-related symptoms.

DESIGN

Within-groups design.

SETTING

University research laboratory.

PATIENTS OR OTHER PARTICIPANTS

Twenty-one National Collegiate Athletic Association Division I collegiate athletes (16 men, 5 women). Age = 19.81 +/- 1.25 years, height = 180.95 +/- 10.62 cm, mass = 93.66 +/- 27.60 kg, and previous concussions = 1.76 +/- 2.02.

MAIN OUTCOME MEASURE(S)

The ImPACT concussion assessment test was administered to concussed athletes at baseline, when symptomatic (within 72 hours of injury), and when asymptomatic. Index scores of verbal memory, visual memory, visual-motor speed, reaction time, and concussion-related symptoms were recorded at each session. The Symptom Assessment Scale was administered daily after injury to establish when the athlete became asymptomatic.

RESULTS

When assessed within 72 hours of concussion, 81% of the athletes showed deficits on at least 1 ImPACT variable. At the asymptomatic time point, 38% of the concussed athletes continued to demonstrate neurocognitive impairment on at least 1 ImPACT variable.

CONCLUSIONS

Neurocognitive decrements may persist when athletes no longer report concussion-related symptoms. The exclusive use of symptom reports in making a return-to-play decision is not advised. A multifaceted approach to concussion assessment that includes evaluation of a myriad of functions is warranted.

Reliable change assessment with practice effects in sport concussion research: a comment on Hinton-Bayre

In his comments on our previous article, Hinton-Bayre advocates the use of the regression based approach in most cases of determining reliable change. This article comments on Hinton-Bayre's argument, discusses cases where the regression method might not be the preferred method, and presents adjustments that make the method more generally preferable.

Repeated assessment of cognition in children and the measurement of performance change

There is limited understanding of the problems associated with repeated neuropsychological assessment in children, including the statistics used to guide decisions about cognitive change. This study investigated the utility of a computerized assessment battery that was specifically designed for the repeated assessment of cognitive function in children. Eighty-seven participants aged 8 to 12 years completed the battery four times within a 2-hour testing session. The results support the application of this assessment battery for measuring cognitive change in children. A novel method for calculating measurement error is employed, and its use in the detection of cognitive change in individual children is discussed. An estimate of the measurement error within each of the tests is provided, and recommendations are made regarding the application of this assessment battery for measuring cognitive change in children.

Management of sport-related concussion in young athletes

Sport-related head injuries are a common clinical problem. Most head injuries in young athletes are mild traumatic brain injuries or concussions. The highest number of sport-related concussions has been reported in American football. In addition to the well described physical and psychosocial growth, there is ongoing neurocognitive development of the brain during childhood and through adolescence. This developmental process has direct implications in the assessment and management of head injuries in young athletes. Research on the management and long-term outcome following brain injuries in young athletes is limited. Traditionally, the assessment of concussion has been based on clinical history and physical and neurological examination. Increasingly, neuropsychological testing, especially computerised testing, is providing objective measures for the initial assessment and follow-up of young athletes following brain injuries. Numerous guidelines have been published for grading and return to play criteria following concussion; however, none of these have been prospectively validated by research and none are specifically applicable to children and adolescents.

Reliable change assessment in sport concussion research: a comment on the proposal and reviews of Collie et al

The proposal of Collie et al for determining reliable change in sports medicine and their review of other current procedures did not address mathematical or theoretical aspects. This article attempts to fill this lacuna, in order to examine the validity of their proposal and the suitability of their review. The conclusions drawn are that Collie et al presented their method too carelessly and their review of other methods leaves serious drawbacks and mistakes unnoticed. A scheme is here provided showing which of the current methods can be used in which situation.