False-Positive Rates of Reliable Change Indices for Concussion Test Batteries: A Monte Carlo Simulation

Psychometric Properties of Computerized Cognitive Tools and Standard Neuropsychological Tests Used to Assess Sport Concussion: A Systematic Review

Athletic programs are more frequently turning to computerized cognitive tools in order to increase efficiencies in concussion assessment. However, assessment using a traditional neuropsychological test battery may provide a more comprehensive and individualized evaluation. Our goal was to inform sport clinicians of the best practices for concussion assessment through a systematic literature review describing the psychometric properties of standard neuropsychological tests and computerized tools. We conducted our search in relevant databases including Ovid Medline, Web of Science, PsycINFO, and Scopus. Journal articles were included if they evaluated psychometric properties (e.g., reliability, sensitivity) of a cognitive assessment within pure athlete samples (up to 30 days post-injury). Searches yielded 4,758 unique results. Ultimately, 103 articles met inclusion criteria, all of which focused on adolescent or young adult participants. Test-retest reliability estimates ranged from .14 to .93 for computerized tools and .02 to .95 for standard neuropsychological tests, with strongest correlations on processing speed tasks for both modalities, although processing speed tasks were most susceptible to practice effects. Reliability was improved with a 2-factor model (processing speed and memory) and by aggregating multiple baseline exams, yet remained below acceptable limits for some studies. Sensitivity to decreased cognitive performance within 72 h of injury ranged from 45%-93% for computerized tools and 18%-80% for standard neuropsychological test batteries. The method for classifying cognitive decline (normative comparison, reliable change indices, regression-based methods) affected sensitivity estimates. Combining computerized tools and standard neuropsychological tests with the strongest psychometric performance provides the greatest value in clinical assessment. To this end, future studies should evaluate the efficacy of hybrid test batteries comprised of top-performing measures from both modalities.

Cognitive Outcome 1 Year After Mild Traumatic Brain Injury: Results From the TRACK-TBI Study

BACKGROUND AND OBJECTIVES

The objectives of this study were to develop and establish concurrent validity of a clinically relevant definition of poor cognitive outcome 1 year after mild traumatic brain injury (mTBI), to compare baseline characteristics across cognitive outcome groups, and to determine whether poor 1-year cognitive outcome can be predicted by routinely available baseline clinical variables.

METHODS

Prospective cohort study included 656 participants ≥17 years of age presenting to level 1 trauma centers within 24 hours of mTBI (Glasgow Coma Scale score 13-15) and 156 demographically similar healthy controls enrolled in the Transforming Research and Clinical Knowledge in TBI (TRACK-TBI) study. Poor 1-year cognitive outcome was defined as cognitive impairment (below the ninth percentile of normative data on ≥2 cognitive tests), cognitive decline (change score [1-year score minus best 2-week or 6-month score] exceeding the 90% reliable change index on ≥2 cognitive tests), or both. Associations of poor 1-year cognitive outcome with 1-year neurobehavioral outcomes were performed to establish concurrent validity. Baseline characteristics were compared across cognitive outcome groups, and backward elimination logistic regression was used to build a prediction model.

RESULTS

Mean age of participants with mTBI was 40.2 years; 36.6% were female; 76.6% were White. Poor 1-year cognitive outcome was associated with worse 1-year functional outcome, more neurobehavioral symptoms, greater psychological distress, and lower satisfaction with life (all p < 0.05), establishing concurrent validity. At 1 year, 13.5% of participants with mTBI had a poor cognitive outcome vs 4.5% of controls (p = 0.003). In univariable analyses, poor 1-year cognitive outcome was associated with non-White race, lower education, lower income, lack of health insurance, hyperglycemia, preinjury depression, and greater injury severity (all p < 0.05). The final multivariable prediction model included education, health insurance, preinjury depression, hyperglycemia, and Rotterdam CT score ≥3 and achieved an area under the curve of 0.69 (95% CI 0.62-0.75) for the prediction of a poor 1-year cognitive outcome, with each variable associated with >2-fold increased odds of poor 1-year cognitive outcome.

DISCUSSION

Poor 1-year cognitive outcome is common, affecting 13.5% of patients with mTBI vs 4.5% of controls. These results highlight the need for better understanding of mechanisms underlying poor cognitive outcome after mTBI to inform interventions to optimize cognitive recovery.

Utility of VOMS, SCAT3, and ImPACT Baseline Evaluations for Acute Concussion Identification in Collegiate Athletes: Findings From the NCAA-DoD Concussion Assessment, Research and Education (CARE) Consortium

BACKGROUND

The Vestibular/Ocular-Motor Screening (VOMS) is a valuable component of acute (<72 hours) sports-related concussion (SRC) assessments and is increasingly used with the Immediate Post-concussion Assessment and Cognitive Testing (ImPACT) instrument and the third edition of the Sport Concussion Assessment Tool (SCAT3). Research has suggested that VOMS acute postinjury scores are useful in identifying acute concussion. However, the utility of preseason baseline measurements to improve diagnostic accuracy remains ambiguous. To this end, there is a need to determine how reliable VOMS baseline assessments are across years and whether incorporating individuals' baseline performance improves diagnostic yield for acute concussions.

PURPOSE

To analyze VOMS, SCAT3, and ImPACT to evaluate the test-retest reliability of consecutive-year preseason baseline assessments to directly compare the diagnostic utility of these tools when incorporating baseline assessments versus using postinjury data alone to identify acute SRC.

STUDY DESIGN

Cohort study (diagnosis); Level of evidence, 2.

METHODS

Preseason and postinjury VOMS, SCAT3, ImPACT Post-Concussion Symptom Scale (PCSS), and ImPACT composite scores were analyzed for 3958 preseason (47.7% female) and 496 acute (≤48 hours) SRC (37.5% female) collegiate athlete evaluations in the National Collegiate Athletic Association-Department of Defense Concussion Assessment Research and Education Consortium. Descriptive statistics, Kolmogorov-Smirnov significance, and Cohen d effect size were calculated. Consecutive-year baseline reliability was evaluated for a subset of 447 athlete encounters using Pearson r, Cohen κ, Cohen d, and 2-way mixed intraclass correlation coefficients (ICCs). Wilcoxon signed rank tests were used to determine the statistical significance between population performances, and the 90% reliable change index (RCI) was calculated from the test-retest results. Preseason to postinjury change scores were then calculated from each tool's RCI. Finally, receiver operating characteristic (ROC) curve analyses were conducted, and DeLong method was used to compare the area under the curve (AUC) of raw postinjury scores versus change scores from preseason baseline assessments. Potential effects of sex, medical history (learning disorders or attention-deficit/hyperactivity disorder), and outlier data were also explored.

RESULTS

Effect sizes were large, and overall predictive utilities were clinically useful for postinjury VOMS Total (d = 2.44; AUC = 0.85), the SCAT3 Symptom Evaluation total severity score (d = 1.74; AUC = 0.82), and the ImPACT PCSS total severity score (d = 1.67; AUC = 0.80). Comparatively, effect sizes were small and predictive utilities were poor for Standardized Assessment of Concussion (SAC), modified Balance Error Scoring System (mBESS), and all ImPACT composites (d = 0.11-0.46; AUC = 0.48-0.59). Preseason baseline test-retest reliability was poor to moderate (r = 0.23-0.52; κ = 0.32-0.36; ICC = 0.36-0.68) for all assessments except ImPACT Visual Motion Sensitivity (r = 0.73; ICC = 0.85). Incorporating baseline scores for VOMS Total, SCAT3 (Symptom Evaluation, SAC, mBESS), ImPACT PCSS, or ImPACT composites did not significantly improve AUCs.

CONCLUSION

VOMS Total and symptom severity (SCAT3, PCSS) total scores had large effect sizes and clinically useful AUCs for identifying acute concussion. However, all tools demonstrated high within-patient test-retest variability, resulting in poor reliability. The findings in this sample of collegiate athletes suggest that incorporating baseline assessments does not significantly increase diagnostic yield for acute concussion.

Sensitivity and Specificity of a Multimodal Approach for Concussion Assessment in Youth Athletes

CONTEXT

Current international consensus endorses a multimodal approach to concussion assessment. However, the psychometric evaluation of clinical measures used to identify postconcussion performance deficits once an athlete is asymptomatic remains limited, particularly in the pediatric population.

OBJECTIVE

To describe and compare the sensitivity and specificity of a multimodal assessment battery (balance, cognition, and upper and lower body strength) versus individual clinical measures at discriminating between concussed youth athletes and noninjured controls when asymptomatic.

DESIGN

Prospective cohort study.

SETTING

Hospital laboratory setting.

PARTICIPANTS

A total of 32 youth athletes with a concussion and 32 matched (age and sex) noninjured control participants aged 10-18 years.

INTERVENTION(S)

Participants were administered preinjury (baseline) assessments of cognition (Immediate Post-Concussion Assessment and Cognitive Testing [ImPACT]), balance (BioSway), and upper and lower body strength (grip strength and standing long jump). Assessments were readministered when concussed participants reported symptom resolution (asymptomatic time point). Noninjured control participants were reassessed using the same time interval as their concussion matched pair. Sensitivity and specificity were calculated using standardized regression-based methods and receiver operating characteristic curves.

MAIN OUTCOME MEASURES

Outcome measures included baseline and postinjury ImPACT, BioSway, grip strength, and standing long jump scores.

RESULTS

When asymptomatic, declines in performance on each individual clinical measure were seen in 3% to 22% of the concussion group (sensitivity = 3%-22%) compared with 3% to 13% of the noninjured control group (specificity = 87%-97%) (90% confidence interval). The multimodal battery of all combined clinical measures yielded a sensitivity of 41% and a specificity of 77% (90% confidence interval). Based on discriminative analyses, the multimodal approach was statistically superior compared with an individual measures approach for balance and upper and lower body strength, but not for cognition.

CONCLUSIONS

Results provide a foundation for understanding which domains of assessment (cognition, balance, and strength) may be sensitive and specific to deficits once symptoms resolve in youth athletes. More work is needed prior to clinical implementation of a preinjury (baseline) to postinjury multimodal approach to assessment following concussion in youth athletes.

Neurocognitive assessment on a tablet device: Test-retest reliability and practice effects of ANAM Mobile

New technology has allowed for the transition of computerized neurocognitive assessments to increasingly user-friendly mobile platforms. While this increased portability facilitates neurocognitive assessment in a wider variety of settings, this transition necessitates further examination of the psychometric properties of tests that have been previously validated on alternate platforms. The present study evaluated the test-retest reliability and practice effects for a new version of the Automated Neuropsychological Assessment Metrics (ANAM), ANAM Mobile, designed to be administered on a tablet computer. A total of 108 undergraduate students completed ANAM Mobile and returned after one week for repeated testing. Observed test-retest reliabilities were consistent with previously established estimates across similar time intervals and ranged from .55 (Simple Response Time) to .87 (Matching to Sample). Modest practice effects were observed. Base rates of reliable decline were low and suggested that declines on two or more tests are uncommon among healthy college students. The present study demonstrates that ANAM Mobile subtests have good-to-excellent reliability across a 7-day retest interval with minimal practice effects. Future research should explore within-subject reliability across repeated ANAM administrations on a tablet device and further examine retest reliabilities over varying time courses and in more diverse samples.

A systematic review of acute concussion assessment selection in research

Background: Several organisations developed guidelines and assessments for aiding in the diagnosis of concussions. The growing number of concussion assessments increases the difficulty for researchers and clinicians to determine the best method of concussion diagnosis. Purpose: To systematically review the current assessments selected for acute sports-related concussion. Methods: Academic Search Premier, CINHAL, MEDLINE, PsycINFO and SPORTDiscus were searched. English-language, peer-review published studies of acute (<72 h) concussion assessments were included. Results: A total of 31 studies met inclusion criteria (of 291 evaluated); 27 studies provided sufficient information to be included in the descriptive statistics of the assessments. Six of these assessments were used in at least three studies. Only 12 percent of the population studied was female. The age range for these assessments was 9-67 years, although most participants ranged in age between 18 and 35 years. Conclusion: There is a need for a 'gold' standard concussion assessment to enable consistency across research and clinical outcomes. We found a large discrepancy between the number of males and females assessed, suggesting that future studies are needed to determine if these current assessments identify concussion signs and symptoms unique to females. Further studies are needed to determine which assessments are appropriate and valid for youth athletes.

Prospective, Head-to-Head Study of Three Computerized Neurocognitive Assessment Tools (CNTs): Reliability and Validity for the Assessment of Sport-Related Concussion

Limited data exist comparing the performance of computerized neurocognitive tests (CNTs) for assessing sport-related concussion. We evaluated the reliability and validity of three CNTs-ANAM, Axon Sports/Cogstate Sport, and ImPACT-in a common sample. High school and collegiate athletes completed two CNTs each at baseline. Concussed (n=165) and matched non-injured control (n=166) subjects repeated testing within 24 hr and at 8, 15, and 45 days post-injury. Roughly a quarter of each CNT's indices had stability coefficients (M=198 day interval) over .70. Group differences in performance were mostly moderate to large at 24 hr and small by day 8. The sensitivity of reliable change indices (RCIs) was best at 24 hr (67.8%, 60.3%, and 47.6% with one or more significant RCIs for ImPACT, Axon, and ANAM, respectively) but diminished to near the false positive rates thereafter. Across time, the CNTs' sensitivities were highest in those athletes who became asymptomatic within 1 day before neurocognitive testing but was similar to the tests' false positive rates when including athletes who became asymptomatic several days earlier. Test-retest reliability was similar among these three CNTs and below optimal standards for clinical use on many subtests. Analyses of group effect sizes, discrimination, and sensitivity and specificity suggested that the CNTs may add incrementally (beyond symptom scores) to the identification of clinical impairment within 24 hr of injury or within a short time period after symptom resolution but do not add significant value over symptom assessment later. The rapid clinical recovery course from concussion and modest stability probably jointly contribute to limited signal detection capabilities of neurocognitive tests outside a brief post-injury window. (JINS, 2016, 22, 24-37).