Short-term test–retest reliability of the ImPACT in healthy young athletes

Psychometric Properties of Computerized Cognitive Tools and Standard Neuropsychological Tests Used to Assess Sport Concussion: A Systematic Review

Athletic programs are more frequently turning to computerized cognitive tools in order to increase efficiencies in concussion assessment. However, assessment using a traditional neuropsychological test battery may provide a more comprehensive and individualized evaluation. Our goal was to inform sport clinicians of the best practices for concussion assessment through a systematic literature review describing the psychometric properties of standard neuropsychological tests and computerized tools. We conducted our search in relevant databases including Ovid Medline, Web of Science, PsycINFO, and Scopus. Journal articles were included if they evaluated psychometric properties (e.g., reliability, sensitivity) of a cognitive assessment within pure athlete samples (up to 30 days post-injury). Searches yielded 4,758 unique results. Ultimately, 103 articles met inclusion criteria, all of which focused on adolescent or young adult participants. Test-retest reliability estimates ranged from .14 to .93 for computerized tools and .02 to .95 for standard neuropsychological tests, with strongest correlations on processing speed tasks for both modalities, although processing speed tasks were most susceptible to practice effects. Reliability was improved with a 2-factor model (processing speed and memory) and by aggregating multiple baseline exams, yet remained below acceptable limits for some studies. Sensitivity to decreased cognitive performance within 72 h of injury ranged from 45%-93% for computerized tools and 18%-80% for standard neuropsychological test batteries. The method for classifying cognitive decline (normative comparison, reliable change indices, regression-based methods) affected sensitivity estimates. Combining computerized tools and standard neuropsychological tests with the strongest psychometric performance provides the greatest value in clinical assessment. To this end, future studies should evaluate the efficacy of hybrid test batteries comprised of top-performing measures from both modalities.

Test-Retest Reliability of Remote ImPACT Administration

OBJECTIVE

To evaluate the performance and test-retest reliability obtained when administering a computerized baseline neurocognitive exam to NCAA Division I student-athletes in a controlled laboratory setting versus an uncontrolled remote location.

METHOD

A sample of 129 (female = 100) Division I student-athletes completed Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT) pre-season assessments for two distinct and respective sports seasons in a controlled laboratory environment and an uncontrolled remote environment. Depending on the environment, participants were given verbal (controlled) or written (uncontrolled) guidelines for taking the test.

RESULTS

Multivariate repeated-measures ANOVA's determined that there were no within-subject differences between testing environments on ImPACT composite scores and cognitive efficiency index (CEI). The Chi-square test did not find any significant differences in impulse control or the number of invalid test scores, as determined by ImPACT, between environments. Intraclass correlations found the ImPACT subtest scores to range in test-retest reliability across testing environments, demonstrating moderate (verbal memory composite, r = 0.46; visual memory composite, r = 0.64; reaction time, r = 0.61; impulse control, r = 0.52; and CEI, r = 0.61) and good (visual motor composite, r = 0.77) test-retest reliability.

CONCLUSIONS

Results indicate that ImPACT is reliable between controlled and uncontrolled testing environments. This further suggests that ImPACT can be administered in a remote environment, pending specific adherence to testing instructions, or in the event of social distancing or isolation policies.

Methodological considerations of assessing meaningful/reliable change in computerized neurocognitive testing following sport-related concussion

OBJECTIVE

Traditional reliable change index (RCI) methods may be psychometrically limited due to their inability to account for particularly high or low baseline performance and regression to the mean following serial testing. The current study sought to examine differences between RCI and standardized regression-based (RBz) methods in a cohort of adolescent athletes engaged in sport-related concussion recovery.

METHODS

Consultation records and results of computerized testing data via the Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT) battery were retrospectively reviewed among 240 adolescent athletes involved in return-to-play protocols following a sport-related concussion.

RESULTS

High concordance rates were found between RCI and RBz metrics across ImPACT composites, consistent with previous literature. However, especially for those with lower baseline performance, RBz scores tended to suggest cognitive performance not meeting or exceeding baseline scores despite RCI metrics being appropriate across speed-based ImPACT composites. In contrast, results revealed high rates of RCI scores suggesting continued cognitive difficulties despite RBz metrics being within normal limits, especially for adolescents with higher baseline performance.

CONCLUSIONS

Results suggest value in interpreting RBz values in addition to RCI values as these allow for clinical interpretation more sensitive to statistical confounds, including baseline performance and regression to the mean.

Performance Validity Testing in Patients Presenting to a Specialty Clinic With a Mild Traumatic Brain Injury

OBJECTIVE

To evaluate predictors of performance validity testing (PVT) and clinical outcome in patients presenting to a specialty clinic with a mild traumatic brain injury (mTBI).

SETTING

An outpatient mTBI specialty clinic.

PARTICIPANTS

Seventy-six (47% female) patients aged 16 to 66 (mean = 40.58, SD = 14.18) years within 3 to 433 days (mean = 30.63, SD = 54.88, median = 17.00) of a suspected mTBI between 2018 and 2019.

DESIGN

A cross-sectional, observational study comparing patients who passed PVT (n = 43) with those who failed (n = 33). A logistic regression (LR) was conducted to evaluate factors that predicted failed PVT. Independent-samples t tests and general linear model were used to evaluate PVT groups on clinical outcomes. The LR with a receiver operating characteristic (ROC) curve was conducted to evaluate embedded validity indicators.

MAIN MEASURES

Performance validity testing, computerized neurocognitive testing, vestibular/oculomotor screening, symptom reports.

RESULTS

At their initial clinic visit 43% of patients failed PVT. PVT failure was predicted by presence of secondary gain (odds ratio [OR] = 8.11, P = .02), while a history of mental health predicted passing of PVT (OR = 0.29, P = .08). Those who failed PVT performed significantly worse on computerized neurocognitive testing (P < .05) and took an average of 33 days longer to return to work (P = .02). There was no significant difference (P = .20) in recovery time between failed/passed PVT groups when covarying for those who sustained a work injury. Word memory learning percentage less than 69% and design memory learning percentage less than 50% accurately classified patients who failed PVT (area under the ROC curve = 0.74; P < .001).

CONCLUSION

Secondary gain was the best predictor of failed PVT. Patients presenting for mTBI evaluation and rehabilitation who fail PVT demonstrate worse performance on cognitive testing and take longer to return to work post-injury, but recover in a similar time frame compared with those who pass PVT. Clinicians should be cautious in discounting patients who yield invalid test results, as these patients appear to be able to achieve recovery in a treatment setting.

A short digital eye-tracking assessment predicts cognitive status among adults

Current cognitive assessments suffer from limited scalability and high user burden. This study aimed to (1) examine the relationship between a brief eye-tracking-based visual paired-comparison (VPC) and gold standard cognitive assessments, (2) examine longitudinal stability of the VPC task, (3) determine the ability of the VPC task to differentiate between cognitively normal (CN) individuals and individuals with mild cognitive impairment (MCI). Fifty-five adults (n = 44 CN, n = 11 MCI; 56.4 ± 26.7 years) were tested on two occasions, separated by at least 14 days. Visit 1 included VPC, Montreal Cognitive Assessment (MoCA), Digit Symbol Coding test (DSC), and NIH Toolbox Cognitive Battery (NIHTB-CB). Visit 2 included VPC, DSC, NIHTB-CB, and dual-task (DT). Significant differences existed between baseline VPC scores for CN and MCI groups (p < .001). VPC scores remained stable over time in both groups (p < .05). Significant associations existed between VPC and MoCA (p < .01), DSC (p < .001), and various NIHTB-CB subtests at both time points. The VPC test significantly predicts cognitive outcomes (p < .05), with age and VPC being the only significant predictors. Additionally, area under the curve (receiver operator characteristic = 0.80) for VPC scores demonstrated good classification accuracy. VPC reliably predicted cognitive status while remaining stable over time and displayed significant associations with gold standard cognitive assessments. VPC is a less burdensome and more scalable assessment than traditional tests, enabling longitudinal monitoring of cognitive status in resource-limited environments.

Validation of a Machine Learning Brain Electrical Activity-Based Index to Aid in Diagnosing Concussion Among Athletes

IMPORTANCE

An objective, reliable indicator of the presence and severity of concussive brain injury and of the readiness for the return to activity has the potential to reduce concussion-related disability.

OBJECTIVE

To validate the classification accuracy of a previously derived, machine learning, multimodal, brain electrical activity-based Concussion Index in an independent cohort of athletes with concussion.

DESIGN, SETTING, AND PARTICIPANTS

This prospective diagnostic cohort study was conducted at 10 clinical sites (ie, US universities and high schools) between February 4, 2017, and March 20, 2019. A cohort comprising a consecutive sample of 207 athletes aged 13 to 25 years with concussion and 373 matched athlete controls without concussion were assessed with electroencephalography, cognitive testing, and symptom inventories within 72 hours of injury, at return to play, and 45 days after return to play. Variables from the multimodal assessment were used to generate a Concussion Index at each time point. Athletes with concussion had experienced a witnessed head impact, were removed from play for 5 days or more, and had an initial Glasgow Coma Scale score of 13 to 15. Participants were excluded for known neurologic disease or history within the last year of traumatic brain injury. Athlete controls were matched to athletes with concussion for age, sex, and type of sport played.

MAIN OUTCOMES AND MEASURES

Classification accuracy of the Concussion Index at time of injury using a prespecified cutoff of 70 or less (total range, 0-100, where ≤70 indicates it is likely the individual has a concussion and >70 indicates it is likely the individual does not have a concussion).

RESULTS

Of 580 eligible participants with analyzable data, 207 had concussion (124 male participants [59.9%]; mean [SD] age, 19.4 [2.5] years), and 373 were athlete controls (187 male participants [50.1%]; mean [SD] age, 19.6 [2.2] years). The Concussion Index had a sensitivity of 86.0% (95% CI, 80.5%-90.4%), specificity of 70.8% (95% CI, 65.9%-75.4%), negative predictive value of 90.1% (95% CI, 86.1%-93.3%), positive predictive value of 62.0% (95% CI, 56.1%-67.7%), and area under receiver operator characteristic curve of 0.89. At day 0, the mean (SD) Concussion Index among athletes with concussion was significantly lower than among athletes without concussion (75.0 [14.0] vs 32.7 [27.2]; P < .001). Among athletes with concussion, there was a significant increase in the Concussion Index between day 0 and return to play, with a mean (SD) paired difference between these time points of -41.2 (27.0) (P < .001).

CONCLUSIONS AND RELEVANCE

These results suggest that the multimodal brain activity-based Concussion Index has high classification accuracy for identification of the likelihood of concussion at time of injury and may be associated with the return to control values at the time of recovery. The Concussion Index has the potential to aid in the clinical diagnosis of concussion and in the assessment of athletes' readiness to return to play.