Preliminary validation of the computerized N-Tri - A Tri-Choice naming and response bias test

The study describes the validation of a computerized adaptation of the novel Tri-Choice Naming and Response Bias Measure (N-Tri) developed to detect untruthful responding while being less susceptible to coaching than existing measures. We hypothesized that the N-Tri would have comparable sensitivity and specificity to traditional tests but would have improved accuracy for detecting coached simulators. Four-hundred volunteers were randomly assigned to one of three groups: uncoached simulators' group (n = 118), coached simulators' group (n = 136), or control group (n = 146). Both simulator groups were asked to feign concussion symptoms, but the coached group received a test-taking strategy and a description of concussion symptoms. The participants were administered the computerized version of the new measure in conjunction with computerized adaptations of two well-validated response bias tests commonly used to detect cognitive malingering, the Reliable Digit Span (RDS) and Portland Digit Recognition Test (PDRT). Our data show the new measure correlated highly with other established measures. However, the classification accuracy did not significantly increase when compared to the traditional tests. Our findings support that the N-Tri performs at a comparable level to existing forced choice measures of response bias. Nevertheless, the N-Tri could potentially improve the detection of response bias as existing tests become more recognizable by the public.

Determining whether false positive rates increase with performance validity test battery expansion

OBJECTIVE

Performance validity test (PVT) misclassification is an important concern for neuropsychologists. The present study determined whether expanding PVT analysis from 4-PVTs to 8-PVTs could lead to elevated rates of false positive performance validity misclassifications.

METHOD

Retrospective analysis of 443 patients who underwent a fixed neuropsychological test battery in a mixed clinical and forensic setting. Rates of failing two PVTs were compared to those predicted by Monte Carlo simulations when PVT analysis extended from 4-PVTs to 8-PVTs. Indeterminate performers (IDT; n = 42; those who failed two PVTs only after PVT analysis extended from 4-PVTs to 8-PVTs) were compared to a PVT-Fail group (n = 148; those who failed two PVTs in the 4-PVT battery or failed >2 PVTs).

RESULTS

Rate of failing two PVTs remained stable when PVT analysis extended from 4- to 8-PVTs (12.9 to 11.9%) and was significantly lower than those predicted by Monte Carlo simulations. Compared to PVT-Fail, the IDT group was significantly younger, had stronger neuropsychological test performance, and demonstrated comparable rates of forensic referral and conditions with known neurocognitive sequelae (e.g. stroke, moderate-to-severe TBI).

CONCLUSIONS

Monte Carlo simulations significantly overestimated rates of individuals failing two PVTs as PVT battery length doubled. IDT did not differ from PVT-Fail across variables with known PVT effects (e.g. age, referral context, neurologic diagnoses), lowering concern that this group is comprised entirely of false-positive PVT classifications. More research is needed to determine the effect of PVT battery length on validity classification accuracy.

CVLT-II short form forced choice recognition in a clinical dementia sample: Cautions for performance validity assessment

Performance validity tests are susceptible to false positives from genuine cognitive impairment (e.g., dementia); this has not been explored with the short form of the California Verbal Learning Test II (CVLT-II-SF). In a memory clinic sample, we examined whether CVLT-II-SF Forced Choice Recognition (FCR) scores differed across diagnostic groups, and how the severity of impairment [Clinical Dementia Rating Sum of Boxes (CDR-SOB) or Mini-Mental State Examination (MMSE)] modulated test performance. Three diagnostic groups were identified: subjective cognitive impairment (SCI; n = 85), amnestic mild cognitive impairment (a-MCI; n = 17), and dementia due to Alzheimer's Disease (AD; n = 50). Significant group differences in FCR were observed using one-way ANOVA; post-hoc analysis indicated the AD group performed significantly worse than the other groups. Using multiple regression, FCR performance was modeled as a function of the diagnostic group, severity (MMSE or CDR-SOB), and their interaction. Results yielded significant main effects for MMSE and diagnostic group, with a significant interaction. CDR-SOB analyses were non-significant. Increases in impairment disproportionately impacted FCR performance for persons with AD, adding caution to research-based cutoffs for performance validity in dementia. Caution is warranted when assessing performance validity in dementia populations. Future research should examine whether CVLT-II-SF-FCR is appropriately specific for best-practice testing batteries for dementia.

Forced choice test score of p ≤ .20 and failures on ≥ six performance validity tests results in similar Overall Test Battery Means

Objective: To determine if similar levels of performance on the Overall Test Battery Mean (OTBM) occur at different forced choice test (FCT) p-value score failures. Second, to determine the OTBM levels that are associated with failures at above chance on various performance validity (PVT) tests. Method: OTBMs were computed from archival data obtained from four practices. We calculated each examinee's Estimated Premorbid Global Ability (EPGA) and OTBM. The sample size was 5,103 examinees with 282 (5.5%) of these scoring below chance at p ≤ .20 on at least one FCT. Results: The OTBM associated with a failure at p ≤ .20 was equivalent to the OTBM that was associated with failing 6 or more PVTs at above-chance cutoffs. The mean OTBMs relative to increasingly strict FCT p cutoffs were similar (T scores in the 30s). As expected, there was an inverse relationship between the number of PVTs failed and examinees' OTBMs. Conclusions: The data support the use of p ≤ .20 as the probability level for testing the significance of below chance performance on FCTs. The OTBM can be used to index the influence of invalid performance on outcomes, especially when an examinee scores below chance.

Importance of validity testing in psychiatric assessment: evidence from a sample of multimorbid post-9/11 veterans

OBJECTIVE

Performance validity (PVTs) and symptom validity tests (SVTs) are necessary components of neuropsychological testing to identify suboptimal performances and response bias that may impact diagnosis and treatment. The current study examined the clinical and functional characteristics of veterans who failed PVTs and the relationship between PVT and SVT failures.

METHOD

Five hundred and sixteen post-9/11 veterans participated in clinical interviews, neuropsychological testing, and several validity measures.

RESULTS

Veterans who failed 2+ PVTs performed significantly worse than veterans who failed one PVT in verbal memory (Cohen's d = .60-.69), processing speed (Cohen's d = .68), working memory (Cohen's d = .98), and visual memory (Cohen's d = .88-1.10). Individuals with 2+ PVT failures had greater posttraumatic stress (PTS; β = 0.16; p = .0002), and worse self-reported depression (β = 0.17; p = .0001), anxiety (β = 0.15; p = .0007), sleep (β = 0.10; p = .0233), and functional outcomes (β = 0.15; p = .0009) compared to veterans who passed PVTs. 7.8% veterans failed the SVT (Validity-10; ≥19 cutoff); Multiple PVT failures were significantly associated with Validity-10 failure at the ≥19 and ≥23 cutoffs (p's < .0012). The Validity-10 had moderate correspondence in predicting 2+ PVTs failures (AUC = 0.83; 95% CI = 0.76, 0.91).

CONCLUSION

PVT failures are associated with psychiatric factors, but not traumatic brain injury (TBI). PVT failures predict SVT failure and vice versa. Standard care should include SVTs and PVTs in all clinical assessments, not just neuropsychological assessments, particularly in clinically complex populations.

Ability of the Wisconsin Card-Sorting Test-64 as an embedded measure to identify noncredible neurocognitive performance in mild traumatic brain injury litigants

OBJECTIVE

To investigate the ability of selective measures on the Wisconsin Card Sorting Test-64 (WCST-64) to predict noncredible neurocognitive dysfunction in a large sample of mild traumatic brain injury (mTBI) litigants.

METHOD

Participants included 114 adults who underwent a comprehensive neuropsychological examination. Criterion groups were formed based upon their performance on stand-alone measures of cognitive performance validity (PVT).

RESULTS

Participants failing PVTs performed worse across all WCST-64 dependent variables of interest compared to participants who passed PVTs. Receiver operating curve analysis revealed that only categories completed was a significant predictors of PVT status. Multivariate logistic regression did not add to classification accuracy.

CONCLUSION

Consideration of noncredible executive functioning may be warranted in mild traumatic brain injury (mTBI) litigants who complete ≤ 1 category on the WCST-64.

Visuospatial memory in apraxia: Exploring quantitative drawing metrics to assess the representation of local and global information

Neuropsychological evidence suggests that visuospatial memory is subserved by two separable processing systems, with dorsal underpinnings for global form and ventral underpinnings for the integration of part elements. Previous drawing studies have explored the effects of Gestalt organisation upon memory for hierarchical stimuli, and we here present an exploratory study of an apraxic dorsal stream patient's (MH) performance. We presented MH with a stimulus set (previously reported by Riddoch et al., Cognitive Neuropsychology, 20(7), 641-671, 2003) and devised a novel quantitative scoring system to obtain a finer grain of insight into performance. Stimuli possessed either good or poor Gestalt qualities and were reproduced in a copy condition and two visual memory conditions (with unlimited viewing before the model was removed, or with 3 s viewing). MH's copying performance was impaired in comparison to younger adult and age-matched older adult controls, with a variety of errors at the local level but relatively few at the global level. However, his performance in the visual memory conditions revealed impairments at the global level. For all participants, drawing errors were modulated by the Gestalt qualities of the stimuli, with accuracy at the global and local levels being lesser for poor global stimuli in all conditions. These data extend previous observations of this patient, and support theories that posit interaction between dorsal and ventral streams in the representation of hierarchical stimuli. We discuss the implications of these findings for our understanding of visuospatial memory in neurological patients, and also evaluate the application of quantitative metrics to the interpretation of drawings.

Lowered cutoffs to reduce false positives on the Word Memory Test

OBJECTIVE

To adjust the decision criterion for the Word Memory Test (WMT, Green, 2003) to minimize the frequency of false positives.

METHOD

Archival data were combined into a database (n = 3,210) to examine the best cut score for the WMT. We compared results based on the original scoring rules and those based on adjusted scoring rules using a criterion based on 16 performance validity tests (PVTs) exclusive of the WMT. Cutoffs based on peer-reviewed publications and test manuals were used. The resulting PVT composite was considered the best estimate of validity status. We focused on a specificity of .90 with a false-positive rate of less than .10 across multiple samples.

RESULTS

Each examinee was administered the WMT, as well as on average 5.5 (SD = 2.5) other PVTs. Based on the original scoring rules of the WMT, 31.8% of examinees failed. Using a single failure on the criterion PVT (C-PVT), the base rate of failure was 45.9%. When requiring two or more failures on the C-PVT, the failure rate dropped to 22.8%. Applying a contingency analysis (i.e., X2) to the two failures model on the C-PVT measure and using the original rules for the WMT resulted in only 65.3% agreement. However, using our adjusted rules for the WMT, which consisted of relying on only the IR and DR WMT subtest scores with a cutoff of 77.5%, agreement between the adjusted and the C-PVT criterion equaled 80.8%, for an improvement of 12.1% identified. The adjustmeny resulted in a 49.2% reduction in false positives while preserving a sensitivity of 53.6%. The specificity for the new rules was 88.8%, for a false positive rate of 11.2%.

CONCLUSIONS

Results supported lowering of the cut score for correct responding from 82.5% to 77.5% correct. We also recommend discontinuing the use of the Consistency subtest score in the determination of WMT failure.

Strategies of feigning mild head injuries related to validity indicators and types of coaching: Results of two experimental studies

OBJECTIVE

In this paper, we analyzed differences between uncoached, symptom-coached, and test-coached simulators regarding strategies of feigning mild head injuries.

METHOD

Healthy undergraduates (n = 67 in the first study; n = 48 in the second study), randomized into three simulator groups, were assessed with four experimental memory tests. In the first study, tests were administered face-to-face, while in the second study, the procedure was adapted for online testing.

RESULTS

Online simulators showed a different approach to testing than face-to-face participants (U tests  < 920, p < .05). Nevertheless, both samples favored strategies like memory loss, error making, concentration difficulties, and slow responding. Except for slow responding and concentration difficulties, the favorite strategies correlated with validity indicators. In the first study, test-coached simulators (m = 4.58-5.68, SD = 2.2-3) used strategies less than uncoached participants (m = 5.25-5.88, SD = 2.26-2.84). In the second study, test-coached participants (m = 3.8-5.6, SD = 1.51-2.2) employed strategies less than uncoached (m = 6.21-7.29, SD = 1.25-1.85) and symptom-coached participants (m = 6.14-6.79, SD = 1.69-2.76).

DISCUSSION

Similarities and differences between online and face-to-face assessments are discussed. Recommendations to associate heterogeneous indicators for detecting feigning strategies are issued.

Threats to the integrity of psychological assessment: The misuse of test raw data and materials

In the practice of psychological assessment there have been warnings for decades by the American Psychological Association (APA), the National Academy of Neuropsychology (NAN), other associations, and test vendors, against the disclosure of test raw data and test materials. Psychological assessment occurs across several different practice environments, and test raw data is a particularly sensitive aspect of practice considering what it implicitly represents about a client/patient, and this concept is further developed in this paper. Many times, test materials are intellectual property protected by copyrights and user agreements. It follows that improper management of the release of test raw data and test materials threatens the scientific integrity of psychological assessment. Here the matters of test raw data, test materials, and different practice environments are addressed to highlight the challenges involved with improper releases and to offer guidance concerning good-faith efforts to preserve the integrity of psychological assessment and legal agreements. The unique demands of forensic practice are also discussed, including attorneys' needs for cross-examination and discovery, which may place psychologists (and other duly vetted evaluators) in conflict with their commitment to professional ethical codes and legal agreements. To this end, important threats to the proper use of test raw data and test materials include uninformed professionals and compromised evaluators. In this paper, the mishandling of test raw data and materials by both psychologists and other evaluators is reviewed, representative case examples, including those from the literature, are provided, pertinent case law is discussed, and practical stepwise conflict resolutions are offered.

Cognitive Performance, Depression, and Anxiety 1 Year After Traumatic Brain Injury

OBJECTIVES

To evaluate associations between depression, anxiety, and cognitive impairment among individuals with complicated mild to severe traumatic brain injury (TBI) 1 year after injury.

SETTING

Multiple inpatient rehabilitation units across the United States.

PARTICIPANTS

A total of 498 adults 16 years and older who completed inpatient rehabilitation for complicated mild to severe TBI.

DESIGN

Secondary analysis of a prospective, multicenter, cross-sectional observational cohort study.

MAIN MEASURES

Assessments of depression (Traumatic Brain Injury Quality of Life [TBI-QOL] Depression) and anxiety (TBI-QOL Anxiety) as well as a telephone-based brief screening measure of cognitive functioning (Brief Test of Adult Cognition by Telephone [BTACT]).

RESULTS

We found an inverse relationship between self-reported depression symptoms and the BTACT Composite score (β = -0.18, P < .01) and anxiety symptoms and the BTACT Composite score (β = -0.20, P < .01). There was no evidence this relationship varied by injury severity. Exploratory analyses showed depression and anxiety were negatively correlated with both BTACT Executive Function factor score and BTACT Memory factor score.

CONCLUSIONS

Both depression and anxiety have a small but significant negative association with cognitive performance in the context of complicated mild to severe TBI. These findings highlight the importance of considering depression and anxiety when interpreting TBI-related neuropsychological impairments, even among more severe TBI.

Grooved Pegboard adds incremental value over memory-apparent performance validity tests in predicting psychiatric symptom report

The present study evaluated whether Grooved Pegboard (GPB), when used as a performance validity test (PVT), can incrementally predict psychiatric symptom report elevations beyond memory-apparent PVTs. Participants (N = 111) were military personnel and were predominantly White (84%), male (76%), with a mean age of 43 (SD = 12) and having on average 16 years of education (SD = 2). Individuals with disorders potentially compromising motor dexterity were excluded. Participants were administered GPB, three memory-apparent PVTs (Medical Symptom Validity Test, Non-Verbal Medical Symptom Validity Test, Reliable Digit Span), and a symptom validity test (Personality Assessment Inventory Negative Impression Management [NIM]). Results from the three memory-apparent PVTs were entered into a model for predicting NIM, where failure of two or more PVTs was categorized as evidence of non-credible responding. Hierarchical regression revealed that non-dominant hand GPB T-score incrementally predicted NIM beyond memory-apparent PVTs (F(2,108) = 16.30, p < .001; R² change = .05, β = -0.24, p < .01). In a second hierarchical regression, GPB performance was dichotomized into pass or fail, using T-score cutoffs (≤29 for either hand, ≤31 for both). Non-dominant hand GPB again predicted NIM beyond memory-apparent PVTs (F(2,108) = 18.75, p <.001; R² change = .08, β = -0.28, p < .001). Results indicated that noncredible/failing GPB performance adds incremental value over memory-apparent PVTs in predicting psychiatric symptom report.

Failing The b Test: The influence of cutoff scores and criterion group approaches in a sample of adults referred for psychoeducational evaluation

OBJECTIVE

Previous research has shown that both criterion grouping approaches and cutoff scores can impact PVT classification accuracy statistics. This study aimed to examine the influence of cutoff scores and criterion grouping approaches on The b Test, a measure designed to identify feigned impairment in visual scanning, processing speed, and letter identification.

METHOD

Two hundred ninety-seven adults referred for psychoeducational testing were included with the majority of individuals seeking academic accommodations (n = 215). Cutoff scores of ≥82, ≥90, and ≥120 were utilized along with two different criterion group approaches, 0 PVT failures vs. ≥2 PVT failures and 0 PVT failures versus ≥ 1 PVT failures.

RESULTS

Failure rates for The b Test in the overall sample ranged from 12.5% to 16.2%. Subgroup analyses in those referred specifically for ADHD revealed failure rates for The b Test ranging from 10.5% to 14.2%. ROC curves within the full sample and ADHD subsample demonstrated significant AUCs utilizing both criterion group approaches (AUC = .66 - .78). Sensitivity and specificity varied as a function of criterion group approach and cutoff score, with 0 PVT failures vs. ≥ 2 PVT failures resulting in the greatest sensitivity when maximizing specificity at ≥.90 in the full sample and ADHD sample.

CONCLUSIONS

The results demonstrate that criterion approaches and cutoff scores impact classification accuracy of The b Test with 0 PVT vs. ≥ 2 PVT failures demonstrating the greatest classification accuracy. Special considerations should be made with regard to clinical decision making in the context of psychoeducational evaluations given that a large portion of individuals seeking accommodations fail only one PVT. Limitations of this study are also discussed.

Utility of WAIS-IV Digit Span indices as measures of performance validity in moderate to severe traumatic brain injury

Objective: The addition of Sequencing to WAIS-IV Digit Span (DS) brought about new Reliable Digit Span (RDS) indices and an Age-Corrected Scaled Score that includes Sequencing trials. Reports have indicated that these new performance validity tests (PVTs) are superior to the traditional RDS; however, comparisons in the context of known neurocognitive impairment are sparse. This study compared DS-derived PVT classification accuracies in a design that included adults with verified TBI. Methods: Participants included 64 adults with moderate-to-severe TBI (TBI), 51 healthy adults coached to simulate TBI (SIM), and 78 healthy comparisons (HC). Participants completed the WAIS-IV DS subtest in the context of a larger test battery. Results: Kruskal-Wallis tests indicated that all DS indices differed significantly across groups. Post hoc contrasts revealed that only RDS Forward and the traditional RDS differed significantly between SIM and TBI. ROC analyses indicated that RDS variables were comparable predictors of SIM vs. HC; however, the traditional RDS showed the highest sensitivity when approximating 90% specificity for SIM vs. TBI. A greater percentage of TBI scored RDS Sequencing < 1 compared to SIM and HC. Conclusion: In the context of moderate-to-severe TBI, the DS-derived PVTs showed comparable discriminability. However, the Greiffenstein et al. traditional RDS demonstrated the best classification accuracy with respect to specificity/sensitivity balance. This relative superiority may reflect that individuals with verified TBI are more likely to perseverate on prior instructions during DS Sequencing. Findings highlight the importance of including individuals with verified TBI when evaluating and developing PVTs.

Assessment of differential neurocognitive performance based on the number of performance validity tests failures: A cross-validation study across multiple mixed clinical samples

Objective: This cross-sectional study examined the effect of number of Performance Validity Test (PVT) failures on neuropsychological test performance among a demographically diverse Veteran (VA) sample (n = 76) and academic medical sample (AMC; n = 128). A secondary goal was to investigate the psychometric implications of including versus excluding those with one PVT failure when cross-validating a series of embedded PVTs. Method: All patients completed the same six criterion PVTs, with the AMC sample completing three additional embedded PVTs. Neurocognitive test performance differences were examined based on number of PVT failures (0, 1, 2+) for both samples, and effect of number of criterion failures on embedded PVT performance was analyzed among the AMC sample. Results: Both groups with 0 or 1 PVT failures performed better than those with ≥2 PVT failures across most cognitive tests. There were nonsignificant differences between those with 0 or 1 PVT failures except for one test in the AMC sample. Receiver operator characteristic curve analyses found no differences in optimal cut score based on number of PVT failures when retaining/excluding one PVT failure. Conclusion: Findings support the use of ≥2 PVT failures as indicative of performance invalidity. These findings strongly support including those with one PVT failure with those with zero PVT failures in diagnostic accuracy studies, given that their inclusion reflects actual clinical practice, does not reduce sample sizes, and does not artificially deflate neurocognitive test results or inflate PVT classification accuracy statistics.

Profile of Embedded Validity Indicators in Criminal Defendants with Verified Valid Neuropsychological Test Performance

Objective

Few studies have examined the use of embedded validity indicators (EVIs) in criminal-forensic practice settings, where judgements regarding performance validity can carry severe consequences for the individual and society. This study sought to examine how various EVIs perform in criminal defendant populations, and determine relationships between EVI scores and intrapersonal variables thought to influence performance validity.

Method

Performance on 16 empirically established EVI cutoffs were examined in a sample of 164 criminal defendants with valid performance who were referred for forensic neuropsychological evaluation. Subsequent analyses examined the relationship between EVI scores and intrapersonal variables in 83 of these defendants.

Results

Half of the EVIs (within the Wechsler Adult Intelligence Scale Digit Span Total, Conners’ Continuous Performance Test Commissions, Wechsler Memory Scale Logical Memory I and II, Controlled Oral Word Association Test, Trail Making Test Part B, and Stroop Word and Color) performed as intended in this sample. The EVIs that did not perform as intended were significantly influenced by relevant intrapersonal variables, including below-average intellectual functioning and history of moderate–severe traumatic brain injury and neurodevelopmental disorder.

Conclusions

This study identifies multiple EVIs appropriate for use in criminal-forensic settings. However, based on these findings, practitioners may wish to be selective in choosing and interpreting EVIs for forensic evaluations of criminal court defendants.

A Validity Measure for the Automated Neuropsychological Assessment Metrics

The Automated Neuropsychological Assessment Metrics (ANAM) is one of the most widely used and validated neuropsychological instruments for assessing cognition. The ANAM Test System includes a reporting tool, the ANAM Validity Indicator Report that generates scores for the embedded effort measure, the ANAM Performance Validity Index (APVI). The current study seeks to develop a proxy for the APVI, using raw subtest summary test scores. This would be useful for situations where the APVI score is unavailable (e.g., validity report not generated at the time of the assessment) or when the item level data needed to generate this score are inaccessible. ANAM scores from a large data set of 1,000,000+ observations were used for this retrospective analysis. Results of linear regression analysis suggest that the APVI can be reasonably estimated from the raw subtest summary test scores that are presented on the ANAM Performance Report. Clinically, this means that an important step in the interpretation process, checking the validity of test data, can still be performed even when the APVI is not available.

Objective Cognitive Outcomes and Subjective Emotional Sequelae in Litigating Adults with a Traumatic Brain Injury: The Impact of Performance and Symptom Validity Measures

OBJECTIVE

This study examined the relative contribution of performance and symptom validity in litigating adults with traumatic brain injury (TBI), as a function of TBI severity, and examined the relationship between self-reported emotional symptoms and cognitive tests scores while controlling for validity test performance.

METHOD

Participants underwent neuropsychological assessment between January 2012 and June 2021 in the context of compensation-seeking claims related to a TBI. All participants completed a cognitive test battery, the Personality Assessment Inventory (including symptom validity tests; SVTs), and multiple performance validity tests (PVTs). Data analyses included independent t-tests, one-way ANOVAs, correlation analyses, and hierarchical multiple regression.

RESULTS

A total of 370 participants were included. Atypical PVT and SVT performance were associated with poorer cognitive test performance and higher emotional symptom report, irrespective of TBI severity. PVTs and SVTs had an additive effect on cognitive test performance for uncomplicated mTBI, but less so for more severe TBI. The relationship between emotional symptoms and cognitive test performance diminished substantially when validity test performance was controlled, and validity test performance had a substantially larger impact than emotional symptoms on cognitive test performance.

CONCLUSION

Validity test performance has a significant impact on the neuropsychological profiles of people with TBI, irrespective of TBI severity, and plays a significant role in the relationship between emotional symptoms and cognitive test performance. Adequate validity testing should be incorporated into every neuropsychological assessment, and associations between emotional symptoms and cognitive outcomes that do not consider validity testing should be interpreted with extreme caution.

The Relationship Between Cognitive Functioning and Symptoms of Depression, Anxiety, and Post-Traumatic Stress Disorder in Adults with a Traumatic Brain Injury: a Meta-Analysis

A thorough understanding of the relationship between cognitive test performance and symptoms of depression, anxiety, or post-traumatic stress disorder (PTSD) in people with traumatic brain injury (TBI) is important given the high prevalence of these emotional symptoms following injury. It is also important to understand whether these relationships are affected by TBI severity, and the validity of test performance and symptom report. This meta-analysis was conducted to investigate whether these symptoms are associated with cognitive test performance alterations in adults with a TBI. This meta-analysis was prospectively registered on the PROSPERO International Prospective Register of Systematic Reviews website (registration number: CRD42018089194). The electronic databases Medline, PsycINFO, and CINAHL were searched for journal articles published up until May 2020. In total, 61 studies were included, which enabled calculation of pooled effect sizes for the cognitive domains of immediate memory (verbal and visual), recent memory (verbal and visual), attention, executive function, processing speed, and language. Depression had a small, negative relationship with most cognitive domains. These relationships remained, for the most part, when samples with mild TBI (mTBI)-only were analysed separately, but not for samples with more severe TBI (sTBI)-only. A similar pattern of results was found in the anxiety analysis. PTSD had a small, negative relationship with verbal memory, in samples with mTBI-only. No data were available for the PTSD analysis with sTBI samples. Moderator analyses indicated that the relationships between emotional symptoms and cognitive test performance may be impacted to some degree by exclusion of participants with atypical performance on performance validity tests (PVTs) or symptom validity tests (SVTs), however there were small study numbers and changes in effect size were not statistically significant. These findings are useful in synthesising what is currently known about the relationship between cognitive test performance and emotional symptoms in adults with TBI, demonstrating significant, albeit small, relationships between emotional symptoms and cognitive test performance in multiple domains, in non-military samples. Some of these relationships appeared to be mildly impacted by controlling for performance validity or symptom validity, however this was based on the relatively few studies using validity tests. More research including PVTs and SVTs whilst examining the relationship between emotional symptoms and cognitive outcomes is needed.

Examining the Clinical Utility of Selected Memory-Based Embedded Performance Validity Tests in Neuropsychological Assessment of Patients with Multiple Sclerosis

Within the neuropsychological assessment, clinicians are responsible for ensuring the validity of obtained cognitive data. As such, increased attention is being paid to performance validity in patients with multiple sclerosis (pwMS). Experts have proposed batteries of neuropsychological tests for use in this population, though none contain recommendations for standalone performance validity tests (PVTs). The California Verbal Learning Test, Second Edition (CVLT-II) and Brief Visuospatial Memory Test, Revised (BVMT-R)—both of which are included in the aforementioned recommended neuropsychological batteries—include previously validated embedded PVTs (which offer some advantages, including expedience and reduced costs), with no prior work exploring their utility in pwMS. The purpose of the present study was to determine the potential clinical utility of embedded PVTs to detect the signal of non-credibility as operationally defined by below criterion standalone PVT performance. One hundred thirty-three (133) patients (M age = 48.28; 76.7% women; 85.0% White) with MS were referred for neuropsychological assessment at a large, Midwestern academic medical center. Patients were placed into “credible” (n = 100) or “noncredible” (n = 33) groups based on a standalone PVT criterion. Classification statistics for four CVLT-II and BVMT-R PVTs of interest in isolation were poor (AUCs = 0.58–0.62). Several arithmetic and logistic regression-derived multivariate formulas were calculated, all of which similarly demonstrated poor discriminability (AUCs = 0.61–0.64). Although embedded PVTs may arguably maximize efficiency and minimize test burden in pwMS, common ones in the CVLT-II and BVMT-R may not be psychometrically appropriate, sufficiently sensitive, nor substitutable for standalone PVTs in this population. Clinical neuropsychologists who evaluate such patients are encouraged to include standalone PVTs in their assessment batteries to ensure that clinical care conclusions drawn from neuropsychological data are valid.

Nonmemory Composite Embedded Performance Validity Formulas in Patients with Multiple Sclerosis

OBJECTIVE

Research regarding performance validity tests (PVTs) in patients with multiple sclerosis (MS) is scant, with recommended batteries for neuropsychological evaluations in this population lacking suggestions to include PVTs. Moreover, limited work has examined embedded PVTs in this population. As previous investigations indicated that nonmemory-based embedded PVTs provide clinical utility in other populations, this study sought to determine if a logistic regression-derived PVT formula can be identified from selected nonmemory variables in a sample of patients with MS.

METHOD

A total of 184 patients (M age = 48.45; 76.6% female) with MS were referred for neuropsychological assessment at a large, Midwestern academic medical center. Patients were placed into "credible" (n = 146) or "noncredible" (n = 38) groups according to performance on standalone PVT. Missing data were imputed with HOTDECK.

RESULTS

Classification statistics for a variety of embedded PVTs were examined, with none appearing psychometrically appropriate in isolation (areas under the curve [AUCs] = .48-.64). Four exponentiated equations were created via logistic regression. Six, five, and three predictor equations yielded acceptable discriminability (AUC = .71-.74) with modest sensitivity (.34-.39) while maintaining good specificity (≥.90). The two predictor equation appeared unacceptable (AUC = .67).

CONCLUSIONS

Results suggest that multivariate combinations of embedded PVTs may provide some clinical utility while minimizing test burden in determining performance validity in patients with MS. Nonetheless, the authors recommend routine inclusion of several PVTs and utilization of comprehensive clinical judgment to maximize signal detection of noncredible performance and avoid incorrect conclusions. Clinical implications, limitations, and avenues for future research are discussed.

Five shades of gray: Conceptual and methodological issues around multivariate models of performance validity

OBJECTIVE

This study was designed to empirically investigate the signal detection profile of various multivariate models of performance validity tests (MV-PVTs) and explore several contested assumptions underlying validity assessment in general and MV-PVTs specifically.

METHOD

Archival data were collected from 167 patients (52.4%male; MAge = 39.7) clinicially evaluated subsequent to a TBI. Performance validity was psychometrically defined using two free-standing PVTs and five composite measures, each based on five embedded PVTs.

RESULTS

MV-PVTs had superior classification accuracy compared to univariate cutoffs. The similarity between predictor and criterion PVTs influenced signal detection profiles. False positive rates (FPR) in MV-PVTs can be effectively controlled using more stringent multivariate cutoffs. In addition to Pass and Fail, Borderline is a legitimate third outcome of performance validity assessment. Failing memory-based PVTs was associated with elevated self-reported psychiatric symptoms.

CONCLUSIONS

Concerns about elevated FPR in MV-PVTs are unsubstantiated. In fact, MV-PVTs are psychometrically superior to individual components. Instrumentation artifacts are endemic to PVTs, and represent both a threat and an opportunity during the interpretation of a given neurocognitive profile. There is no such thing as too much information in performance validity assessment. Psychometric issues should be evaluated based on empirical, not theoretical models. As the number/severity of embedded PVT failures accumulates, assessors must consider the possibility of non-credible presentation and its clinical implications to neurorehabilitation.

Psychometric implications of failure on one performance validity test: a cross-validation study to inform criterion group definition

Introduction: Research to date has supported the use of multiple performance validity tests (PVTs) for determining validity status in clinical settings. However, the implications of including versus excluding patients failing one PVT remains a source of debate, and methodological guidelines for PVT research are lacking. This study evaluated three validity classification approaches (i.e. 0 vs. ≥2, 0-1 vs. ≥2, and 0 vs. ≥1 PVT failures) using three reference standards (i.e. criterion PVT groupings) to recommend approaches best suited to establishing validity groups in PVT research methodology.Method: A mixed clinical sample of 157 patients was administered freestanding (Medical Symptom Validity Test, Dot Counting Test, Test of Memory Malingering, Word Choice Test), and embedded PVTs (Reliable Digit Span, RAVLT Effort Score, Stroop Word Reading, BVMT-R Recognition Discrimination) during outpatient neuropsychological evaluation. Three reference standards (i.e. two freestanding and three embedded PVTs from the above list) were created. Rey 15-Item Test and RAVLT Forced Choice were used solely as outcome measures in addition to two freestanding PVTs not employed in the reference standard. Receiver operating characteristic curve analyses evaluated classification accuracy using the three validity classification approaches for each reference standard.Results: When patients failing only one PVT were excluded or classified as valid, classification accuracy ranged from acceptable to excellent. However, classification accuracy was poor to acceptable when patients failing one PVT were classified as invalid. Sensitivity/specificity across two of the validity classification approaches (0 vs. ≥2; 0-1 vs. ≥2) remained reasonably stable.Conclusions: These results reflect that both inclusion and exclusion of patients failing one PVT are acceptable approaches to PVT research methodology and the choice of method likely depends on the study rationale. However, including such patients in the invalid group yields unacceptably poor classification accuracy across a number of psychometrically robust outcome measures and therefore is not recommended.

The Impact of Learning and Memory on Performance Validity Tests in a Mixed Clinical Pediatric Population

OBJECTIVE

This study examined the degree to which verbal and visuospatial memory abilities influence performance validity test (PVT) performance in a mixed clinical pediatric sample.

METHOD

Data from 252 consecutive clinical pediatric cases (Mage=11.23 years, SD=4.02; 61.9% male) seen for outpatient neuropsychological assessment were collected. Measures of learning and memory (e.g., The California Verbal Learning Test-Children's Version; Child and Adolescent Memory Profile [ChAMP]), performance validity (Test of Memory Malingering Trial 1 [TOMM T1]; Wechsler Intelligence Scale for Children-Fifth Edition [WISC-V] or Wechsler Adult Intelligence Scale-Fourth Edition Digit Span indices; ChAMP Overall Validity Index), and intellectual abilities (e.g., WISC-V) were included.

RESULTS

Learning/memory abilities were not significantly correlated with TOMM T1 and accounted for relatively little variance in overall TOMM T1 performance (i.e., ≤6%). Conversely, ChAMP Validity Index scores were significantly correlated with verbal and visual learning/memory abilities, and learning/memory accounted for significant variance in PVT performance (12%-26%). Verbal learning/memory performance accounted for 5%-16% of the variance across the Digit Span PVTs. No significant differences in TOMM T1 and Digit Span PVT scores emerged between verbal/visual learning/memory impairment groups. ChAMP validity scores were lower for the visual learning/memory impairment group relative to the nonimpaired group.

CONCLUSIONS

Findings highlight the utility of including PVTs as standard practice for pediatric populations, particularly when memory is a concern. Consistent with the adult literature, TOMM T1 outperformed other PVTs in its utility even among the diverse clinical sample with/without learning/memory impairment. In contrast, use of Digit Span indices appear to be best suited in the presence of visuospatial (but not verbal) learning/memory concerns. Finally, the ChAMP's embedded validity measure was most strongly impacted by learning/memory performance.

Establishing the base rate of performance invalidity in a clinical electrical injury sample: Implications for neuropsychological test performance

Objective: The base rate of neuropsychological performance invalidity in electrical injury, a clinically-distinct and frequently compensation-seeking population, is not well established. This study determined the base rate of performance invalidity in a large electrical injury sample, and examined patient characteristics, injury parameters, and neuropsychological test performance based on validity status.Method: This cross-sectional study included data from 101 patients with electrical injury consecutively referred for post-acute neuropsychological evaluation. Eighty-five percent of the sample was compensation-seeking. Multiple performance validity tests (PVTs) were administered as part of standard clinical evaluation. For patients with four or more PVTs, valid performance was operationalized as less than or equal to one PVT failure and invalid performance as two or more failures.Results: Frequency analysis revealed 66% (n = 67) had valid performance while 29% (n = 29) demonstrated probable invalid performance; the remaining 5% (n = 5) had indeterminate validity. No significant differences in demographics or injury parameters emerged between validity groups (0 vs. 1 vs. ≥2 PVT failures). In contrast, the electrical injury group with invalid performance performed significantly worse across tests of processing speed and executive abilities than those with valid performance (ps< .05, η_p² = .19-.25).Conclusions: The current study is the first to establish the base rate of neuropsychological performance invalidity in electrical injury survivors using empirical methods and current practice standards. Patient and clinical variables, including compensation-seeking status, did not differ between validity groups; however, neuropsychological test performance did, supporting the need for multi-method, objective performance validity assessment.

Performance validity and symptom validity tests: Are they measuring different constructs?

OBJECTIVE

To evaluate the relationships among performance validity, symptom validity, symptom self-report, and objective cognitive testing.

METHOD

Combat Veterans (N = 338) completed a neurocognitive assessment battery and several self-report symptom measures assessing depression, posttraumatic stress disorder (PTSD) symptoms, sleep quality, pain interference, and neurobehavioral complaints. All participants also completed two performance validity tests (PVTs) and one stand-alone symptom validity test (SVT) along with two embedded SVTs.

RESULTS

Results of an exploratory factor analysis revealed a three-factor solution: performance validity, cognitive performance, and symptom report (SVTs loaded on the third factor). Results of t tests demonstrated that participants who failed PVTs displayed significantly more severe symptoms and significantly worse performance on most measures of neurocognitive functioning compared to those who passed. Participants who failed a stand-alone SVT also reported significantly more severe symptomatology on all symptom report measures, but the pattern of cognitive performance differed based on the selected SVT cutoff. Multiple linear regressions revealed that both SVT and PVT failure explained unique variance in symptom report, but only PVT failure significantly predicted cognitive performance.

CONCLUSIONS

Performance and symptom validity tests measure distinct but related constructs. SVTs and PVTs are significantly related to both cognitive performance and symptom report; however, the relationship between symptom validity and symptom report is strongest. SVTs are also differentially related to cognitive performance and symptom report based on the utilized cutoff score. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Comparing the Psychometric Properties of Eight Embedded Performance Validity Tests in the Rey Auditory Verbal Learning Test, Wechsler Memory Scale Logical Memory, and Brief Visuospatial Memory Test–Revised Recognition Trials for Detecting Invalid Neuropsychological Test Performance

This cross-sectional study evaluated eight embedded performance validity tests (PVTs) previously derived from the Rey Auditory Verbal Learning Test (RAVLT), Wechsler Memory Scale–IV–Logical Memory (LM), and Brief Visuospatial Memory Test–Revised (BVMT-R) recognition trials among a single mixed clinical sample of 108 neuropsychiatric patients (83 valid/25 invalid) with ( n = 54) and without ( n = 29) mild neurocognitive disorder. Among the overall sample, all eight recognition PVTs significantly differentiated valid from invalid performance (areas under the curve [AUCs] = .64-.81) with 26% to 44% sensitivity (≥89% specificity) at optimal cut-scores depending on the specific PVT. After subdividing the sample by cognitive impairment status, all eight PVTs continued to reliably identify invalid performance (AUC = .68-.91) with markedly increased sensitivities of 56% to 80% (≥89% specificity) in the unimpaired group. In contrast, among those with mild neurocognitive disorder, RAVLT False Positives and LM became nonsignificant, whereas the other six PVTs remained significant (AUC = .64-.77), albeit with reduced sensitivities of 32% to 44% (≥89% specificity) at optimal cut-scores. Taken together, results cross-validated BVMT-R and most RAVLT recognition indices as effective embedded PVTs for identifying invalid neuropsychological test performance with diverse populations including examinees with and without suspected mild neurocognitive disorder, whereas LM had more limited utility as an embedded PVT, particularly when mild neurocognitive disorder was present.

The Minnesota Multiphasic Personality Inventory-2-RF in Treatment-Seeking Veterans with History of Mild Traumatic Brain Injury

OBJECTIVE

This study examined the Minnesota Multiphasic Personality Inventory-Second Edition-Restructured Form (MMPI-2-RF) to better understand symptom presentation in a sample of treatment-seeking Operation Enduring Freedom/Operation Iraqi Freedom (OEF/OIF) Veterans with self-reported history of mild traumatic brain injury (mTBI).

METHOD

Participants underwent a comprehensive clinical neuropsychological battery including performance and symptom validity measures and self-report measures of depressive, posttraumatic, and post-concussive symptomatology. Those with possible symptom exaggeration (SE+) on the MMPI-2-RF were compared with those without (SE-) with regard to injury, psychiatric, validity, and cognitive variables.

RESULTS

Between 50% and 87% of participants demonstrated possible symptom exaggeration on one or more MMPI-2-RF validity scales, and a large majority were elevated on content scales related to cognitive, somatic, and emotional complaints. The SE+ group reported higher depressive, posttraumatic, and post-concussive symptomatology, had higher scores on symptom validity measures, and performed more poorly on neuropsychological measures compared with the SE- group. There were no group differences with regard to injury variables or performance validity measures. Participants were more likely to exhibit possible symptom exaggeration on cognitive/somatic compared with traditional psychopathological validity scales.

CONCLUSIONS

A sizable portion of treatment-seeking OEF/OIF Veterans demonstrated possible symptom exaggeration on MMPI-2-RF validity scales, which was associated with elevated scores on self-report measures and poorer cognitive performance, but not higher rates of performance validity failure, suggesting symptom and performance validity are distinct concepts. These findings have implications for the interpretation of clinical data in the context of possible symptom exaggeration and treatment in Veterans with persistent post-concussive symptoms.

Research methods in performance validity testing studies: Criterion grouping approach impacts study outcomes

OBJECTIVE

Performance validity test (PVT) research studies commonly utilize a known-groups design, but the criterion grouping approaches within the design vary greatly from one study to another. At the present time, it is unclear as to what degree different criterion grouping approaches might impact PVT classification accuracy statistics.

METHOD

To analyze this, the authors used three different criterion grouping approaches to examine how classification accuracy statistics of a PVT (Word Choice Test; WCT) would differ. The three criterion grouping approaches included: (1) failure of 2+ PVTs versus failure of 0 PVTs, (2) failure of 2+ PVTs versus failure of 0-1 PVT, and (3) failure of a stand-alone PVT versus passing of a stand-alone PVT (Test of Memory Malingering).

RESULTS

When setting specificity at ≥.90, WCT cutoff scores ranged from 41 to 44 and associated sensitivity values ranged from .64 to .88, depending on the criterion grouping approach that was utilized.

CONCLUSIONS

When using a stand-alone PVT to define criterion group status, classification accuracy rates of the WCT were higher than expected, likely due to strong correlations between the reference PVT and the WCT. This held true even when considering evidence that this grouping approach results in higher rates of criterion group misclassification. Conversely, when using criterion grouping approaches that utilized failure of 2+ PVTs, accuracy rates were more consistent with expectations. These findings demonstrate that criterion grouping approaches can impact PVT classification accuracy rates and resultant cutoff scores. Strengths, weaknesses, and practical implications of each of the criterion grouping approaches are discussed.

Use of Latent Class Analysis to define groups based on validity, cognition, and emotional functioning

OBJECTIVE

Latent Class Analysis (LCA) was used to classify a heterogeneous sample of neuropsychology data. In particular, we used measures of performance validity, symptom validity, cognition, and emotional functioning to assess and describe latent groups of functioning in these areas.

METHOD

A data-set of 680 neuropsychological evaluation protocols was analyzed using a LCA. Data were collected from evaluations performed for clinical purposes at an urban medical center.

RESULTS

A four-class model emerged as the best fitting model of latent classes. The resulting classes were distinct based on measures of performance validity and symptom validity. Class A performed poorly on both performance and symptom validity measures. Class B had intact performance validity and heightened symptom reporting. The remaining two Classes performed adequately on both performance and symptom validity measures, differing only in cognitive and emotional functioning. In general, performance invalidity was associated with worse cognitive performance, while symptom invalidity was associated with elevated emotional distress.

CONCLUSIONS

LCA appears useful in identifying groups within a heterogeneous sample with distinct performance patterns. Further, the orthogonal nature of performance and symptom validities is supported.

Assessment of Performance Validity Using Embedded Saccadic and Manual Indices on a Continuous Performance Test

OBJECTIVE

In addition to manual (i.e., "button press") metrics, oculomotor metrics demonstrate considerable promise as tools for detecting invalid responding in neurocognitive assessment. This study was conducted to evaluate saccadic and manual metrics from a computerized continuous performance test as embedded indices of performance validity.

METHOD

Receiver operating characteristic analyses, logistic regressions, and ANOVAs were performed to evaluate saccadic and manual metrics in classification of healthy adults instructed to feign deficits ("Fake Bad" group; n = 24), healthy adults instructed to perform their best ("Best Effort" group; n = 26), and adults with a history of mild traumatic brain injury (TBI) who passed a series of validity indices ("mTBI-Pass" group; n = 19).

RESULTS

Several saccadic and manual metrics achieved outstanding classification accuracy between Fake Bad versus Best Effort and mTBI-Pass groups, including variability (consistency) of saccadic and manual response time (RT), saccadic commission errors, and manual omission errors. Very large effect sizes were obtained between Fake Bad and Best Effort groups (Cohen's d range: 1.89-2.90; r range: .75-.78) as well as between Fake Bad and mTBI-Pass groups (Cohen's d range: 1.32-2.21; r range: .69-.71). The Fake Bad group consistently had higher saccadic and manual RT variability, more saccadic commission errors, and more manual omission errors than the Best Effort and mTBI-Pass groups.

CONCLUSIONS

These findings are the first to demonstrate that eye movements can be used to detect invalid responding in neurocognitive assessment. These results also provide compelling evidence that concurrently measured saccadic and manual metrics can detect invalid responding with high levels of sensitivity and specificity.

Rates of Abnormally Low TOPF Word Reading Scores in Individuals Failing Versus Passing Performance Validity Testing

The present study examined the impact of performance validity test (PVT) failure on the Test of Premorbid Functioning (TOPF) in a sample of 252 neuropsychological patients. Word reading performance differed significantly according to PVT failure status, and number of PVTs failed accounted for 7.4% of the variance in word reading performance, even after controlling for education. Furthermore, individuals failing ≥2 PVTs were twice as likely as individuals passing all PVTs (33% vs. 16%) to have abnormally low obtained word reading scores relative to demographically predicted scores when using a normative base rate of 10% to define abnormality. When compared with standardization study clinical groups, those failing ≥2 PVTs were twice as likely as patients with moderate to severe traumatic brain injury and as likely as patients with Alzheimer’s dementia to obtain abnormally low TOPF word reading scores. Findings indicate that TOPF word reading based estimates of premorbid functioning should not be interpreted in individuals invalidating cognitive testing.