Feigning or forgetfulness: The effect of memory impairment severity on word choice test performance

Does neuropsychological intraindividual variability index cognitive dysfunction, an invalid presentation, or both? Preliminary findings from a mixed clinical older adult veteran sample

INTRODUCTION

Intraindividual variability across a battery of neuropsychological tests (IIV-dispersion) can reflect normal variation in scores or arise from cognitive impairment. An alternate interpretation is IIV-dispersion reflects reduced engagement/invalid test data, although extant research addressing this interpretation is significantly limited.

METHOD

We used a sample of 97 older adult (mean age: 69.92), predominantly White (57%) or Black/African American (34%), and predominantly cis-gender male (87%) veterans. Examinees completed a comprehensive neuropsychological battery, including measures of reduced engagement/invalid test data (a symptom validity test [SVT], multiple performance validity tests [PVTs]), as part of a clinical evaluation. IIV-dispersion was indexed using the coefficient of variance (CoV). We tested 1) the relationships of raw scores and "failures" on SVT/PVTs with IIV-dispersion, 2) the relationship between IIV-dispersion and validity/neurocognitive disorder status, and 3) whether IIV-dispersion discriminated the validity/neurocognitive disorder groups using receiver operating characteristic (ROC) curves.

RESULTS

IIV-dispersion was significantly and independently associated with a selection of PVTs, with small to very large effect sizes. Participants with invalid profiles and cognitively impaired participants with valid profiles exhibited medium to large (d = .55-1.09) elevations in IIV-dispersion compared to cognitively unimpaired participants with valid profiles. A non-significant but small to medium (d = .35-.60) elevation in IIV-dispersion was observed for participants with invalid profiles compared to those with a neurocognitive disorder. IIV-dispersion was largely accurate at differentiating participants without a neurocognitive disorder from invalid participants and those with a neurocognitive disorder (areas under the Curve [AUCs]=.69-.83), while accuracy was low for differentiating invalid participants from those with a neurocognitive disorder (AUCs=.58-.65).

CONCLUSIONS

These preliminary data suggest IIV-dispersion may be sensitive to both neurocognitive disorders and compromised engagement. Clinicians and researchers should exercise due diligence and consider test validity (e.g. PVTs, behavioral signs of engagement) as an alternate explanation prior to interpretation of intraindividual variability as an indicator of cognitive impairment.

Effect of processing speed and memory performance on classification accuracy of the dot counting test in a mixed neuropsychiatric sample

OBJECTIVE

This study examined the impact of impairment in two specific cognitive abilities, processing speed and memory, on Dot Counting Test (DCT) classification accuracy by evaluating performance validity classification accuracy across cognitively unimpaired, single-domain impairment, and multidomain impairment subgroups within a mixed clinical sample.

METHOD

Cross-sectional data were analyzed from 348 adult outpatients classified as valid (n = 284) or invalid (n = 64) based on four independent criterion performance validity tests (PVTs). Unimpaired (n = 164), single-domain processing speed impairment (n = 24), single-domain memory impairment (n = 53), and multidomain processing speed and memory impairment (n = 43) clinical subgroups were established among the valid group. Both the traditional DCT E-score and unrounded E-score were examined.

RESULTS

Overall, the DCT demonstrated acceptable to excellent classification accuracy across the unimpaired (area under the curve [AUC] traditional E-score=.855; unrounded E-score=.855) and single-domain impairment groups (traditional E-score AUCs = .690-.754; unrounded E-score AUCs = .692-747). However, it did not reliably discriminate the multidomain processing speed and memory impairment group from the invalid performers (traditional and unrounded E-scores AUC = .557).

CONCLUSIONS

Findings support the DCT as a non-memory-based freestanding PVT for use with single-domain cognitive impairment, with traditional E-score ≥17 (unrounded E-score ≥16.95) recommended for those with memory impairment and traditional E-score ≥19 (unrounded ≥18.08) with processing speed impairment. Moreover, results replicated previously established optimal cutoffs for unimpaired groups using both the traditional (≥14) and unrounded (≥13.84) E-scores. However, the DCT did not reliably discriminate between invalid performance and multidomain cognitive impairment, indicating caution is warranted when using the DCT with patients suspected of greater cognitive impairment.

The impact of visual memory impairment on Victoria Symptom Validity Test performance: A known-groups analysis

OBJECTIVE

We assessed the effect of visual learning and recall impairment on Victoria Symptom Validity Test (VSVT) accuracy and response latency for Easy, Difficult, and Total Items.

METHOD

A sample of 163 adult patients administered the VSVT and Brief Visuospatial Memory Test-Revised were classified as valid (114/163) or invalid (49/163) groups via independent criterion performance validity tests (PVTs). Classification accuracies for all VSVT indices were examined for the overall sample, and separately for subgroups based on visual memory functioning.

RESULTS

In the overall sample, all indices produced acceptable classification accuracy (areas under the curve [AUCs] ≥ 0.79). When stratified by visual learning/recall impairment, accuracy indices yielded acceptable classification for both the unimpaired (AUCs ≥0.79) and impaired subsamples (AUCs ≥0.75). Latency indices had acceptable classification accuracy for the unimpaired subsample (AUCs ≥0.74), but accuracy and sensitivity dropped for the impaired sample (AUCs ≥0.67).

CONCLUSIONS

VSVT accuracy and response latency yielded acceptable classification accuracies in the overall sample, and this effect was maintained in those with and without visual learning/recall impairment for the accuracy indices. Findings indicate that the VSVT is a psychometrically robust PVT with largely invariant cut-scores, even in the presence of bona fide visual learning/recall impairment.

Performance Validity Test Failure in the Clinical Population: A Systematic Review and Meta-Analysis of Prevalence Rates

Performance validity tests (PVTs) are used to measure the validity of the obtained neuropsychological test data. However, when an individual fails a PVT, the likelihood that failure truly reflects invalid performance (i.e., the positive predictive value) depends on the base rate in the context in which the assessment takes place. Therefore, accurate base rate information is needed to guide interpretation of PVT performance. This systematic review and meta-analysis examined the base rate of PVT failure in the clinical population (PROSPERO number: CRD42020164128). PubMed/MEDLINE, Web of Science, and PsychINFO were searched to identify articles published up to November 5, 2021. Main eligibility criteria were a clinical evaluation context and utilization of stand-alone and well-validated PVTs. Of the 457 articles scrutinized for eligibility, 47 were selected for systematic review and meta-analyses. Pooled base rate of PVT failure for all included studies was 16%, 95% CI [14, 19]. High heterogeneity existed among these studies (Cochran's Q = 697.97, p < .001; I2 = 91%; τ2 = 0.08). Subgroup analysis indicated that pooled PVT failure rates varied across clinical context, presence of external incentives, clinical diagnosis, and utilized PVT. Our findings can be used for calculating clinically applied statistics (i.e., positive and negative predictive values, and likelihood ratios) to increase the diagnostic accuracy of performance validity determination in clinical evaluation. Future research is necessary with more detailed recruitment procedures and sample descriptions to further improve the accuracy of the base rate of PVT failure in clinical practice.

Cross-validation of the Trail Making Test as a non-memory-based embedded performance validity test among veterans with and without cognitive impairment

OBJECTIVE

This study cross-validated multiple Trail Making Test (TMT) Parts A and B scores as non-memory-based embedded performance validity tests (PVTs) for detecting invalid neuropsychological performance among veterans with and without cognitive impairment.

METHOD

Data were collected from a demographically and diagnostically diverse mixed clinical sample of 100 veterans undergoing outpatient neuropsychological evaluation at a Southwestern VA Medical Center. As part of a larger battery of neuropsychological tests, all veterans completed TMT A and B and four independent criterion PVTs, which were used to classify veterans into valid (n = 75) and invalid (n = 25) groups. Among the valid group 47% (n = 35) were cognitively impaired.

RESULTS

Among the overall sample, all embedded PVTs derived from TMT A and B raw and demographically corrected T-scores significantly differed between validity groups (ηp2 = .21-.31) with significant areas under the curve (AUCs) of .72-.78 and 32-48% sensitivity (≥91% specificity) at optimal cut-scores. When subdivided by cognitive impairment status (i.e., valid-unimpaired vs. invalid; valid-impaired vs. invalid), all TMT scores yielded significant AUCs of .80-.88 and 56%-72% sensitivity (≥90% specificity) at optimal cut-scores. Among veterans with cognitive impairment, neither TMT A or B raw scores were able to significantly differentiate the invalid from the valid-cognitively impaired group; however, demographically corrected T-scores were able to significantly differentiate groups but had poor classification accuracy (AUCs = .66-.68) and reduced sensitivity of 28%-44% (≥91% specificity).

CONCLUSIONS

Embedded PVTs derived from TMT Parts A and B raw and T-scores were able to accurately differentiate valid from invalid neuropsychological performance among veterans without cognitive impairment; however, the demographically corrected T-scores generally were more robust and consistent with prior studies compared to raw scores. By contrast, TMT embedded PVTs had poor accuracy and low sensitivity among veterans with cognitive impairment, suggesting limited utility as PVTs among populations with cognitive dysfunction.

Using the Grooved Pegboard Test as an Embedded Validity Indicator in a Mixed Neuropsychiatric Sample with Varying Cognitive Impairment: Cross-Validation Problems

Embedded validity indicators (EVIs) derived from motor tests have received less empirical attention than those derived from tests of other neuropsychological abilities, particularly memory. Preliminary evidence suggests that the Grooved Pegboard Test (GPB) may function as an EVI, but existing studies were largely conducted using simulators and population samples without cognitive impairment. In this study we aimed to evaluate the GPB's classification accuracy as an EVI among a mixed clinical neuropsychiatric sample with and without cognitive impairment. This cross-sectional study comprised 223 patients clinically referred for neuropsychological testing. GPB raw and T-scores for both dominant and nondominant hands were examined as EVIs. A known-groups design, based on ≤1 failure on a battery of validated, independent criterion PVTs, showed that GPB performance differed significantly by validity group. Within the valid group, receiver operating characteristic curve analyses revealed that only the dominant hand raw score displayed acceptable classification accuracy for detecting invalid performance (area under curve [AUC] = .72), with an optimal cut-score of ≥106 seconds (33% sensitivity/88% specificity). All other scores had marginally lower classification accuracy (AUCs = .65-.68) for differentiating valid from invalid performers. Therefore, the GPB demonstrated limited utility as an EVI in a clinical sample containing patients with bona fide cognitive impairment.

The TOMM1 discrepancy index (TDI): A new performance validity test (PVT) that differentiates between invalid cognitive testing and those diagnosed with dementia

There is a need to develop performance validity tests (PVTs) that accurately identify those with severe cognitive decline but also remain sensitive to those suspected of invalid cognitive testing. The TOMM1 Discrepancy Index (TDI) attempts to address both of these issues. Veterans diagnosed with dementia (n = 251) were administered TOMM1 and the MSVT in order to develop the TDI (TOMM1 percent correct minus MSVT Free Recall percent correct). Cut offs based on the dementia sample were then used to identify those in the non-dementia sample (n = 1,226) suspected of invalid test performance (n = 401). Combining TOMM1 and the TDI in the dementia sample greatly reduced the false positive rate (specificity = 0.97) at a cut off of 28 points or less on the TDI. Those suspected of invalid testing were identified at much higher rates (sensitivity = 0.75) compared to the MSVT genuine memory impairment profile (GMIP, sensitivity = 0.49). By utilizing a neurologically plausible pattern of scores across two PVTs, the TDI correctly classified those with dementia and identified a large percentage with invalid test performance. PVTs utilizing a complex pattern of performance may help reduce one's ability to fabricate cognitive deficits.

Comprehensive Analysis of MMPI-2-RF Symptom Validity Scales and Performance Validity Test Relationships in a Diverse Mixed Neuropsychiatric Setting

The utility of symptom (SVT) and performance (PVT) validity tests has been independently established in neuropsychological evaluations, yet research on the relationship between these two types of validity indices is limited to circumscribed populations and measures. This study examined the relationship between SVTs on the Minnesota Multiphasic Personality Inventory-2-Restructured Form (MMPI-2-RF) and PVTs in a mixed neuropsychiatric setting. This cross-sectional study included data from 181 diagnostically and demographically diverse patients with neuropsychiatric conditions referred for outpatient clinical neuropsychological evaluation at an academic medical center. All patients were administered a uniform neuropsychological battery, including the MMPI-2-RF and five PVTs (i.e., Dot Counting Test; Medical Symptom Validity Test; Reliable Digit Span; Test of Memory Malingering-Trial 1; Word Choice Test). Nonsignificant associations emerged between SVT and PVT performance. Although the Response Bias Scale was most predictive of PVT performance, MMPI-2-RF SVTs generally had low classification accuracy for predicting PVT performance. Neuropsychological test performance was related to MMPI-2-RF SVT status only when overreporting elevations were at extreme scores. The current study further supports that SVTs and PVTs measure unique and dissociable constructs among diverse patients with neuropsychiatric conditions, consistent with literature from other clinical contexts. Therefore, objective evidence of symptom overreporting on MMPI-2-RF SVTs cannot be interpreted as definitively indicating invalid performance on tests of neurocognitive abilities. As such, clinicians should include both SVTs and PVTs as part of a comprehensive neuropsychological evaluation as they provide unique information regarding performance and symptom validity.

A case report and literature review of cognitive malingering and psychopathology

Malingering of cognitive difficulties constitutes a major issue in psychiatric forensic settings. Here, we present a selective literature review related to the topic of cognitive malingering, psychopathology and their possible connections. Furthermore, we report a single case study of a 60-year-old man with a long and ongoing judicial history who exhibits a suspicious multi-domain neurocognitive disorder with significant reduction of autonomy in daily living, alongside a longtime history of depressive symptoms. Building on this, we suggest the importance of evaluating malingering conditions through both psychiatric and neuropsychological assessment tools. More specifically, the use of Performance Validity Tests (PVTs)-commonly but not quite correctly considered as tests of "malingering"-alongside the collection of clinical history and the use of routine psychometric testing, seems to be crucial in order to detect discrepancies between self-reported patient's symptoms, embedded validity indicators and psychometric results.

Multivariable analysis of the relative utility and additive value of eight embedded performance validity tests for classifying invalid neuropsychological test performance

INTRODUCTION

This study investigated a combination of eight embedded performance validity tests (PVTs) derived from commonly administered neuropsychological tests to optimize sensitivity/specificity for detecting invalid neuropsychological test performance. The goal of this study was to evaluate what combination of these common embedded PVTs that have the most robust predictive power for detecting invalid neuropsychological test performance in a single diverse clinical sample.

METHOD

Eight previously validated memory- and nonmemory-based embedded PVTs were examined among 231 patients undergoing neuropsychological evaluation. Patients were classified into valid/invalid groups based on four independent criterion PVTs. Embedded PVT accuracy was assessed using standard and stepwise multiple logistic regression models.

RESULTS

Three PVTs, the Brief Visuospatial Memory Test-Revised Recognition Discrimination (BVMT-R-RD), Rey Auditory Verbal Learning Test Forced Choice, and WAIS-IV Digit Span Age Corrected Scaled Score, predicted 45.5% of the variance in validity group membership. BVMT-RD independently accounted for 32% of the variance in prediction of independent, criterion-defined validity group membership.

CONCLUSIONS

This study demonstrated the incremental predictive power of multiple embedded PVTs derived from common neuropsychological measures in detecting invalid test performance and those measures accounting for the greatest portion of the variance. These results provide guidance for evaluating the most fruitful embedded PVTs and proof of concept to better guide selection of embedded validity indices. Further, this offers clinicians an efficient, empirically derived approach to assessing performance validity when time restraints potentially limit the use of freestanding PVTs.

All of the accuracy in half of the time: Assessing abbreviated versions of the Test of Memory Malingering in the context of verbal and visual memory impairment

ObjectiveThe Test of Memory Malingering (TOMM) Trial 1 (T1) and errors on the first 10 items of T1 (T1-e10) were developed as briefer versions of the TOMM to minimize evaluation time and burden, although the effect of genuine memory impairment on these indices is not well established. This study examined whether increasing material-specific verbal and visual memory impairment affected T1 and T1-e10 performance and accuracy for detecting invalidity. Method: Data from 155 neuropsychiatric patients administered the TOMM, Rey Auditory Verbal Learning Test (RAVLT), and Brief Visuospatial Memory Test-Revised (BVMT-R) during outpatient evaluation were examined. Valid (N = 125) and invalid (N = 30) groups were established by four independent criterion performance validity tests. Verbal/visual memory impairment was classified as ≥37 T (normal memory); 30 T-36T (mild impairment); and ≤29 T (severe impairment). Results: Overall, T1 had outstanding accuracy, with 77% sensitivity/90% specificity. T1-e10 was less accurate but had excellent discriminability, with 60% sensitivity/87% specificity. T1 maintained excellent accuracy regardless of memory impairment severity, with 77% sensitivity/≥88% specificity and a relatively invariant cut-score even among those with severe verbal/visual memory impairment. T1-e10 had excellent classification accuracy among those with normal memory and mild impairment, but accuracy and sensitivity dropped with severe impairment and the optimal cut-score had to be increased to maintain adequate specificity. Conclusion: TOMM T1 is an effective performance validity test with strong psychometric properties regardless of material-specificity and severity of memory impairment. By contrast, T1-e10 functions relatively well in the context of mild memory impairment but has reduced discriminability with severe memory impairment.

Comparing the Independent and Aggregated Accuracy of Trial 1 and the First 10 TOMM Items for Detecting Invalid Neuropsychological Test Performance Across Civilian and Veteran Clinical Samples

Previous studies support using two abbreviated tests of the Test of Memory Malingering (TOMM), including (a) Trial 1 (T1) and (b) the number of errors on the first 10 items of T1 (T1e10), as performance validity tests (PVTs). In this study, we examined the independent and aggregated predictive utility of TOMM T1 and T1e10 for identifying invalid neuropsychological test performance across two clinical samples. We employed cross-sectional research to examine two independent and demographically diverse mixed samples of military veterans and civilians (VA = 108; academic medical center = 234) of patients who underwent neuropsychological evaluations. We determined validity groups by patient performance on four independent criterion PVTs. We established concordances between passing/failing the TOMM T1e10 and T1, followed by logistic regression to determine individual and aggregated accuracy of T1e10 and T1 for predicting validity group membership. Concordance between passing T1e10 and T1 was high, as was overall validity (87-98%) across samples. By contrast, T1e10 failure was more highly concordant with T1 failure (69-77%) than with overall invalidity status (59-60%) per criterion PVTs, whereas T1 failure was more highly concordant with invalidity status (72-88%) per criterion PVTs. Logistic regression analyses demonstrated similar results, with T1 accounting for more variance than T1e10. However, combining T1e10 and T1 accounted for the most variance of any model, with T1e10 and T1 each emerging as significant predictors. TOMM T1 and, to a lesser extent, T1e10 were significant predictors of independent criterion-derived validity status across two distinct clinical samples, but they did not offer improved classification accuracy when aggregated.

The Impact of Learning and Memory on Performance Validity Tests in a Mixed Clinical Pediatric Population

OBJECTIVE

This study examined the degree to which verbal and visuospatial memory abilities influence performance validity test (PVT) performance in a mixed clinical pediatric sample.

METHOD

Data from 252 consecutive clinical pediatric cases (Mage=11.23 years, SD=4.02; 61.9% male) seen for outpatient neuropsychological assessment were collected. Measures of learning and memory (e.g., The California Verbal Learning Test-Children's Version; Child and Adolescent Memory Profile [ChAMP]), performance validity (Test of Memory Malingering Trial 1 [TOMM T1]; Wechsler Intelligence Scale for Children-Fifth Edition [WISC-V] or Wechsler Adult Intelligence Scale-Fourth Edition Digit Span indices; ChAMP Overall Validity Index), and intellectual abilities (e.g., WISC-V) were included.

RESULTS

Learning/memory abilities were not significantly correlated with TOMM T1 and accounted for relatively little variance in overall TOMM T1 performance (i.e., ≤6%). Conversely, ChAMP Validity Index scores were significantly correlated with verbal and visual learning/memory abilities, and learning/memory accounted for significant variance in PVT performance (12%-26%). Verbal learning/memory performance accounted for 5%-16% of the variance across the Digit Span PVTs. No significant differences in TOMM T1 and Digit Span PVT scores emerged between verbal/visual learning/memory impairment groups. ChAMP validity scores were lower for the visual learning/memory impairment group relative to the nonimpaired group.

CONCLUSIONS

Findings highlight the utility of including PVTs as standard practice for pediatric populations, particularly when memory is a concern. Consistent with the adult literature, TOMM T1 outperformed other PVTs in its utility even among the diverse clinical sample with/without learning/memory impairment. In contrast, use of Digit Span indices appear to be best suited in the presence of visuospatial (but not verbal) learning/memory concerns. Finally, the ChAMP's embedded validity measure was most strongly impacted by learning/memory performance.

Examining Traditional and Novel Validity Indicators from the Medical Symptom Validity Test Across Levels of Verbal and Visual Memory Impairment

OBJECTIVE

This cross-sectional study examined accuracy of traditional Medical Symptom Validity Test (MSVT) validity indicators, including immediate recognition (IR), delayed recognition (DR), and consistency (CNS), as well as a novel indicator derived from the mean performance on IR, DR, and CNS across verbal, visual, and combined learning and memory impairment bands.

METHOD

A sample of 180 adult outpatients was divided into valid (n = 150) and invalid (n = 30) groups based on results of four independent criterion performance validity tests. Verbal and visual learning and recall were classified as indicative of no impairment, mild impairment, or severe impairment based on performance on the Rey Auditory Verbal Learning Test and Brief Visuospatial Memory Test-Revised, respectively.

RESULTS

In general, individual MSVT subtests were able to accurately classify performance as valid or invalid, even in the context of severe learning and memory deficits. However, as verbal and visual memory impairment increased, optimal MSVT cut-scores diverged from manual-specified cutoffs such that DR and CNS required cut-scores to be lowered to maintain adequate specificity. By contrast, the newly proposed scoring algorithm generally showed more robust psychometric properties across the memory impairment bands.

CONCLUSIONS

The mean performance index, a novel scoring algorithm using the mean of the three primary MSVT subtests, may be a more robust validity indicator than the individual MSVT subtests in the context of bona fide memory impairment.

A Systematic Review and Meta-Analysis of the Diagnostic Accuracy of the Advanced Clinical Solutions Word Choice Test as a Performance Validity Test

Thorough assessment of performance validity has become an established standard of practice in neuropsychological assessment. While there has been a large focus on the development and cross-validation of embedded performance validity tests (PVTs) in recent years, new freestanding PVTs have also been developed, including the Word Choice Test (WCT) as part of the Advanced Clinical Solutions Effort System. And, while the WCT's general utility for identifying invalid performance has been demonstrated in the ensuing decade since its initial publication, optimal cut-scores and associated psychometric properties have varied widely across studies. This study sought to synthesize the existing diagnostic accuracy literature regarding the WCT via a systematic review and to conduct a meta-analysis to determine the performance validity cut-score that best maximizes sensitivity while maintaining acceptable specificity. A systematic search of the literature resulted in 14 studies for synthesis, with eight of those available for meta-analysis. Meta-analytic results revealed an optimal cut-score of ≤ 42 with 54% sensitivity and 93% specificity for identifying invalid neuropsychological test performance. Collectively, the WCT demonstrated adequate diagnostic accuracy as a PVT across a variety of populations. Recommendations for future studies are also provided.

A Known-Groups Validation of the Medical Symptom Validity Test and Analysis of the Genuine Memory Impairment Profile

This study cross-validated the Medical Symptom Validity Test (MSVT) in a mixed neuropsychiatric sample and examined its accuracy for identifying invalid neuropsychological performance using a known-groups design. Cross-sectional data from 129 clinical patients who completed the MSVT were examined. Validity groups were established using six, independent criterion performance validity tests, which yielded 98 patients in the valid group and 31 in the invalid group. All MSVT subtest scores were significantly lower in the invalid group (η p2=.22-.39). Using published cut-scores, sensitivities of 42% to 71% were found among the primary effort subtests, and 74% sensitivity/90% specificity was observed for the overall MSVT. Among this sample, the MSVT component validity scales produced areas under the curve of .78-.86, suggesting moderate classification accuracy. At optimal cut-scores, the MSVT primary effort validity scales demonstrated 55% to 71% sensitivity/91% to 93% specificity, with the Consistency subtest exhibiting the strongest psychometric properties. The MSVT exhibited relatively robust sensitivity and specificity, supporting its utility as a briefer freestanding performance validity test to its predecessor, the Word Memory Test. Finally, the Genuine Memory Impairment Profile appears promising for patients with Major Neurocognitive Disorder, but is cautioned against for those without significant functional decline in activities of daily living at this time.