Not so fast! Limitations of processing speed and working memory indices as embedded performance validity tests in a mixed neuropsychiatric sample

The impact of visual memory impairment on Victoria Symptom Validity Test performance: A known-groups analysis

OBJECTIVE

We assessed the effect of visual learning and recall impairment on Victoria Symptom Validity Test (VSVT) accuracy and response latency for Easy, Difficult, and Total Items.

METHOD

A sample of 163 adult patients administered the VSVT and Brief Visuospatial Memory Test-Revised were classified as valid (114/163) or invalid (49/163) groups via independent criterion performance validity tests (PVTs). Classification accuracies for all VSVT indices were examined for the overall sample, and separately for subgroups based on visual memory functioning.

RESULTS

In the overall sample, all indices produced acceptable classification accuracy (areas under the curve [AUCs] ≥ 0.79). When stratified by visual learning/recall impairment, accuracy indices yielded acceptable classification for both the unimpaired (AUCs ≥0.79) and impaired subsamples (AUCs ≥0.75). Latency indices had acceptable classification accuracy for the unimpaired subsample (AUCs ≥0.74), but accuracy and sensitivity dropped for the impaired sample (AUCs ≥0.67).

CONCLUSIONS

VSVT accuracy and response latency yielded acceptable classification accuracies in the overall sample, and this effect was maintained in those with and without visual learning/recall impairment for the accuracy indices. Findings indicate that the VSVT is a psychometrically robust PVT with largely invariant cut-scores, even in the presence of bona fide visual learning/recall impairment.

Time and money: Exploring enhancements to performance validity research designs

INTRODUCTION

The study examined the effect of preparation time and financial incentives on healthy adults' ability to simulate traumatic brain injury (TBI) during neuropsychological evaluation.

METHOD

A retrospective comparison of two TBI simulator group designs: a traditional design employing a single-session of standard coaching immediately before participation (SIM-SC; n = 46) and a novel design that provided financial incentive and preparation time (SIM-IP; n = 49). Both groups completed an ecologically valid neuropsychological test battery that included widely-used cognitive tests and five common performance validity tests (PVTs).

RESULTS

Compared to SIM-SC, SIM-IP performed significantly worse and had higher rates of impairment on tests of processing speed and executive functioning (Trails A and B). SIM-IP were more likely than SIM-SC to avoid detection on one of the PVTs and performed somewhat better on three of the PVTs, but the effects were small and non-significant. SIM-IP did not demonstrate significantly higher rates of successful simulation (i.e., performing impaired on cognitive tests with <2 PVT failures). Overall, the rate of the successful simulation was ∼40% with a liberal criterion, requiring cognitive impairment defined as performance >1 SD below the normative mean. At a more rigorous criterion defining impairment (>1.5 SD below the normative mean), successful simulation approached 35%.

CONCLUSIONS

Incentive and preparation time appear to add limited incremental effect over traditional, single-session coaching analog studies of TBI simulation. Moreover, these design modifications did not translate to meaningfully higher rates of successful simulation and avoidance of detection by PVTs.

The impact of race and other demographic factors on the false positive rates of five embedded Performance Validity Tests (PVTs) in a Veteran sample

INTRODUCTION

It is common to use normative adjustments based on race to maintain accuracy when interpreting cognitive test results during neuropsychological assessment. However, embedded performance validity tests (PVTs) do not adjust for these racial differences and may result in elevated rates of false positives in African American/Black (AA) samples compared to European American/White (EA) samples.

METHODS

Veterans without Major Neurocognitive Disorder completed an outpatient neuropsychological assessment and were deemed to be performing in a valid manner (e.g., passing both the Test of Memory Malingering Trial 1 (TOMM1) and the Medical Symptom Validity Test (MSVT), (n = 531, EA = 473, AA = 58). Five embedded PVTs were administered to all patients: WAIS-III/IV Processing Speed Index (PSI), Brief Visuospatial Memory Test-Revised: Discrimination Index (BVMT-R), TMT-A (secs), California Verbal Learning Test-II (CVLT-II) Forced Choice, and WAIS-III/IV Digit Span Scaled Score. Individual PVT false positive rates, as well as the rate of failing two or more embedded PVTs, were calculated.

RESULTS

Failure rates of two embedded PVTs (PSI, TMT-A), and the total number of PVTs failed, were higher in the AA sample. The PSI and TMT-A remained significantly impacted by race after accounting for age, education, sex, and presence of Mild Neurocognitive Disorder. There were PVT failure rates greater than 10% (and considered false positives) in both groups (AA: PSI, TMT-A, and BVMT-R, 12-24%; EA: BVMT-R, 17%). Failing 2 or more PVTs (AA = 9%, EA = 4%) was impacted by education and Mild Neurocognitive Disorder but not by race.

CONCLUSIONS

Individual (timed) PVTs showed higher false positive rates in the AA sample even after accounting for demographic factors and diagnosis of Mild Neurocognitive Disorder. Requiring failure on 2 or more embedded PVTs reduced false positive rates to acceptable levels across both groups (10% or less) and was not significantly influenced by race.

Cross-validation of the Trail Making Test as a non-memory-based embedded performance validity test among veterans with and without cognitive impairment

OBJECTIVE

This study cross-validated multiple Trail Making Test (TMT) Parts A and B scores as non-memory-based embedded performance validity tests (PVTs) for detecting invalid neuropsychological performance among veterans with and without cognitive impairment.

METHOD

Data were collected from a demographically and diagnostically diverse mixed clinical sample of 100 veterans undergoing outpatient neuropsychological evaluation at a Southwestern VA Medical Center. As part of a larger battery of neuropsychological tests, all veterans completed TMT A and B and four independent criterion PVTs, which were used to classify veterans into valid (n = 75) and invalid (n = 25) groups. Among the valid group 47% (n = 35) were cognitively impaired.

RESULTS

Among the overall sample, all embedded PVTs derived from TMT A and B raw and demographically corrected T-scores significantly differed between validity groups (ηp2 = .21-.31) with significant areas under the curve (AUCs) of .72-.78 and 32-48% sensitivity (≥91% specificity) at optimal cut-scores. When subdivided by cognitive impairment status (i.e., valid-unimpaired vs. invalid; valid-impaired vs. invalid), all TMT scores yielded significant AUCs of .80-.88 and 56%-72% sensitivity (≥90% specificity) at optimal cut-scores. Among veterans with cognitive impairment, neither TMT A or B raw scores were able to significantly differentiate the invalid from the valid-cognitively impaired group; however, demographically corrected T-scores were able to significantly differentiate groups but had poor classification accuracy (AUCs = .66-.68) and reduced sensitivity of 28%-44% (≥91% specificity).

CONCLUSIONS

Embedded PVTs derived from TMT Parts A and B raw and T-scores were able to accurately differentiate valid from invalid neuropsychological performance among veterans without cognitive impairment; however, the demographically corrected T-scores generally were more robust and consistent with prior studies compared to raw scores. By contrast, TMT embedded PVTs had poor accuracy and low sensitivity among veterans with cognitive impairment, suggesting limited utility as PVTs among populations with cognitive dysfunction.

Validity assessment in Eastern Europe: cross-validation of the Dot Counting Test and MODEMM against the TOMM-1 and Rey-15 in a Romanian mixed clinical sample

OBJECTIVE

This study investigated performance validity in the understudied Romanian clinical population by exploring classification accuracies of the Dot Counting Test (DCT) and the first Romanian performance validity test (PVT) (Memory of Objects and Digits and Evaluation of Memory Malingering/MODEMM) in a heterogeneous clinical sample.

METHODS

We evaluated 54 outpatients (26 females; MAge = 62.02; SDAge = 12.3; MEducation = 2.41, SDEducation = 2.82) with the Test of Memory Malingering 1 (TOMM-1), Rey Fifteen Items Test (Rey-15) (free recall and recognition trials), DCT, MODEMM, and MMSE/MoCA as part of their neuropsychological assessment. Accuracy parameters and base failure rates were computed for the DCT and MODEMM indicators against the TOMM-1 and Rey-15. Two patient groups were constructed according to psychometrically defined credible/noncredible performance (i.e., pass/fail both TOMM-1 and Rey-15).

RESULTS

Similar to other cultures, a cutoff of ≥18 on the DCT E score produced the best combination between sensitivity (0.50-0.57) and specificity (≥0.90). MODEMM indicators based on recognition accuracy, inconsistencies, and inclusion false positives generated 0.75-0.86 sensitivities at ≥0.90 specificities. Multivariable models of MODEMM indicators reached perfect sensitivities at ≥0.90 specificities against two PVTs. Patients who failed the TOMM-1 and Rey-15 were significantly more likely to fail the DCT and MODEMM than patients who passed both PVTs.

CONCLUSIONS

Our results offer proof of concept for the DCT's cross-cultural validity and the applicability of the MODEMM on Romanian clinical examinees, further recommending the use of heterogeneous validity indicators in clinical assessments.

Multivariate models provide an effective psychometric solution to the variability in classification accuracy of D-KEFS Stroop performance validity cutoffs

ObjectiveThe study was designed to expand on the results of previous investigations on the D-KEFS Stroop as a performance validity test (PVT), which produced diverging conclusions. Method The classification accuracy of previously proposed validity cutoffs on the D-KEFS Stroop was computed against four different criterion PVTs in two independent samples: patients with uncomplicated mild TBI (n = 68) and disability benefit applicants (n = 49). Results Age-corrected scaled scores (ACSSs) ≤6 on individual subtests often fell short of specificity standards. Making the cutoffs more conservative improved specificity, but at a significant cost to sensitivity. In contrast, multivariate models (≥3 failures at ACSS ≤6 or ≥2 failures at ACSS ≤5 on the four subtests) produced good combinations of sensitivity (.39-.79) and specificity (.85-1.00), correctly classifying 74.6-90.6% of the sample. A novel validity scale, the D-KEFS Stroop Index correctly classified between 78.7% and 93.3% of the sample. Conclusions A multivariate approach to performance validity assessment provides a methodological safeguard against sample- and instrument-specific fluctuations in classification accuracy, strikes a reasonable balance between sensitivity and specificity, and mitigates the invalid before impaired paradox.

Feigning Adult ADHD on a Comprehensive Neuropsychological Test Battery: An Analogue Study

The evaluation of performance validity is an essential part of any neuropsychological evaluation. Validity indicators embedded in routine neuropsychological tests offer a time-efficient option for sampling performance validity throughout the assessment while reducing vulnerability to coaching. By administering a comprehensive neuropsychological test battery to 57 adults with ADHD, 60 neurotypical controls, and 151 instructed simulators, we examined each test's utility in detecting noncredible performance. Cut-off scores were derived for all available outcome variables. Although all ensured at least 90% specificity in the ADHD Group, sensitivity differed significantly between tests, ranging from 0% to 64.9%. Tests of selective attention, vigilance, and inhibition were most useful in detecting the instructed simulation of adult ADHD, whereas figural fluency and task switching lacked sensitivity. Five or more test variables demonstrating results in the second to fourth percentile were rare among cases of genuine adult ADHD but identified approximately 58% of instructed simulators.

Using the Grooved Pegboard Test as an Embedded Validity Indicator in a Mixed Neuropsychiatric Sample with Varying Cognitive Impairment: Cross-Validation Problems

Embedded validity indicators (EVIs) derived from motor tests have received less empirical attention than those derived from tests of other neuropsychological abilities, particularly memory. Preliminary evidence suggests that the Grooved Pegboard Test (GPB) may function as an EVI, but existing studies were largely conducted using simulators and population samples without cognitive impairment. In this study we aimed to evaluate the GPB's classification accuracy as an EVI among a mixed clinical neuropsychiatric sample with and without cognitive impairment. This cross-sectional study comprised 223 patients clinically referred for neuropsychological testing. GPB raw and T-scores for both dominant and nondominant hands were examined as EVIs. A known-groups design, based on ≤1 failure on a battery of validated, independent criterion PVTs, showed that GPB performance differed significantly by validity group. Within the valid group, receiver operating characteristic curve analyses revealed that only the dominant hand raw score displayed acceptable classification accuracy for detecting invalid performance (area under curve [AUC] = .72), with an optimal cut-score of ≥106 seconds (33% sensitivity/88% specificity). All other scores had marginally lower classification accuracy (AUCs = .65-.68) for differentiating valid from invalid performers. Therefore, the GPB demonstrated limited utility as an EVI in a clinical sample containing patients with bona fide cognitive impairment.

Attention Performance as an Embedded Validity Indicator in the Cognitive Assessment of Early Retirement Claimants

The assessment of performance validity is essential in any neuropsychological evaluation. However, relatively few measures exist that are based on attention performance embedded within routine cognitive tasks. The present study explores the potential value of a computerized attention test, the Cognitrone, as an embedded validity indicator in the neuropsychological assessment of early retirement claimants. Two hundred and sixty-five early retirement claimants were assessed with the Word Memory Test (WMT) and the Cognitrone. WMT scores were used as the independent criterion to determine performance validity. Speed and accuracy measures of the Cognitrone were analyzed in receiver operating characteristics (ROC) to classify group membership. The Cognitrone was sensitive in revealing attention deficits in early retirement claimants. Further, 54% (n = 143) of the individuals showed noncredible cognitive performance, whereas 46% (n = 122) showed credible cognitive performance. Individuals failing the performance validity assessment showed slower (AUC = 79.1%) and more inaccurate (AUC = 79.5%) attention performance than those passing the performance validity assessment. A compound score integrating speed and accuracy revealed incremental value as indicated by AUC = 87.9%. Various cut scores are suggested, resulting in equal rates of 80% sensitivity and specificity (cut score = 1.297) or 69% sensitivity with 90% specificity (cut score = 0.734). The present study supports the sensitivity of the Cognitrone for the assessment of attention deficits in early retirement claimants and its potential value as an embedded validity indicator. Further research on different samples and with multidimensional criteria for determining invalid performance are required before clinical application can be suggested.

A Direct Comparison of 10 WAIS-IV Digit Span Embedded Validity Indicators among a Mixed Neuropsychiatric Sample with Varying Degrees of Cognitive Impairment

Objective

Reliable Digit Span (RDS), RDS-Revised (RDS-R), and age-corrected scaled score (ACSS) have been previously validated as embedded performance validity tests (PVTs) from the Wechsler Adult Intelligence Scale-IV Digit Span subtest (WAIS-IV DS). However, few studies have directly compared the relative utility of these and other proposed WAIS-IV DS validity indicators within a single sample.

Method

This study compared classification accuracies of 10 WAIS-IV DS indices in a mixed neuropsychiatric sample of 227 outpatients who completed a standardized neuropsychological battery. Participants with ≤1 PVT failures of the four, freestanding criterion PVTs constituted the valid group (n = 181), whereas those with ≥2 PVT failures formed the invalid group (n = 46). Among the valid group, 113 met criteria for mild cognitive impairment (MCI).

Results

Classification accuracies for all DS indicators were statistically significant across the overall sample and subsamples with and without MCI, apart from indices derived from the Forward trial in the MCI sample. DS Sequencing ACSS, working memory RDS (wmRDS), and DS ACSS emerged as the most effective predictors of validity status, with acceptable to excellent classification accuracy for the overall sample (AUCs = 0.792–0.816; 35%–50% sensitivity/88%–96% specificity).

Conclusions

Although most DS indices demonstrated clinical utility as embedded PVTs, DS Sequencing ACSS, wmRDS, and DS ACSS may be particularly robust to cognitive impairment, minimizing risk of false positive errors while identifying noncredible performance. Moreover, DS indices incorporating data from multiple trials (i.e., wmRDS, DS ACSS) also generally yielded greater classification accuracy than those derived from a single trial.

Multivariable analysis of the relative utility and additive value of eight embedded performance validity tests for classifying invalid neuropsychological test performance

INTRODUCTION

This study investigated a combination of eight embedded performance validity tests (PVTs) derived from commonly administered neuropsychological tests to optimize sensitivity/specificity for detecting invalid neuropsychological test performance. The goal of this study was to evaluate what combination of these common embedded PVTs that have the most robust predictive power for detecting invalid neuropsychological test performance in a single diverse clinical sample.

METHOD

Eight previously validated memory- and nonmemory-based embedded PVTs were examined among 231 patients undergoing neuropsychological evaluation. Patients were classified into valid/invalid groups based on four independent criterion PVTs. Embedded PVT accuracy was assessed using standard and stepwise multiple logistic regression models.

RESULTS

Three PVTs, the Brief Visuospatial Memory Test-Revised Recognition Discrimination (BVMT-R-RD), Rey Auditory Verbal Learning Test Forced Choice, and WAIS-IV Digit Span Age Corrected Scaled Score, predicted 45.5% of the variance in validity group membership. BVMT-RD independently accounted for 32% of the variance in prediction of independent, criterion-defined validity group membership.

CONCLUSIONS

This study demonstrated the incremental predictive power of multiple embedded PVTs derived from common neuropsychological measures in detecting invalid test performance and those measures accounting for the greatest portion of the variance. These results provide guidance for evaluating the most fruitful embedded PVTs and proof of concept to better guide selection of embedded validity indices. Further, this offers clinicians an efficient, empirically derived approach to assessing performance validity when time restraints potentially limit the use of freestanding PVTs.

Utility of WAIS-IV Digit Span indices as measures of performance validity in moderate to severe traumatic brain injury

Objective: The addition of Sequencing to WAIS-IV Digit Span (DS) brought about new Reliable Digit Span (RDS) indices and an Age-Corrected Scaled Score that includes Sequencing trials. Reports have indicated that these new performance validity tests (PVTs) are superior to the traditional RDS; however, comparisons in the context of known neurocognitive impairment are sparse. This study compared DS-derived PVT classification accuracies in a design that included adults with verified TBI. Methods: Participants included 64 adults with moderate-to-severe TBI (TBI), 51 healthy adults coached to simulate TBI (SIM), and 78 healthy comparisons (HC). Participants completed the WAIS-IV DS subtest in the context of a larger test battery. Results: Kruskal-Wallis tests indicated that all DS indices differed significantly across groups. Post hoc contrasts revealed that only RDS Forward and the traditional RDS differed significantly between SIM and TBI. ROC analyses indicated that RDS variables were comparable predictors of SIM vs. HC; however, the traditional RDS showed the highest sensitivity when approximating 90% specificity for SIM vs. TBI. A greater percentage of TBI scored RDS Sequencing < 1 compared to SIM and HC. Conclusion: In the context of moderate-to-severe TBI, the DS-derived PVTs showed comparable discriminability. However, the Greiffenstein et al. traditional RDS demonstrated the best classification accuracy with respect to specificity/sensitivity balance. This relative superiority may reflect that individuals with verified TBI are more likely to perseverate on prior instructions during DS Sequencing. Findings highlight the importance of including individuals with verified TBI when evaluating and developing PVTs.

Assessment of differential neurocognitive performance based on the number of performance validity tests failures: A cross-validation study across multiple mixed clinical samples

Objective: This cross-sectional study examined the effect of number of Performance Validity Test (PVT) failures on neuropsychological test performance among a demographically diverse Veteran (VA) sample (n = 76) and academic medical sample (AMC; n = 128). A secondary goal was to investigate the psychometric implications of including versus excluding those with one PVT failure when cross-validating a series of embedded PVTs. Method: All patients completed the same six criterion PVTs, with the AMC sample completing three additional embedded PVTs. Neurocognitive test performance differences were examined based on number of PVT failures (0, 1, 2+) for both samples, and effect of number of criterion failures on embedded PVT performance was analyzed among the AMC sample. Results: Both groups with 0 or 1 PVT failures performed better than those with ≥2 PVT failures across most cognitive tests. There were nonsignificant differences between those with 0 or 1 PVT failures except for one test in the AMC sample. Receiver operator characteristic curve analyses found no differences in optimal cut score based on number of PVT failures when retaining/excluding one PVT failure. Conclusion: Findings support the use of ≥2 PVT failures as indicative of performance invalidity. These findings strongly support including those with one PVT failure with those with zero PVT failures in diagnostic accuracy studies, given that their inclusion reflects actual clinical practice, does not reduce sample sizes, and does not artificially deflate neurocognitive test results or inflate PVT classification accuracy statistics.

All of the accuracy in half of the time: Assessing abbreviated versions of the Test of Memory Malingering in the context of verbal and visual memory impairment

ObjectiveThe Test of Memory Malingering (TOMM) Trial 1 (T1) and errors on the first 10 items of T1 (T1-e10) were developed as briefer versions of the TOMM to minimize evaluation time and burden, although the effect of genuine memory impairment on these indices is not well established. This study examined whether increasing material-specific verbal and visual memory impairment affected T1 and T1-e10 performance and accuracy for detecting invalidity. Method: Data from 155 neuropsychiatric patients administered the TOMM, Rey Auditory Verbal Learning Test (RAVLT), and Brief Visuospatial Memory Test-Revised (BVMT-R) during outpatient evaluation were examined. Valid (N = 125) and invalid (N = 30) groups were established by four independent criterion performance validity tests. Verbal/visual memory impairment was classified as ≥37 T (normal memory); 30 T-36T (mild impairment); and ≤29 T (severe impairment). Results: Overall, T1 had outstanding accuracy, with 77% sensitivity/90% specificity. T1-e10 was less accurate but had excellent discriminability, with 60% sensitivity/87% specificity. T1 maintained excellent accuracy regardless of memory impairment severity, with 77% sensitivity/≥88% specificity and a relatively invariant cut-score even among those with severe verbal/visual memory impairment. T1-e10 had excellent classification accuracy among those with normal memory and mild impairment, but accuracy and sensitivity dropped with severe impairment and the optimal cut-score had to be increased to maintain adequate specificity. Conclusion: TOMM T1 is an effective performance validity test with strong psychometric properties regardless of material-specificity and severity of memory impairment. By contrast, T1-e10 functions relatively well in the context of mild memory impairment but has reduced discriminability with severe memory impairment.

BNT-15: Revised Performance Validity Cutoffs and Proposed Clinical Classification Ranges

BACKGROUND

Abbreviated neurocognitive tests offer a practical alternative to full-length versions but often lack clear interpretive guidelines, thereby limiting their clinical utility.

OBJECTIVE

To replicate validity cutoffs for the Boston Naming Test-Short Form (BNT-15) and to introduce a clinical classification system for the BNT-15 as a measure of object-naming skills.

METHOD

We collected data from 43 university students and 46 clinical patients. Classification accuracy was computed against psychometrically defined criterion groups. Clinical classification ranges were developed using a z -score transformation.

RESULTS

Previously suggested validity cutoffs (≤11 and ≤12) produced comparable classification accuracy among the university students. However, a more conservative cutoff (≤10) was needed with the clinical patients to contain the false-positive rate (0.20-0.38 sensitivity at 0.92-0.96 specificity). As a measure of cognitive ability, a perfect BNT-15 score suggests above average performance; ≤11 suggests clinically significant deficits. Demographically adjusted prorated BNT-15 T-scores correlated strongly (0.86) with the newly developed z -scores.

CONCLUSION

Given its brevity (<5 minutes), ease of administration and scoring, the BNT-15 can function as a useful and cost-effective screening measure for both object-naming/English proficiency and performance validity. The proposed clinical classification ranges provide useful guidelines for practitioners.

Assessing performance validity during attention-deficit/hyperactivity disorder evaluations: Cross-validation of non-memory embedded validity indicators

Embedded performance validity tests (PVTs) are key components of neuropsychological evaluations. However, most are memory-based and may be less useful in the assessment of attention-deficit/hyperactivity disorder (ADHD). Four non-memory-based validity indices derived from processing speed and executive functioning measures commonly included in ADHD evaluations, namely Verbal Fluency (VF) and the Trail Making Test (TMT), were cross-validated using the Rey 15-Item Test (RFIT) Recall and Recall/Recognition as memory-based comparison measures. This consecutive case series included data from 416 demographically-diverse adults who underwent outpatient neuropsychological evaluation for ADHD. Validity classifications were established, with ≤1 PVT failure of five independent criterion PVTs as indicative of valid performance (374 valid performers/42 invalid performers). Among the statistically significant validity indicators, TMT-A and TMT-B T-scores (AUCs = .707-.723) had acceptable classification accuracy ranges and sensitivities ranging from 29%-36% (≥89% specificity). RFIT Recall/Recognition produced similar results as TMT-B T-score with 42% sensitivity/90% specificity, but with lower classification accuracy. In evaluating adult ADHD, VF and TMT embedded PVTs demonstrated comparable sensitivity and specificity values to those found in other clinical populations but necessitated alternate cut-scores. Results also support use of RFIT Recall/Recognition over the standard RFIT Recall as a PVT for adult ADHD evaluations.

Cross-Validation of Multiple WAIS-IV Digit Span Embedded Performance Validity Indices Among a Large Sample of Adult Attention Deficit/Hyperactivity Disorder Clinical Referrals

This study investigated the utility of four WAIS-IV Digit Span (DS) indices (traditional Reliable Digit Span [RDS], RDS-Working Memory [RDS-WM], RDS-Revised [RDS-R], and DS Age-Corrected Scaled Score [ACSS]) as embedded performance validity tests (PVTs) among a sample of 342 consecutive adults referred for neuropsychological evaluation of ADHD. All DS indices had acceptable classification accuracy (areas under the curve: .73–.76) for detecting invalid performance with optimal cut-scores of RDS ≤7 (35% sensitivity/93% specificity), RDS-WM ≤7 (56% sensitivity/86% specificity), RDS-R ≤12 (48% sensitivity/85% specificity), and ACSS ≤7 (46% sensitivity/87% specificity). Although all indices were able to detect invalid performance, DS indices incorporating the more complex working memory trials of the task yielded the best accuracy for identification of invalid test performance among adults referred for ADHD evaluation.

Comparing the Independent and Aggregated Accuracy of Trial 1 and the First 10 TOMM Items for Detecting Invalid Neuropsychological Test Performance Across Civilian and Veteran Clinical Samples

Previous studies support using two abbreviated tests of the Test of Memory Malingering (TOMM), including (a) Trial 1 (T1) and (b) the number of errors on the first 10 items of T1 (T1e10), as performance validity tests (PVTs). In this study, we examined the independent and aggregated predictive utility of TOMM T1 and T1e10 for identifying invalid neuropsychological test performance across two clinical samples. We employed cross-sectional research to examine two independent and demographically diverse mixed samples of military veterans and civilians (VA = 108; academic medical center = 234) of patients who underwent neuropsychological evaluations. We determined validity groups by patient performance on four independent criterion PVTs. We established concordances between passing/failing the TOMM T1e10 and T1, followed by logistic regression to determine individual and aggregated accuracy of T1e10 and T1 for predicting validity group membership. Concordance between passing T1e10 and T1 was high, as was overall validity (87-98%) across samples. By contrast, T1e10 failure was more highly concordant with T1 failure (69-77%) than with overall invalidity status (59-60%) per criterion PVTs, whereas T1 failure was more highly concordant with invalidity status (72-88%) per criterion PVTs. Logistic regression analyses demonstrated similar results, with T1 accounting for more variance than T1e10. However, combining T1e10 and T1 accounted for the most variance of any model, with T1e10 and T1 each emerging as significant predictors. TOMM T1 and, to a lesser extent, T1e10 were significant predictors of independent criterion-derived validity status across two distinct clinical samples, but they did not offer improved classification accuracy when aggregated.

Are there differences in performance validity test scores between African American and White American neuropsychology clinic patients?

OBJECTIVE

The purpose of the present study was to compare performance on a wide range of PVTs in a neuropsychology clinic sample of African Americans and White Americans to determine if there are differences in mean scores or cut-off failure rates between the two groups, and to identify factors that may account for false positive PVT results in African American patients.

METHOD

African American and White American non-compensation-seeking neuropsychology clinic patients were compared on a wide range of standalone and embedded PVTs: Dot Counting Test, b Test, Warrington Recognition Memory Test, Rey 15-item plus recognition, Rey Word Recognition Test, Digit Span (ACSS, RDS, 3-digit time, 4-digit time), WAIS-III Picture Completion (Most discrepant index), WAIS-III Digit Symbol/Coding (recognition equation), Rey Auditory Verbal Learning Test, Rey Complex figure, WMS-III Logical Memory, Comalli Stroop Test, Trails A, and Wisconsin Card Sorting Test.

RESULTS

When groups were equated for age and education, African Americans obtained mean performances significantly worse than White Americans on only four of 25 PVT scores across the 14 different measures (Stroop Word Reading and Color Naming, Trails A, Digit Span 3-digit time); however, FSIQ was also significantly higher in White American patients. When subjects with borderline IQ (FSIQ = 70 to 79) were excluded (resulting in 74 White Americans and 25 African Americans), groups no longer differed in IQ and only continued to differ on a single PVT cutoff (Trails A). Further, specificity rates in African Americans were comparable to those of White Americans with the exception of the b Test, the Dot Counting Test, and Stroop B.

CONCLUSIONS

PVT performance generally does not differ as a function of Black versus White race once the impact of intellectual level is controlled, and most PVT cutoffs appear appropriate for use in African Americans of low average IQ or higher.

Examining Conners Continuous Performance Test-3 (CPT-3) Embedded Performance Validity Indicators in an Adult Clinical Sample Referred for ADHD Evaluation

This study evaluated multiple previously-identified Continuous Performance Test-Third Edition (CPT-3) scores as embedded validity indicators (EVIs) among 201 adults undergoing neuropsychological evaluation for Attention-Deficit/Hyperactivity Disorder (ADHD) divided into valid (n = 169) and invalid (n = 32) groups based on seven criterion measures. Although 6/10 CPT-3 scores accurately detected invalidity, only two reached minimally acceptable classification accuracy of ≥0.70. The remaining four had unacceptably low accuracy (AUCs = 0.62-0.69) with 0.19-0.41 sensitivity at ≥0.90 specificity. Composite scores did not provide better classification accuracy than individual CPT-3 scores. In sum, CPT-3 individual and composite scores generally are not accurate PVTs among adults undergoing clinical evaluation for ADHD.

Psychometric implications of failure on one performance validity test: a cross-validation study to inform criterion group definition

Introduction: Research to date has supported the use of multiple performance validity tests (PVTs) for determining validity status in clinical settings. However, the implications of including versus excluding patients failing one PVT remains a source of debate, and methodological guidelines for PVT research are lacking. This study evaluated three validity classification approaches (i.e. 0 vs. ≥2, 0-1 vs. ≥2, and 0 vs. ≥1 PVT failures) using three reference standards (i.e. criterion PVT groupings) to recommend approaches best suited to establishing validity groups in PVT research methodology.Method: A mixed clinical sample of 157 patients was administered freestanding (Medical Symptom Validity Test, Dot Counting Test, Test of Memory Malingering, Word Choice Test), and embedded PVTs (Reliable Digit Span, RAVLT Effort Score, Stroop Word Reading, BVMT-R Recognition Discrimination) during outpatient neuropsychological evaluation. Three reference standards (i.e. two freestanding and three embedded PVTs from the above list) were created. Rey 15-Item Test and RAVLT Forced Choice were used solely as outcome measures in addition to two freestanding PVTs not employed in the reference standard. Receiver operating characteristic curve analyses evaluated classification accuracy using the three validity classification approaches for each reference standard.Results: When patients failing only one PVT were excluded or classified as valid, classification accuracy ranged from acceptable to excellent. However, classification accuracy was poor to acceptable when patients failing one PVT were classified as invalid. Sensitivity/specificity across two of the validity classification approaches (0 vs. ≥2; 0-1 vs. ≥2) remained reasonably stable.Conclusions: These results reflect that both inclusion and exclusion of patients failing one PVT are acceptable approaches to PVT research methodology and the choice of method likely depends on the study rationale. However, including such patients in the invalid group yields unacceptably poor classification accuracy across a number of psychometrically robust outcome measures and therefore is not recommended.

The Impact of Learning and Memory on Performance Validity Tests in a Mixed Clinical Pediatric Population

OBJECTIVE

This study examined the degree to which verbal and visuospatial memory abilities influence performance validity test (PVT) performance in a mixed clinical pediatric sample.

METHOD

Data from 252 consecutive clinical pediatric cases (Mage=11.23 years, SD=4.02; 61.9% male) seen for outpatient neuropsychological assessment were collected. Measures of learning and memory (e.g., The California Verbal Learning Test-Children's Version; Child and Adolescent Memory Profile [ChAMP]), performance validity (Test of Memory Malingering Trial 1 [TOMM T1]; Wechsler Intelligence Scale for Children-Fifth Edition [WISC-V] or Wechsler Adult Intelligence Scale-Fourth Edition Digit Span indices; ChAMP Overall Validity Index), and intellectual abilities (e.g., WISC-V) were included.

RESULTS

Learning/memory abilities were not significantly correlated with TOMM T1 and accounted for relatively little variance in overall TOMM T1 performance (i.e., ≤6%). Conversely, ChAMP Validity Index scores were significantly correlated with verbal and visual learning/memory abilities, and learning/memory accounted for significant variance in PVT performance (12%-26%). Verbal learning/memory performance accounted for 5%-16% of the variance across the Digit Span PVTs. No significant differences in TOMM T1 and Digit Span PVT scores emerged between verbal/visual learning/memory impairment groups. ChAMP validity scores were lower for the visual learning/memory impairment group relative to the nonimpaired group.

CONCLUSIONS

Findings highlight the utility of including PVTs as standard practice for pediatric populations, particularly when memory is a concern. Consistent with the adult literature, TOMM T1 outperformed other PVTs in its utility even among the diverse clinical sample with/without learning/memory impairment. In contrast, use of Digit Span indices appear to be best suited in the presence of visuospatial (but not verbal) learning/memory concerns. Finally, the ChAMP's embedded validity measure was most strongly impacted by learning/memory performance.

Examining Traditional and Novel Validity Indicators from the Medical Symptom Validity Test Across Levels of Verbal and Visual Memory Impairment

OBJECTIVE

This cross-sectional study examined accuracy of traditional Medical Symptom Validity Test (MSVT) validity indicators, including immediate recognition (IR), delayed recognition (DR), and consistency (CNS), as well as a novel indicator derived from the mean performance on IR, DR, and CNS across verbal, visual, and combined learning and memory impairment bands.

METHOD

A sample of 180 adult outpatients was divided into valid (n = 150) and invalid (n = 30) groups based on results of four independent criterion performance validity tests. Verbal and visual learning and recall were classified as indicative of no impairment, mild impairment, or severe impairment based on performance on the Rey Auditory Verbal Learning Test and Brief Visuospatial Memory Test-Revised, respectively.

RESULTS

In general, individual MSVT subtests were able to accurately classify performance as valid or invalid, even in the context of severe learning and memory deficits. However, as verbal and visual memory impairment increased, optimal MSVT cut-scores diverged from manual-specified cutoffs such that DR and CNS required cut-scores to be lowered to maintain adequate specificity. By contrast, the newly proposed scoring algorithm generally showed more robust psychometric properties across the memory impairment bands.

CONCLUSIONS

The mean performance index, a novel scoring algorithm using the mean of the three primary MSVT subtests, may be a more robust validity indicator than the individual MSVT subtests in the context of bona fide memory impairment.

Moving Toward Accurate Assessment of Working Memory in Adults With Neurogenically Based Communication Disorders

Purpose This article presents a viewpoint highlighting concerns regarding currently available assessments of working memory in adults with neurogenic communication disorders. Additionally, we provide recommendations for improving working memory assessment in this population. Method This viewpoint includes a critique of clinical and experimental working memory tests relevant to speech-language pathologists. We consider the terminology used to describe memory, as well as discuss language demands and test construction. Results Clinical and experimental testing of working memory in adults with neurogenic communication disorders is challenged due to theoretical, methodological, and practical limitations. The major limitations are characterized as linguistic and task demands, presentation and response modality effects, test administration, and scoring parameters. Taking these limitations into consideration, several modifications to working memory testing and their relevance to neurogenic populations are discussed. Conclusions The recommendations provided in this article can better guide clinicians and researchers to advocate for improved tests of working memory in adults with neurogenic communication disorders. Future research should continue to address these concerns and consider our recommendations.

Mean response latency indices on the Victoria Symptom Validity Test do not contribute meaningful predictive value over accuracy scores for detecting invalid performance

The utility of the Victoria Symptom Validity Test (VSVT) as a performance validity test (PVT) has been primarily established using response accuracy scores. However, the degree to which response latency may contribute to accurate classification of performance invalidity over and above accuracy scores remains understudied. Therefore, this study investigated whether combining VSVT accuracy and response latency scores would increase predictive utility beyond use of accuracy scores alone. Data from a mixed clinical sample of 163 patients, who were administered the VSVT as part of a larger neuropsychological battery, were analyzed. At least four independent criterion PVTs were used to establish validity groups (121 valid/42 invalid). Logistic regression models examining each difficulty level revealed that all VSVT measures were useful in classifying validity groups, both independently and when combined. Individual predictor classification accuracy ranged from 77.9 to 81.6%, indicating acceptable to excellent discriminability across the validity indices. The results of this study support the value of both accuracy and latency scores on the VSVT to identify performance invalidity, although the accuracy scores had superior classification statistics compared to response latency, and mean latency indices provided no unique benefit for classification accuracy beyond dimensional accuracy scores alone.

A Systematic Review and Meta-Analysis of the Diagnostic Accuracy of the Advanced Clinical Solutions Word Choice Test as a Performance Validity Test

Thorough assessment of performance validity has become an established standard of practice in neuropsychological assessment. While there has been a large focus on the development and cross-validation of embedded performance validity tests (PVTs) in recent years, new freestanding PVTs have also been developed, including the Word Choice Test (WCT) as part of the Advanced Clinical Solutions Effort System. And, while the WCT's general utility for identifying invalid performance has been demonstrated in the ensuing decade since its initial publication, optimal cut-scores and associated psychometric properties have varied widely across studies. This study sought to synthesize the existing diagnostic accuracy literature regarding the WCT via a systematic review and to conduct a meta-analysis to determine the performance validity cut-score that best maximizes sensitivity while maintaining acceptable specificity. A systematic search of the literature resulted in 14 studies for synthesis, with eight of those available for meta-analysis. Meta-analytic results revealed an optimal cut-score of ≤ 42 with 54% sensitivity and 93% specificity for identifying invalid neuropsychological test performance. Collectively, the WCT demonstrated adequate diagnostic accuracy as a PVT across a variety of populations. Recommendations for future studies are also provided.

Victoria Symptom Validity Test: A Systematic Review and Cross-Validation Study

The Victoria Symptom Validity Test (VSVT) is a performance validity test (PVT) with over two decades of empirical backing, although methodological limitations within the extant literature restrict its clinical and research generalizability. Chief among these constraints includes limited consensus on the most accurate index within the VSVT and the most appropriate cut-scores within each VSVT validity index. The current systematic review synthesizes existing VSVT validation studies and provides additional cross-validation in an independent sample using a known-groups design. We completed a systematic search of the literature, identifying 17 peer-reviewed studies for synthesis (7 simulation designs, 7 differential prevalence designs, and 3 known-groups designs). The independent cross-validation sample consisted of 200 mixed clinical neuropsychiatric patients referred for outpatient neuropsychological evaluation. Across all indices, Total item accuracy produced the strongest psychometric properties at an optimal cut-score of ≤ 40 (62% sensitivity/88% specificity). However, ROC curve analyses for all VSVT indices yielded statistically significant areas under the curve (AUCs; .73-81), suggestive of moderate classification accuracy. Cut-scores derived using the independent cross-validation sample converged with some previous findings supporting cut-scores of ≤ 22 for Easy item accuracy and ≤ 40 for Total item accuracy, although divergent findings were noted for Difficult item accuracy. Overall, VSVT validity indicators have adequate diagnostic accuracy across populations, with the current study providing additional support for its use as a psychometrically sound PVT in clinical settings. However, caution is recommended among patients with certain verified clinical conditions (e.g., dementia) and those with pronounced working memory deficits due to concerns for increased risk of false positives.