A systematic review and meta-analysis of the Test of Memory Malingering in adults: Two decades of deception detection

Forced choice test score of p ≤ .20 and failures on ≥ six performance validity tests results in similar Overall Test Battery Means

Objective: To determine if similar levels of performance on the Overall Test Battery Mean (OTBM) occur at different forced choice test (FCT) p-value score failures. Second, to determine the OTBM levels that are associated with failures at above chance on various performance validity (PVT) tests. Method: OTBMs were computed from archival data obtained from four practices. We calculated each examinee's Estimated Premorbid Global Ability (EPGA) and OTBM. The sample size was 5,103 examinees with 282 (5.5%) of these scoring below chance at p ≤ .20 on at least one FCT. Results: The OTBM associated with a failure at p ≤ .20 was equivalent to the OTBM that was associated with failing 6 or more PVTs at above-chance cutoffs. The mean OTBMs relative to increasingly strict FCT p cutoffs were similar (T scores in the 30s). As expected, there was an inverse relationship between the number of PVTs failed and examinees' OTBMs. Conclusions: The data support the use of p ≤ .20 as the probability level for testing the significance of below chance performance on FCTs. The OTBM can be used to index the influence of invalid performance on outcomes, especially when an examinee scores below chance.

Litigation, Performance Validity Testing, and Treatment Outcomes in Adults With Mild Traumatic Brain Injury

OBJECTIVE

To investigate whether involvement in litigation and performance validity test (PVT) failure predict adherence to treatment and treatment outcomes in adults with persistent symptoms after mild traumatic brain injury (mTBI).

SETTING

Outpatient concussion clinics in British Columbia, Canada. Participants were assessed at intake (average 12.9 weeks postinjury) and again following 3 to 4 months of rehabilitation.

PARTICIPANTS

Adults who met the World Health Organization Neurotrauma Task Force definition of mTBI. Litigation status was known for 69 participants ( n = 21 reported litigation), and 62 participants completed a PVT ( n = 13 failed the Test of Memory Malingering) at clinic intake.

DESIGN

Secondary analysis of a clinical trial (ClinicalTrials.gov #NCT03972579).

MAIN MEASURES

Outcomes included number of completed sessions, homework adherence, symptoms (Rivermead Post Concussion Symptoms Questionnaire), disability ratings (World Health Organization Disability Assessment Schedule 2.0), and patient-rated global impression of change.

RESULTS

We did not observe substantial differences in session and homework adherence associated with litigation or PVT failure. Disability and postconcussion symptoms generally improved with treatment. Involvement in litigation was associated with a smaller improvement in outcomes, particularly disability ( B = 2.57, 95% confidence interval [CI] [0.25-4.89], P = .03) and patient-reported global impression of change (odds ratio [OR] = 4.19, 95% CI [1.40-12.57], P = .01). PVT failure was not associated with considerable differences in treatment outcomes. However, participants who failed the PVT had a higher rate of missing outcomes (31% vs 8%) and perceived somewhat less global improvement (OR = 3.47, 95% CI [0.86-14.04]; P = .08).

CONCLUSION

Adults with mTBI who are in litigation or who failed PVTs tend to adhere to and improve following treatment. However, involvement in litigation may be associated with attenuated improvements, and pretreatment PVT failure may predict lower engagement in the treatment process.

Ability of the Wisconsin Card-Sorting Test-64 as an embedded measure to identify noncredible neurocognitive performance in mild traumatic brain injury litigants

OBJECTIVE

To investigate the ability of selective measures on the Wisconsin Card Sorting Test-64 (WCST-64) to predict noncredible neurocognitive dysfunction in a large sample of mild traumatic brain injury (mTBI) litigants.

METHOD

Participants included 114 adults who underwent a comprehensive neuropsychological examination. Criterion groups were formed based upon their performance on stand-alone measures of cognitive performance validity (PVT).

RESULTS

Participants failing PVTs performed worse across all WCST-64 dependent variables of interest compared to participants who passed PVTs. Receiver operating curve analysis revealed that only categories completed was a significant predictors of PVT status. Multivariate logistic regression did not add to classification accuracy.

CONCLUSION

Consideration of noncredible executive functioning may be warranted in mild traumatic brain injury (mTBI) litigants who complete ≤ 1 category on the WCST-64.

Validity of the Groningen Effort Test in patients with suspected chronic solvent-induced encephalopathy

INTRODUCTION

The use of performance validity tests (PVTs) in a neuropsychological assessment to determine indications of invalid performance has been a common practice for over a decade. Most PVTs are memory-based; therefore, the Groningen Effort Test (GET), a non-memory-based PVT, has been developed.

OBJECTIVES

This study aimed to validate the GET in patients with suspected chronic solvent-induced encephalopathy (CSE) using the criterion standard of 2PVTs. A second goal was to determine diagnostic accuracy for GET.

METHOD

Sixty patients with suspected CSE referred for NPA were included. The GET was compared to the criterion standard of 2PVTs based on the Test of Memory Malingering and the Amsterdam Short Term Memory Test.

RESULTS

The frequency of invalid performance using the GET was significantly higher compared to the criterion of 2PVTs (51.7% vs. 20.0% respectively; p < 0.001). For the GET index, the sensitivity was 75% and the specificity was 54%, with a Youden's Index of 27.

CONCLUSION

The GET showed significantly more invalid performance compared to the 2PVTs criterion suggesting a high number of false positives. The general accepted minimum norm of specificity for PVTs of >90% was not met. Therefore, the GET is of limited use in clinical practice with suspected CSE patients.

Recall and recognition of similarities items in neuropsychological assessment: Memory, validity, and meaning

The current study examined whether the Memory Similarities Extended Test (M-SET), a memory test based on the Similarities subtest of the Wechsler Abbreviated Scale of Intelligence, Second Edition (WASI-II), has value in neuropsychological testing. The relationship of M-SET measures of cued recall (CR) and recognition memory (REC) to brain injury severity and memory scores from the Wechsler Memory Scale, Fourth Edition (WMS-IV) was analyzed in examinees with traumatic brain injuries ranging from mild to severe. Examinees who passed standard validity tests were divided into groups with intracranial injury (CT + ve, n = 18) and without intracranial injury (CT-ve, n = 50). In CT + ve only, CR was significantly correlated with Logical Memory I (LMI: rs = .62) and Logical Memory II (LMII: rs = .65). In both groups, there were smaller correlations with delayed visual memory (VRII: rs = .38; rs = .44) and psychomotor speed (Coding: rs = .29; rs = .29). The REC score was neither an indicator of memory ability nor an internal indicator of performance validity. There were no differences in M-SET or WMS-IV scores for CT-ve and CT + ve, and reasons for this are discussed. It is concluded that M-SET has utility as an incidental cued recall measure.

Performance Validity Test Failure in the Clinical Population: A Systematic Review and Meta-Analysis of Prevalence Rates

Performance validity tests (PVTs) are used to measure the validity of the obtained neuropsychological test data. However, when an individual fails a PVT, the likelihood that failure truly reflects invalid performance (i.e., the positive predictive value) depends on the base rate in the context in which the assessment takes place. Therefore, accurate base rate information is needed to guide interpretation of PVT performance. This systematic review and meta-analysis examined the base rate of PVT failure in the clinical population (PROSPERO number: CRD42020164128). PubMed/MEDLINE, Web of Science, and PsychINFO were searched to identify articles published up to November 5, 2021. Main eligibility criteria were a clinical evaluation context and utilization of stand-alone and well-validated PVTs. Of the 457 articles scrutinized for eligibility, 47 were selected for systematic review and meta-analyses. Pooled base rate of PVT failure for all included studies was 16%, 95% CI [14, 19]. High heterogeneity existed among these studies (Cochran's Q = 697.97, p < .001; I2 = 91%; τ2 = 0.08). Subgroup analysis indicated that pooled PVT failure rates varied across clinical context, presence of external incentives, clinical diagnosis, and utilized PVT. Our findings can be used for calculating clinically applied statistics (i.e., positive and negative predictive values, and likelihood ratios) to increase the diagnostic accuracy of performance validity determination in clinical evaluation. Future research is necessary with more detailed recruitment procedures and sample descriptions to further improve the accuracy of the base rate of PVT failure in clinical practice.

Examination of the relationship between symptom and performance validity measures across referral subtypes

INTRODUCTION

The extent to which performance validity (PVT) and symptom validity (SVT) tests measure separate constructs is unclear. Prior research using the Minnesota Multiphasic Personality Inventory (MMPI-2 & RF) suggested that PVTs and SVTs are separate but related constructs. However, the relationship between Personality Assessment Inventory (PAI) SVTs and PVTs has not been explored. This study aimed to replicate previous MMPI research using the PAI, exploring the relationship between PVTs and overreporting SVTs across three subsamples, neurodevelopmental (attention deficit-hyperactivity disorder (ADHD)/learning disorder), psychiatric, and mild traumatic brain injury (mTBI).

METHODS

Participants included 561 consecutive referrals who completed the Test of Memory Malingering (TOMM) and the PAI. Three subgroups were created based on referral question. The relationship between PAI SVTs and the PVT was evaluated through multiple regression analysis.

RESULTS

The results demonstrated the relationship between PAI symptom overreporting SVTs, including Negative Impression Management (NIM), Malingering Index (MAL), and Cognitive Bias Scale (CBS), and PVTs varied by referral subgroup. Specifically, overreporting on CBS but not NIM and MAL significantly predicted poorer PVT performance in the full sample and the mTBI sample. In contrast, none of the overreporting SVTs significantly predicted PVT performance in the ADHD/learning disorder sample but conversely, all SVTs predicted PVT performance in the psychiatric sample.

CONCLUSIONS

The results partially replicated prior research comparing SVTs and PVTs and suggested that constructs measured by SVTs and PVTs vary depending upon population. The results support the necessity of both PVTs and SVTs in clinical neuropsychological practice.

The impact of race and other demographic factors on the false positive rates of five embedded Performance Validity Tests (PVTs) in a Veteran sample

INTRODUCTION

It is common to use normative adjustments based on race to maintain accuracy when interpreting cognitive test results during neuropsychological assessment. However, embedded performance validity tests (PVTs) do not adjust for these racial differences and may result in elevated rates of false positives in African American/Black (AA) samples compared to European American/White (EA) samples.

METHODS

Veterans without Major Neurocognitive Disorder completed an outpatient neuropsychological assessment and were deemed to be performing in a valid manner (e.g., passing both the Test of Memory Malingering Trial 1 (TOMM1) and the Medical Symptom Validity Test (MSVT), (n = 531, EA = 473, AA = 58). Five embedded PVTs were administered to all patients: WAIS-III/IV Processing Speed Index (PSI), Brief Visuospatial Memory Test-Revised: Discrimination Index (BVMT-R), TMT-A (secs), California Verbal Learning Test-II (CVLT-II) Forced Choice, and WAIS-III/IV Digit Span Scaled Score. Individual PVT false positive rates, as well as the rate of failing two or more embedded PVTs, were calculated.

RESULTS

Failure rates of two embedded PVTs (PSI, TMT-A), and the total number of PVTs failed, were higher in the AA sample. The PSI and TMT-A remained significantly impacted by race after accounting for age, education, sex, and presence of Mild Neurocognitive Disorder. There were PVT failure rates greater than 10% (and considered false positives) in both groups (AA: PSI, TMT-A, and BVMT-R, 12-24%; EA: BVMT-R, 17%). Failing 2 or more PVTs (AA = 9%, EA = 4%) was impacted by education and Mild Neurocognitive Disorder but not by race.

CONCLUSIONS

Individual (timed) PVTs showed higher false positive rates in the AA sample even after accounting for demographic factors and diagnosis of Mild Neurocognitive Disorder. Requiring failure on 2 or more embedded PVTs reduced false positive rates to acceptable levels across both groups (10% or less) and was not significantly influenced by race.

Cross-validation of the Trail Making Test as a non-memory-based embedded performance validity test among veterans with and without cognitive impairment

OBJECTIVE

This study cross-validated multiple Trail Making Test (TMT) Parts A and B scores as non-memory-based embedded performance validity tests (PVTs) for detecting invalid neuropsychological performance among veterans with and without cognitive impairment.

METHOD

Data were collected from a demographically and diagnostically diverse mixed clinical sample of 100 veterans undergoing outpatient neuropsychological evaluation at a Southwestern VA Medical Center. As part of a larger battery of neuropsychological tests, all veterans completed TMT A and B and four independent criterion PVTs, which were used to classify veterans into valid (n = 75) and invalid (n = 25) groups. Among the valid group 47% (n = 35) were cognitively impaired.

RESULTS

Among the overall sample, all embedded PVTs derived from TMT A and B raw and demographically corrected T-scores significantly differed between validity groups (ηp2 = .21-.31) with significant areas under the curve (AUCs) of .72-.78 and 32-48% sensitivity (≥91% specificity) at optimal cut-scores. When subdivided by cognitive impairment status (i.e., valid-unimpaired vs. invalid; valid-impaired vs. invalid), all TMT scores yielded significant AUCs of .80-.88 and 56%-72% sensitivity (≥90% specificity) at optimal cut-scores. Among veterans with cognitive impairment, neither TMT A or B raw scores were able to significantly differentiate the invalid from the valid-cognitively impaired group; however, demographically corrected T-scores were able to significantly differentiate groups but had poor classification accuracy (AUCs = .66-.68) and reduced sensitivity of 28%-44% (≥91% specificity).

CONCLUSIONS

Embedded PVTs derived from TMT Parts A and B raw and T-scores were able to accurately differentiate valid from invalid neuropsychological performance among veterans without cognitive impairment; however, the demographically corrected T-scores generally were more robust and consistent with prior studies compared to raw scores. By contrast, TMT embedded PVTs had poor accuracy and low sensitivity among veterans with cognitive impairment, suggesting limited utility as PVTs among populations with cognitive dysfunction.

Lowered cutoffs to reduce false positives on the Word Memory Test

OBJECTIVE

To adjust the decision criterion for the Word Memory Test (WMT, Green, 2003) to minimize the frequency of false positives.

METHOD

Archival data were combined into a database (n = 3,210) to examine the best cut score for the WMT. We compared results based on the original scoring rules and those based on adjusted scoring rules using a criterion based on 16 performance validity tests (PVTs) exclusive of the WMT. Cutoffs based on peer-reviewed publications and test manuals were used. The resulting PVT composite was considered the best estimate of validity status. We focused on a specificity of .90 with a false-positive rate of less than .10 across multiple samples.

RESULTS

Each examinee was administered the WMT, as well as on average 5.5 (SD = 2.5) other PVTs. Based on the original scoring rules of the WMT, 31.8% of examinees failed. Using a single failure on the criterion PVT (C-PVT), the base rate of failure was 45.9%. When requiring two or more failures on the C-PVT, the failure rate dropped to 22.8%. Applying a contingency analysis (i.e., X2) to the two failures model on the C-PVT measure and using the original rules for the WMT resulted in only 65.3% agreement. However, using our adjusted rules for the WMT, which consisted of relying on only the IR and DR WMT subtest scores with a cutoff of 77.5%, agreement between the adjusted and the C-PVT criterion equaled 80.8%, for an improvement of 12.1% identified. The adjustmeny resulted in a 49.2% reduction in false positives while preserving a sensitivity of 53.6%. The specificity for the new rules was 88.8%, for a false positive rate of 11.2%.

CONCLUSIONS

Results supported lowering of the cut score for correct responding from 82.5% to 77.5% correct. We also recommend discontinuing the use of the Consistency subtest score in the determination of WMT failure.

The Importance of the Morel Emotional Numbing Test Instructions: A Diagnosis Threat Induction Study

OBJECTIVE

Marketed as a validity test that detects feigning of posttraumatic stress disorder (PTSD), the Morel Emotional Numbing Test for PTSD (MENT) instructs examinees that PTSD may negatively affect performance on the measure. This study explored the potential that MENT performance depends on inclusion of "PTSD" in its instructions and the nature of the MENT as a performance validity versus a symptom validity test (PVT/SVT).

METHOD

358 participants completed the MENT as a part of a clinical neuropsychological evaluation. Participants were either administered the MENT with the standard instructions (SIs) that referenced "PTSD" or revised instructions (RIs) that did not. Others were administered instructions that referenced "ADHD" rather than PTSD (AI). Comparisons were conducted on those who presented with concerns for potential traumatic-stress related symptoms (SI vs. RI-1) or attention deficit (AI vs. RI-2).

RESULTS

Participants in either the SI or AI condition produced more MENT errors than those in their respective RI conditions. The relationship between MENT errors and other S/PVTs was significantly stronger in the SI: RI-1 comparison, such that errors correlated with self-reported trauma-related symptoms in the SI but not RI-1 condition. MENT failure also predicted PVT failure at nearly four times the rate of SVT failure.

CONCLUSIONS

Findings suggest that the MENT relies on overt reference to PTSD in its instructions, which is linked to the growing body of literature on "diagnosis threat" effects. The MENT may be considered a measure of suggestibility. Ethical considerations are discussed, as are the construct(s) measured by PVTs versus SVTs.

Symptom attribution is a stronger predictor of PVT-failure than symptom endorsement in treatment-seeking Veterans with remote mTBI history: A pilot study

OBJECTIVE

To examine relationships between performance validity testing (PVT), neurobehavioral symptom endorsement, and symptom attribution in Veterans with a history of mild traumatic brain injury (mTBI).

METHOD

Participants included treatment-seeking Veterans (n = 37) with remote mTBI histories who underwent a neuropsychological assessment and completed a modified version of the Neurobehavioral Symptom Inventory (NSI) to assess symptom endorsement and symptom attribution (the latter evaluated by having Veterans indicate whether they believed each NSI symptom was caused by their mTBI). Veterans were divided into two subgroups, PVT-Valid (n = 25) and PVT-Invalid (n = 12).

RESULTS

Independent samples t-tests showed that two of five symptom endorsement variables and all five symptom attribution variables were significantly different between PVT groups (PVT-Invalid > PVT-Valid; Cohen's d = 0.67-1.02). Logistic regression analyses adjusting for PTSD symptoms showed that symptom endorsement (Nagelkerke's R2 = .233) and symptom attribution (Nagelkerke's R2 = .279) significantly distinguished between PVT groups. According to the Wald criterion, greater symptom endorsement (OR = 1.09) and higher attribution of symptoms to mTBI (OR = 1.21) each reliably predicted PVT-failure.

CONCLUSIONS

While both symptom endorsement and symptom attribution were significantly associated with PVT-failure, our preliminary results suggest that symptom attribution is a stronger predictor of PVT-failure. Results highlight the importance of assessing symptom attribution to mTBI in this population.

Detection of noncredible cognitive performance with Wechsler Memory Scale-IV measures in mild traumatic brain injury litigants

OBJECTIVE

To investigate the operating characteristics of selective measures on the Wechsler Memory Scale-IV (WMS-IV) to predict noncredible neurocognitive dysfunction in a sample of mild traumatic brain injury (mTBI) litigants.

METHOD

Participants included 110 adults who underwent a comprehensive neuropsychological examination. Criterion groups were formed based upon their performance on stand-alone measures of cognitive performance validity testing (PVT).

RESULTS

Participants failing two stand-alone PVTs exhibited significantly lower scores across all WMS-IV dependent variables of interest compared to participants who passed both PVTs. Participants who failed one PVT were excluded. Bivariate logistic regression revealed that all six dependent variables were significant predictors of PVT status. The best prediction model consisted of three WMS-IV variables including Logical Memory Delayed Recall (LM2), Logical Memory Recognition (LMR), and Visual Reproduction Recognition (VRR). This model demonstrated an accuracy of 90.2%, 0.89 sensitivity, 0.92 specificity, and a Receiver Operating Curve (ROC) of 0.957.

CONCLUSION

The current empirically-derived cutscores and logit equation for the WMS-IV may be an additional consideration in analyzing database validity and noncredible performance in mTBI personal injury litigants ages 18-69.

Factors perpetuating functional cognitive symptoms after mild traumatic brain injury

INTRODUCTION

Self-reported memory difficulties (forgetting familiar names, misplacing objects) often persist long after a mild traumatic brain injury (mTBI), despite normal neuropsychological test performance. This clinical presentation may be a manifestation of a functional cognitive disorder (FCD). Several mechanisms underlying FCD have been proposed, including metacognitive impairment, memory perfectionism, and misdirected attention, as well as depression or anxiety-related explanations. This study aims to explore these candidate perpetuating factors in mTBI, to advance our understanding of why memory symptoms frequently persist following mTBI.

METHODS

A cross-sectional study of 67 adults (n = 39 with mTBI mean = 25 months ago and n = 28 healthy controls). Participants completed standardized questionnaires (including the Functional Memory Disorder Inventory), a metacognitive task (to quantify discrepancies between their trial-by-trial accuracy and confidence), and a brief neuropsychological test battery. We assessed candidate mechanisms in two ways: (1) between-groups, comparing participants with mTBI to healthy controls, and (2) within-group, examining their associations with functional memory symptom severity (FMDI) in the mTBI group.

RESULTS

Participants with mTBI performed similarly to controls on objective measures of memory ability but reported experiencing much more frequent memory lapses in daily life. Contrary to expectations, metacognitive efficiency did not differentiate the mTBI and control groups and was not associated with functional memory symptoms. Memory perfectionism was strongly associated with greater functional memory symptoms among participants with mTBI but did not differ between groups when accounting for age. Depression and checking behaviors produced consistent results across between-groups and within-group analyses: these factors were greater in the mTBI group compared to the control group and were associated with greater functional memory symptoms within the mTBI group.

CONCLUSIONS

This study highlights promising (e.g., depression, checking behaviors) and unlikely (e.g., metacognitive impairment) mechanisms underlying functional memory symptoms after mTBI, to guide future research and treatment.

Validity assessment in Eastern Europe: cross-validation of the Dot Counting Test and MODEMM against the TOMM-1 and Rey-15 in a Romanian mixed clinical sample

OBJECTIVE

This study investigated performance validity in the understudied Romanian clinical population by exploring classification accuracies of the Dot Counting Test (DCT) and the first Romanian performance validity test (PVT) (Memory of Objects and Digits and Evaluation of Memory Malingering/MODEMM) in a heterogeneous clinical sample.

METHODS

We evaluated 54 outpatients (26 females; MAge = 62.02; SDAge = 12.3; MEducation = 2.41, SDEducation = 2.82) with the Test of Memory Malingering 1 (TOMM-1), Rey Fifteen Items Test (Rey-15) (free recall and recognition trials), DCT, MODEMM, and MMSE/MoCA as part of their neuropsychological assessment. Accuracy parameters and base failure rates were computed for the DCT and MODEMM indicators against the TOMM-1 and Rey-15. Two patient groups were constructed according to psychometrically defined credible/noncredible performance (i.e., pass/fail both TOMM-1 and Rey-15).

RESULTS

Similar to other cultures, a cutoff of ≥18 on the DCT E score produced the best combination between sensitivity (0.50-0.57) and specificity (≥0.90). MODEMM indicators based on recognition accuracy, inconsistencies, and inclusion false positives generated 0.75-0.86 sensitivities at ≥0.90 specificities. Multivariable models of MODEMM indicators reached perfect sensitivities at ≥0.90 specificities against two PVTs. Patients who failed the TOMM-1 and Rey-15 were significantly more likely to fail the DCT and MODEMM than patients who passed both PVTs.

CONCLUSIONS

Our results offer proof of concept for the DCT's cross-cultural validity and the applicability of the MODEMM on Romanian clinical examinees, further recommending the use of heterogeneous validity indicators in clinical assessments.

Specificity of performance validity tests in patients with confirmed epilepsy

Objective: While assessment of performance validity is essential to neuropsychological evaluations, use of performance validity tests (PVTs) in an epilepsy population has raised concerns due to factors that may result in performance fluctuations. The current study assessed whether specificity was maintained at previously suggested cutoffs in a confirmed epilepsy population on the Warrington Recognition Memory Test (WRMT) - Words and Test of Memory Malingering (TOMM). Method: Eighty-two confirmed epilepsy patients were administered the WRMT-Words and TOMM as part of a standardized neuropsychological evaluation. Frequency tables were utilized to investigate specificity rates on these two PVTs. Results: The suggested WRMT-Words Accuracy Score cutoff of ≤42 was associated with a specificity rate of 90.2%. Five out of the 8 individuals falling below the Accuracy Score cutoff scored 42, suggesting specificity could be further improved by slightly lowering the cutoff. The WRMT-Words Total Time cutoff of ≥207 seconds was associated with 95.1% specificity. A TOMM Trial 1 cutoff of <40 was associated with 93.9% specificity, while the established cutoff of <45 on Trial 2 and the Retention Trial yielded specificity rates of 98.6% and 100.0%, respectively. Conclusions: Our findings demonstrate acceptable performance on two PVTs in a select confirmed epilepsy population without a history of brain surgery, active seizures during testing, and/or low IQ, irrespective of various factors such as seizure type, seizure lateralization/localization, and language lateralization. The possible presence of interictal discharges were not controlled for in the current study, which may have contributed to reduced PVT performances.

Review of Statistical and Methodological Issues in the Forensic Prediction of Malingering from Validity Tests: Part II-Methodological Issues

Forensic neuropsychological examinations to detect malingering in patients with neurocognitive, physical, and psychological dysfunction have tremendous social, legal, and economic importance. Thousands of studies have been published to develop and validate methods to forensically detect malingering based largely on approximately 50 validity tests, including embedded and stand-alone performance and symptom validity tests. This is Part II of a two-part review of statistical and methodological issues in the forensic prediction of malingering based on validity tests. The Part I companion paper explored key statistical issues. Part II examines related methodological issues through conceptual analysis, statistical simulations, and reanalysis of findings from prior validity test validation studies. Methodological issues examined include the distinction between analog simulation and forensic studies, the effect of excluding too-close-to-call (TCTC) cases from analyses, the distinction between criterion-related and construct validation studies, and the application of the Revised Quality Assessment of Diagnostic Accuracy Studies tool (QUADAS-2) in all Test of Memory Malingering (TOMM) validation studies published within approximately the first 20 years following its initial publication to assess risk of bias. Findings include that analog studies are commonly confused for forensic validation studies, and that construct validation studies are routinely presented as if they were criterion-reference validation studies. After accounting for the exclusion of TCTC cases, actual classification accuracy was found to be well below claimed levels. QUADAS-2 results revealed that extant TOMM validation studies all had a high risk of bias, with not a single TOMM validation study with low risk of bias. Recommendations include adoption of well-established guidelines from the biomedical diagnostics literature for good quality criterion-referenced validation studies and examination of implications for malingering determination practices. Design of future studies may hinge on the availability of an incontrovertible reference standard of the malingering status of examinees.

From "below chance" to "a single error is one too many": Evaluating various thresholds for invalid performance on two forced choice recognition tests

This study was designed to empirically evaluate the classification accuracy of various definitions of invalid performance in two forced-choice recognition performance validity tests (PVTs; FCRCVLT-II and Test of Memory Malingering [TOMM-2]). The proportion of at and below chance level responding defined by the binomial theory and making any errors was computed across two mixed clinical samples from the United States and Canada (N = 470) and two sets of criterion PVTs. There was virtually no overlap between the binomial and empirical distributions. Over 95% of patients who passed all PVTs obtained a perfect score. At chance level responding was limited to patients who failed ≥2 PVTs (91% of them failed 3 PVTs). No one scored below chance level on FCRCVLT-II or TOMM-2. All 40 patients with dementia scored above chance. Although at or below chance level performance provides very strong evidence of non-credible responding, scores above chance level have no negative predictive value. Even at chance level scores on PVTs provide compelling evidence for non-credible presentation. A single error on the FCRCVLT-II or TOMM-2 is highly specific (0.95) to psychometrically defined invalid performance. Defining non-credible responding as below chance level scores is an unnecessarily restrictive threshold that gives most examinees with invalid profiles a Pass.

Review of Statistical and Methodological Issues in the Forensic Prediction of Malingering from Validity Tests: Part I: Statistical Issues

Forensic neuropsychological examinations with determination of malingering have tremendous social, legal, and economic consequences. Thousands of studies have been published aimed at developing and validating methods to diagnose malingering in forensic settings, based largely on approximately 50 validity tests, including embedded and stand-alone performance validity tests. This is the first part of a two-part review. Part I explores three statistical issues related to the validation of validity tests as predictors of malingering, including (a) the need to report a complete set of classification accuracy statistics, (b) how to detect and handle collinearity among validity tests, and (c) how to assess the classification accuracy of algorithms for aggregating information from multiple validity tests. In the Part II companion paper, three closely related research methodological issues will be examined. Statistical issues are explored through conceptual analysis, statistical simulations, and through reanalysis of findings from prior validation studies. Findings suggest extant neuropsychological validity tests are collinear and contribute redundant information to the prediction of malingering among forensic examinees. Findings further suggest that existing diagnostic algorithms may miss diagnostic accuracy targets under most realistic conditions. The review makes several recommendations to address these concerns, including (a) reporting of full confusion table statistics with 95% confidence intervals in diagnostic trials, (b) the use of logistic regression, and (c) adoption of the consensus model on the "transparent reporting of multivariate prediction models for individual prognosis or diagnosis" (TRIPOD) in the malingering literature.

Best Practices and Methodological Strategies for Addressing Generalizability in Neuropsychological Assessment

Generalizability considerations are widely discussed and a core foundation for understanding when and why treatment effects will replicate across sample demographics. However, guidelines on assessing and reporting generalizability-related factors differ across fields and are inconsistently applied. This paper synthesizes obstacles and best practices to apply recent work on measurement and sample diversity. We present a brief history of how knowledge in psychology has been constructed, with implications for who has been historically prioritized in research. We then review how generalizability remains a contemporary threat to neuropsychological assessment and outline best practices for researchers and clinical neuropsychologists. In doing so, we provide concrete tools to evaluate whether a given assessment is generalizable across populations and assist researchers in effectively testing and reporting treatment differences across sample demographics.

Time is money: Examining the time cost and associated charges of common performance validity tests

Objective: This study presents data on the time cost and associated charges for common performance validity tests (PVTs). It also applies an approach from cost effectiveness research to comparison of tests that incorporates cost and classification accuracy. Method: A recent test usage survey was used to identify PVTs in common use among adult neuropsychologists. Data on test administration and scoring time were aggregated. Charges per test were calculated. A cost effectiveness approach was applied to compare pairs of tests from three studies using data on test administration time and classification accuracy operationalized as improvement in posterior probability beyond base rate. Charges per unit increase in posterior probability over base rate were calculated for base rates of invalidity ranging from 10 to 40%. Results: Ten commonly used PVTs measures showed a wide range in test administration and scoring time from 1 to 3 minutes to over 40 minutes with associated charge estimates from $4 to $284. Cost effectiveness comparisons illustrated the nuance in test selection and benefit of considering cost in relation to outcome rather than prioritizing time (i.e. cost minimization) classification accuracy alone. Conclusions: Findings extend recent research efforts to fill knowledge gaps related to the cost of neuropsychological evaluation. The cost effectiveness approach warrants further study in other samples with different neuropsychological and outcome measures.

Adaptation and initial validation of the Arabic version of the Word Memory Test (WMTARB)

BACKGROUND

The feigning of cognitive impairment is common in neuropsychological assessments, especially in a medicolegal setting. The Word Memory Test (WMT) is a forced-choice recognition memory performance validity test (PVT) which is widely used to detect noncredible performance. Though translated to several languages, this was not done for one of the most common languages, Arabic. The aim of the current study was to evaluate the convergent validity of the Arabic adaptation of the WMT (WMT_ARB) among Israeli Arabic speakers.

METHODS

We adapted the WMT to Arabic using the back-translation method and in accordance with relevant guidelines. We then randomly assigned healthy Arabic speaking adults (N = 63) to either a simulation or honest control condition. The participants then performed neuropsychological tests which included the WMT_ARB and the Test of Memory Malingering (TOMM), a well-validated nonverbal PVT.

RESULTS

The WMT_ARB had high split-half reliability and its measures were significantly correlated with that of the TOMM (p < .001). High concordance was found in classification of participants using the WMT_ARB and TOMM (specificity = 94.29% and sensitivity = 100% using the conventional TOMM trial 2 cutoff as gold standard). As expected, simulators' accuracy on the WMT_ARB was significantly lower than that of honest controls. None of the demographic variables significantly correlated with WMT_ARB measures.

CONCLUSION

The WMT_ARB shows initial evidence of reliability and validity, emphasizing its potential use in the large population of Arabic speakers and universality in detecting noncredible performance. The findings, however, are preliminary and mandate validation in clinical settings.

Low rate of performance validity failures among individuals with bipolar disorder

OBJECTIVE

Assessing performance validity is imperative in both clinical and research contexts as data interpretation presupposes adequate participation from examinees. Performance validity tests (PVTs) are utilized to identify instances in which results cannot be interpreted at face value. This study explored the hit rates for two frequently used PVTs in a research sample of individuals with and without histories of bipolar disorder (BD).

METHOD

As part of an ongoing longitudinal study of individuals with BD, we examined the performance of 736 individuals with BD and 255 individuals with no history of mental health disorder on the Test of Memory Malingering (TOMM) and the California Verbal Learning Test forced choice trial (CVLT-FC) at three time points.

RESULTS

Undiagnosed individuals demonstrated 100% pass rate on PVTs and individuals with BD passed over 98% of the time. A mixed effects model adjusting for relevant demographic variables revealed no significant difference in TOMM scores between the groups, a = .07, SE = .07, p = .31. On the CVLT-FC, no clinically significant differences were observed (ps < .001).

CONCLUSIONS

Perfect PVT scores were obtained by the majority of individuals, with no differences in failure rates between groups. The tests have approximately >98% specificity in BD and 100% specificity among non-diagnosed individuals. Further, nearly 90% of individuals with BD obtained perfect scores on both measures, a trend observed at each time point.

Validation of alternative dot counting test E-score cutoffs based on degree of cognitive impairment in veteran and civilian clinical samples

OBJECTIVE

This study examined Dot Counting Test (DCT) performance among patient populations with no/minimal impairment and mild impairment in an attempt to cross-validate a more parsimonious interpretative strategy and to derive optimal E-Score cutoffs.

METHOD

Participants included clinically-referred patients from VA (n = 101) and academic medical center (AMC, n = 183) settings. Patients were separated by validity status (valid/invalid), and subsequently two comparison groups were formed from each sample's valid group. Namely, Group 1 included patients with no to minimal cognitive impairment, and Group 2 included those with mild neurocognitive disorder. Analysis of variance tested for differences between rounded and unrounded DCT E-Scores across both comparison groups and the invalid group. Receiver operating characteristic curve analyses identified optimal validity cut-scores for each sample and stratified by comparison groups.

RESULTS

In the VA sample, cut scores of ≥13 (rounded) and ≥12.58 (unrounded) differentiated Group 1 from the invalid performers (87% sensitivity/88% specificity), and cut scores of ≥17 (rounded; 58% sensitivity/90% specificity) and ≥16.49 (unrounded; 61% sensitivity/90% specificity) differentiated Group 2 from the invalid group. Similarly, in the AMC group, a cut score of ≥13 (rounded and unrounded; 75% sensitivity/90% specificity) differentiated Group 1 from the invalid group, whereas cut scores of ≥18 (rounded; 43% sensitivity/94% specificity) and ≥16.94 (unrounded; 46% sensitivity/90% specificity) differentiated Group 2 from the invalid performers.

CONCLUSIONS

Different cut scores were indicated based on degree of cognitive impairment, and provide proof-of-concept for a more parsimonious interpretative paradigm than using individual cut scores derived for specific diagnostic groups.

Baseline executive functioning moderates treatment-related changes in quality of life in veterans with posttraumatic stress disorder and comorbid traumatic brain injury

Posttraumatic stress disorder (PTSD) treatment has been associated with improvement in quality of life (QOL); however, little is known about factors that moderate treatment-related changes in QOL, particularly cognitive factors. Executive functioning (EF) is important for success across all aspects of everyday life and predicts better psychological and physical health. EF is important to QOL, but more work is needed to better understand the association between EF and QOL improvements following interventions. We hypothesized that poorer baseline EF would be associated with less improvement in overall life satisfaction and satisfaction with health following PTSD treatment. U.S. veterans who served after the September 11, 2001 terrorist attacks (post 9-11; N = 80) with PTSD and a history of mild-to-moderate traumatic brain injury were randomized to standard cognitive processing therapy (CPT) or CPT combined with cognitive rehabilitation (SMART-CPT). Multilevel modeling was used to examine whether baseline EF performance was associated with changes in QOL scores from pretreatment to follow-up across both groups. Results indicated that poorer baseline performance on EF tests of working memory and inhibition were associated with less treatment-related improvements in general life satisfaction and satisfaction with health, rs = .26-.36. Treatment condition did not moderate any results. Future research should examine whether implementing EF-focused techniques before and/or concurrently with CPT for individuals with poorer baseline working memory and inhibition enhances QOL treatment gains, particularly in terms of general life and health-related satisfaction.

Self-reported disability-seeking predicts PVT failure in veterans undergoing clinical neuropsychological evaluation

Objective: This study examined disability-related factors as predictors of PVT performance in Veterans who underwent neuropsychological evaluation for clinical purposes, not for determination of disability benefits. Method: Participants were 1,438 Veterans who were seen for clinical evaluation in a VA Medical Center's Neuropsychology Clinic. All were administered the TOMM, MSVT, or both. Predictors of PVT performance included (1) whether Veterans were receiving VA disability benefits ("service connection") for psychiatric or neurological conditions at the time of evaluation, and (2) whether Veterans reported on clinical interview that they were in the process of applying for disability benefits. Data were analyzed using binary logistic regression, with PVT performance as the dependent variable in separate analyses for the TOMM and MSVT. Results: Veterans who were already receiving VA disability benefits for psychiatric or neurological conditions were significantly more likely to fail both the TOMM and the MSVT, compared to Veterans who were not receiving benefits for such conditions. Independently of receiving such benefits, Veterans who reported that they were applying for disability benefits were significantly more likely to fail the TOMM and MSVT than were Veterans who denied applying for benefits at the time of evaluation. Conclusions: These findings demonstrate that simply being in the process of applying for disability benefits increases the likelihood of noncredible performance. The presence of external incentives can predict the validity of neuropsychological performance even in clinical, non-forensic settings.

The Pain Disability Index: Effects of performance and symptom validity in mild traumatic brain injury litigants with persistent Post-Concussion pain complaints

OBJECTIVE

Objectives of the current study were to cross validate the Pain Disability Index (PDI) as a measure of symptom validity in a large sample of mild traumatic brain injury (MTBI) litigants with persistent post-concussive pain complaints, and investigate the effects of performance and symptom validity testing on PDI scores.

METHODS

Participants included 91 adults who underwent a comprehensive neuropsychological examination. Criterion groups were formed based upon their performance on stand-alone measures of cognitive performance validity (PVT), and the MMPI-2-RF Symptom Validity Scale (FBS-r) as a measure of symptom validity (SVT).

RESULTS

Participants who failed PVT and SVT scored significantly higher on the PDI compared to participants who passed both. Failing both was associated with a large effect size. Failing PVT, but passing SVT, was associated with a medium effect on PDI scores, while passing PVT, but failing SVT demonstrated a small effect. A PDI cutscore of 49 was associated with .90 specificity and .47 sensitivity.

CONCLUSION

The PDI demonstrates external validity as a self-report measure of symptom validity in MTBI litigants with persistent post-concussive pain complaints. A dose response relationship exists between PVT, SVT and PDI scores. Forensic examiners should include both PVT and SVT to optimize clinical decision making when evaluating MTBI litigants with complaints of pain-related disability years post-incident.

The TOMM1 discrepancy index (TDI): A new performance validity test (PVT) that differentiates between invalid cognitive testing and those diagnosed with dementia

There is a need to develop performance validity tests (PVTs) that accurately identify those with severe cognitive decline but also remain sensitive to those suspected of invalid cognitive testing. The TOMM1 Discrepancy Index (TDI) attempts to address both of these issues. Veterans diagnosed with dementia (n = 251) were administered TOMM1 and the MSVT in order to develop the TDI (TOMM1 percent correct minus MSVT Free Recall percent correct). Cut offs based on the dementia sample were then used to identify those in the non-dementia sample (n = 1,226) suspected of invalid test performance (n = 401). Combining TOMM1 and the TDI in the dementia sample greatly reduced the false positive rate (specificity = 0.97) at a cut off of 28 points or less on the TDI. Those suspected of invalid testing were identified at much higher rates (sensitivity = 0.75) compared to the MSVT genuine memory impairment profile (GMIP, sensitivity = 0.49). By utilizing a neurologically plausible pattern of scores across two PVTs, the TDI correctly classified those with dementia and identified a large percentage with invalid test performance. PVTs utilizing a complex pattern of performance may help reduce one's ability to fabricate cognitive deficits.

Utility of CVLT-3 response bias as a measure of performance validity after traumatic brain injury

OBJECTIVE

We sought to determine the utility of a new performance validity index that was recently proposed. In particular, we wanted to determine if this index would be associated with a specificity of at least .90, a sensitivity of at least .40, and an Area Under the Curve of at least .70 in a traumatic brain injury (TBI) sample.

METHOD

We used logistic regression to investigate how well this new index could distinguish persons with TBI (n = 148) who were evaluated within 1-36 months after injury. All participants had been classified on the basis of at least two independent performance validity tests as having provided valid performance (n = 128) or invalid performance (n = 20).

RESULTS

The new performance validity index had acceptable specificity (.96) but had suboptimal sensitivity (.35) and Area Under the Curve (.66). It was concerning that almost half (5/12) of the cases that were identified by this index as providing invalid effort were false positives. Although a slightly more liberal cut-off improved sensitivity, the problem with poor positive predictive power remained. The conventional Forced Choice index had relatively better classification accuracy.

CONCLUSION

Differences in base rates between the original sample of Martin et al. and the current one most likely affected positive predictive power of the new index. Although their performance validity has excellent specificity, the current results do not support the application of this index in the clinical evaluation of patients with traumatic brain injury when base rates of invalid performance differ markedly from those in the original study.

Comprehensive Analysis of MMPI-2-RF Symptom Validity Scales and Performance Validity Test Relationships in a Diverse Mixed Neuropsychiatric Setting

The utility of symptom (SVT) and performance (PVT) validity tests has been independently established in neuropsychological evaluations, yet research on the relationship between these two types of validity indices is limited to circumscribed populations and measures. This study examined the relationship between SVTs on the Minnesota Multiphasic Personality Inventory-2-Restructured Form (MMPI-2-RF) and PVTs in a mixed neuropsychiatric setting. This cross-sectional study included data from 181 diagnostically and demographically diverse patients with neuropsychiatric conditions referred for outpatient clinical neuropsychological evaluation at an academic medical center. All patients were administered a uniform neuropsychological battery, including the MMPI-2-RF and five PVTs (i.e., Dot Counting Test; Medical Symptom Validity Test; Reliable Digit Span; Test of Memory Malingering-Trial 1; Word Choice Test). Nonsignificant associations emerged between SVT and PVT performance. Although the Response Bias Scale was most predictive of PVT performance, MMPI-2-RF SVTs generally had low classification accuracy for predicting PVT performance. Neuropsychological test performance was related to MMPI-2-RF SVT status only when overreporting elevations were at extreme scores. The current study further supports that SVTs and PVTs measure unique and dissociable constructs among diverse patients with neuropsychiatric conditions, consistent with literature from other clinical contexts. Therefore, objective evidence of symptom overreporting on MMPI-2-RF SVTs cannot be interpreted as definitively indicating invalid performance on tests of neurocognitive abilities. As such, clinicians should include both SVTs and PVTs as part of a comprehensive neuropsychological evaluation as they provide unique information regarding performance and symptom validity.

The Continuous Visual Memory Test: Update and extension on the operating characteristics as an embedded measure of cognitive performance validity

Objective: To compare and update predictive models comprised of embedded measures from the Continuous Visual Memory Test (CVMT) in their ability to predict performance validity in personal injury litigants. Methods: Ninety-two personal injury litigants underwent a comprehensive neuropsychological examination. Criterion groups were formed, i.e. PVT-Pass and PVT-Fail, based upon their performance on stand-alone measures of performance validity (PVT). Independent-samples t-tests investigated group differences on dependent variables of interest while logistic regression analyses, as well as a decision tree classification procedure, were employed to identify the best predictive model. Results: The PVT-Fail group scored significantly lower on the 20-item Larrabee Index (LI), and three CVMT variables comprising the Henry-Enders Index (HEI) including Hits, Total Score, and Delayed Recall, but significantly higher on False Alarm Errors. Although the Total score was the best single predictor of PVT status, the addition of LI improved sensitivity. The best predictive model was derived via a classification and regression tree analysis which selected LI and CVMT-FA resulting in .91 specificity, .60 sensitivity, and ROC = 0.832. Conclusion: In the current study total CVMT scores < 70, and LI scores < 18 were rare for PI litigants with MTBI and not seen in litigants with moderate and severe brain injury who passed PVTs. Three predictive CVMT models were derived. When failure on one of the models is observed then concerns about the credibility of visual memory performance should be considered with particular attention to other stand-alone and embedded measures of performance validity.

The rate of psychiatric disorders in veterans undergoing intensive EEG monitoring is associated with symptom and performance invalidity

To determine if the number of participants with psychiatric disorders increased in association with failures on symptom validity tests (SVTs) and a performance validity test (PVT) in Veterans admitted for evaluation of possible seizures.

The 254 participants were Veterans undergoing inpatient video-EEG monitoring for the diagnosis of possible seizures. DSM-IV psychiatric disorders were diagnosed with the SCID IV. Symptom exaggeration was assessed with the MMPI-2-RF and performance validity with the TOMM.

On the MMPI-2-RF, 27.6%-32.7% showed symptom exaggeration. Participants who exaggerated on the MMPI-2-RF were more often diagnosed with psychiatric disorders. The TOMM was failed by 15.4% of the sample. Participants who failed the TOMM were more often diagnosed with an Axis I disorder but not with a personality disorder. The MMPI-2-RF was invalid in more cases than the TOMM, but 7.9% of the sample generated a valid MMPI-2-RF and an invalid TOMM.

The correlational design does not allow conclusions about cause and effect. The invalid groups may have had a higher rate of psychopathology. The number of participants with psychiatric disorders increased in association with symptom exaggeration and performance invalidity. Symptom exaggeration was more frequent than performance invalidity, but the TOMM made a unique contribution to identification of invalidity. The routine clinical use of SVTs and PVTs is supported. The results also suggest the need for caution in diagnosing psychiatric disorders when there is symptom exaggeration or performance invalidity, because diagnostic validity is dependent on the accuracy of symptom reporting.

The minimal effect of depression on cognitive functioning when accounting for TOMM performance in a sample of U.S. veterans

While many studies have demonstrated a relationship between depression and cognitive deficits, most have neglected to include measurements of performance validity. This study examined the relationship between depression and cognition after accounting for noncredible performance. Participants were veterans referred for outpatient clinical evaluation. The first set of regression analyses (N = 187) included age, sex, and education in Model 1, Beck Depression Inventory-2 (BDI-2) added in Model 2, and pass/failure of Test of Memory Malingering (TOMM) added in Model 3 as predictors of 12 neuropsychological test indices. The second set of analyses (N = 559) mirrored the first but with Major Depressive Disorder (MDD) diagnosis in Models 2 and 3. In the first analyses, after including TOMM in the model, only the relationship between BDI-2 and verbal fluency remained significant, but this did not survive a Bonferroni correction. In the second analyses, after including TOMM and Bonferroni correction, MDD diagnosis was a significant predictor only for CVLT-II Short Delay Free Recall. Therefore, the relationship between depression and cognition may not be driven by frank cognitive impairment, but rather by psychological mechanisms, which has implications for addressing depressed individuals' concerns about their cognitive functioning and suggest the value of providing psychoeducation and reassurance.

A Direct Comparison of 10 WAIS-IV Digit Span Embedded Validity Indicators among a Mixed Neuropsychiatric Sample with Varying Degrees of Cognitive Impairment

Objective

Reliable Digit Span (RDS), RDS-Revised (RDS-R), and age-corrected scaled score (ACSS) have been previously validated as embedded performance validity tests (PVTs) from the Wechsler Adult Intelligence Scale-IV Digit Span subtest (WAIS-IV DS). However, few studies have directly compared the relative utility of these and other proposed WAIS-IV DS validity indicators within a single sample.

Method

This study compared classification accuracies of 10 WAIS-IV DS indices in a mixed neuropsychiatric sample of 227 outpatients who completed a standardized neuropsychological battery. Participants with ≤1 PVT failures of the four, freestanding criterion PVTs constituted the valid group (n = 181), whereas those with ≥2 PVT failures formed the invalid group (n = 46). Among the valid group, 113 met criteria for mild cognitive impairment (MCI).

Results

Classification accuracies for all DS indicators were statistically significant across the overall sample and subsamples with and without MCI, apart from indices derived from the Forward trial in the MCI sample. DS Sequencing ACSS, working memory RDS (wmRDS), and DS ACSS emerged as the most effective predictors of validity status, with acceptable to excellent classification accuracy for the overall sample (AUCs = 0.792–0.816; 35%–50% sensitivity/88%–96% specificity).

Conclusions

Although most DS indices demonstrated clinical utility as embedded PVTs, DS Sequencing ACSS, wmRDS, and DS ACSS may be particularly robust to cognitive impairment, minimizing risk of false positive errors while identifying noncredible performance. Moreover, DS indices incorporating data from multiple trials (i.e., wmRDS, DS ACSS) also generally yielded greater classification accuracy than those derived from a single trial.

A case report and literature review of cognitive malingering and psychopathology

Malingering of cognitive difficulties constitutes a major issue in psychiatric forensic settings. Here, we present a selective literature review related to the topic of cognitive malingering, psychopathology and their possible connections. Furthermore, we report a single case study of a 60-year-old man with a long and ongoing judicial history who exhibits a suspicious multi-domain neurocognitive disorder with significant reduction of autonomy in daily living, alongside a longtime history of depressive symptoms. Building on this, we suggest the importance of evaluating malingering conditions through both psychiatric and neuropsychological assessment tools. More specifically, the use of Performance Validity Tests (PVTs)-commonly but not quite correctly considered as tests of "malingering"-alongside the collection of clinical history and the use of routine psychometric testing, seems to be crucial in order to detect discrepancies between self-reported patient's symptoms, embedded validity indicators and psychometric results.

An alternative approach to TOMM cutoff scores using a large sample of military personnel

The accuracy of neuropsychological assessments relies on participants exhibiting their true abilities during administration. The Test of Memory Malingering (TOMM) is a popular performance validity test used to determine whether an individual is providing honest answers. While the TOMM has proven to be highly sensitive to those who are deliberately exaggerating their symptoms, there is a limited explanation regarding the significance of using 45 as a cutoff score. The present study aims to further investigate this question by examining TOMM scores obtained in a large sample of active-duty military personnel (N = 859, M = 26 years, SD = 6.14, 97.31% males, 72.44% white). Results indicated that no notable discrepancies existed between the frequency of participants who scored a 45 and those who scored slightly below a 45 on the TOMM. The sensitivity and specificity of the TOMM were derived using the forced-choice recognition (FCR) scores obtained by participants on the California Verbal Learning Test, Second Edition (CVLT-II). The sensitivity for each trial of the TOMM was 0.84, 0.55, and 0.63, respectively; the specificity for each trial of the TOMM was 0.69, 0.93, and 0.92, respectively. Because sensitivity and specificity rates are both of importance in this study, balanced accuracy scores were also reported. Results suggested that various alternative cutoff scores produced a more accurate classification compared to the traditional cutoff of 45. Further analyses using Fisher's exact test also indicated that there were no significant performance differences on the FCR of the CVLT-II between individuals who received a 44 and individuals who received a 45 on the TOMM. The current study provides evidence on why the traditional cutoff may not be the most effective score. Future research should consider employing alternative methods which do not rely on a single score.

Multivariable analysis of the relative utility and additive value of eight embedded performance validity tests for classifying invalid neuropsychological test performance

INTRODUCTION

This study investigated a combination of eight embedded performance validity tests (PVTs) derived from commonly administered neuropsychological tests to optimize sensitivity/specificity for detecting invalid neuropsychological test performance. The goal of this study was to evaluate what combination of these common embedded PVTs that have the most robust predictive power for detecting invalid neuropsychological test performance in a single diverse clinical sample.

METHOD

Eight previously validated memory- and nonmemory-based embedded PVTs were examined among 231 patients undergoing neuropsychological evaluation. Patients were classified into valid/invalid groups based on four independent criterion PVTs. Embedded PVT accuracy was assessed using standard and stepwise multiple logistic regression models.

RESULTS

Three PVTs, the Brief Visuospatial Memory Test-Revised Recognition Discrimination (BVMT-R-RD), Rey Auditory Verbal Learning Test Forced Choice, and WAIS-IV Digit Span Age Corrected Scaled Score, predicted 45.5% of the variance in validity group membership. BVMT-RD independently accounted for 32% of the variance in prediction of independent, criterion-defined validity group membership.

CONCLUSIONS

This study demonstrated the incremental predictive power of multiple embedded PVTs derived from common neuropsychological measures in detecting invalid test performance and those measures accounting for the greatest portion of the variance. These results provide guidance for evaluating the most fruitful embedded PVTs and proof of concept to better guide selection of embedded validity indices. Further, this offers clinicians an efficient, empirically derived approach to assessing performance validity when time restraints potentially limit the use of freestanding PVTs.

Base rates of invalidity when patients undergoing routine clinical evaluations have social security disability as an external incentive

Objective: Social Security Disability is a common external incentive in neuropsychological evaluations. This study determined base rates of invalidity when patients referred for routine clinical evaluations have Social Security Disability as an external incentive. Method: Patients (n = 242) were grouped as validly or invalidly performing based on the use of multiple performance validity tests. Frequency analyses were then conducted. Results: As a whole, 46.0% of clinically referred patients with Social Security Disability as an external incentive produced invalid data. When divided by disability pursuit status, 58.6% of individuals already receiving Social Security Disability, 44.6% of individuals actively seeking Social Security Disability, and 39.3% of individuals considering seeking Social Security Disability produced invalid data. By comparison, only 8.5% of clinically referred patients without known external incentives produced invalid data. Conclusions: Beyond establishing base rates, these data indicate that the external incentive, not necessarily the evaluation setting, increases the rate of invalidity, as obtained base rates mirror those observed in independent medical examinations. In addition, this study highlights that even patients who report that they are considering but have not committed themselves to pursuing an external incentive frequently invalidate testing.

Replication and cross-validation of the personality assessment inventory (PAI) cognitive bias scale (CBS) in a mixed clinical sample

Objective: This study is a cross-validation of the Cognitive Bias Scale (CBS) from the Personality Assessment Inventory (PAI), a ten-item scale designed to assess symptom endorsement associated with performance validity test failure in neuropsychological samples. The study utilized a mixed neuropsychological sample of consecutively referred patients at a large academic medical center in the Midwest. Participants and Methods: Participants were 332 patients who completed embedded and free-standing performance validity tests (PVTs) and the PAI. Pass and fail groups were created based on PVT performance to evaluate classification accuracy of the CBS. Results: The results were generally consistent with the initial study for overall classification accuracy, sensitivity, and cut-off score. Consistent with the validation study, CBS had better classification accuracy than the original PAI validity scales and a comparable effect size to that obtained in the original validation publication; however, the Somatic Complaints scale (SOM) and the Conversion subscale (SOM-C) also demonstrated good classification accuracy. The CBS had incremental predictive ability compared to existing PAI scales. Conclusions: The results supported the CBS, but further research is needed on specific populations. Findings from this present study also suggest the relationship between conversion tendencies and PVT failure may be stronger in some geographic locations or population types (forensic versus clinical patients).

Validation of the Personality Assessment Inventory (PAI) scale of scales in a mixed clinical sample

Objective: This exploratory study examined the classification accuracy of three derived scales aimed at detecting cognitive response bias in neuropsychological samples. The derived scales are composed of existing scales from the Personality Assessment Inventory (PAI). A mixed clinical sample of consecutive outpatients referred for neuropsychological assessment at a large Midwestern academic medical center was utilized. Participants and Methods: Participants included 332 patients who completed study's embedded and free-standing performance validity tests (PVTs) and the PAI. PASS and FAIL groups were created based on PVT performance to evaluate the classification accuracy of the derived scales. Three new scales, Cognitive Bias Scale of Scales 1-3, (CB-SOS1-3) were derived by combining existing scales by either summing the scales together and dividing by the total number of scales summed, or by logistically deriving a variable from the contributions of several scales. Results: All of the newly derived scales significantly differentiated between PASS and FAIL groups. All of the derived SOS scales demonstrated acceptable classification accuracy (i.e. CB-SOS1 AUC = 0.72; CB-SOS2 AUC = 0.73; CB-SOS3 AUC = 0.75). Conclusions: This exploratory study demonstrates that attending to scale-level PAI data may be a promising area of research in improving prediction of PVT failure.

Assessment of differential neurocognitive performance based on the number of performance validity tests failures: A cross-validation study across multiple mixed clinical samples

Objective: This cross-sectional study examined the effect of number of Performance Validity Test (PVT) failures on neuropsychological test performance among a demographically diverse Veteran (VA) sample (n = 76) and academic medical sample (AMC; n = 128). A secondary goal was to investigate the psychometric implications of including versus excluding those with one PVT failure when cross-validating a series of embedded PVTs. Method: All patients completed the same six criterion PVTs, with the AMC sample completing three additional embedded PVTs. Neurocognitive test performance differences were examined based on number of PVT failures (0, 1, 2+) for both samples, and effect of number of criterion failures on embedded PVT performance was analyzed among the AMC sample. Results: Both groups with 0 or 1 PVT failures performed better than those with ≥2 PVT failures across most cognitive tests. There were nonsignificant differences between those with 0 or 1 PVT failures except for one test in the AMC sample. Receiver operator characteristic curve analyses found no differences in optimal cut score based on number of PVT failures when retaining/excluding one PVT failure. Conclusion: Findings support the use of ≥2 PVT failures as indicative of performance invalidity. These findings strongly support including those with one PVT failure with those with zero PVT failures in diagnostic accuracy studies, given that their inclusion reflects actual clinical practice, does not reduce sample sizes, and does not artificially deflate neurocognitive test results or inflate PVT classification accuracy statistics.

Embedded cognitive and emotional/affective self-reported symptom validity indices on the patient competency rating scale

OBJECTIVE

Although there is an abundance of research on stand-alone and embedded performance validity tests and stand-alone symptom validity tests (SVTs), less emphasis has been placed on embedded SVTs. The goal of the current study was to examine the ability of embedded indicators within the Patient Competency Rating Scale (PCRS) to separately detect invalid cognitive and/or emotional/affective symptom responding.

METHOD

Participants included 299 veterans assessed in a VA medical center epilepsy monitoring unit from 2013-2017 (mean age = 48.8 years, SD = 13.5 years). Two SVT composites were created; self-reported cognitive symptom validity (SVT-C) and self-reported emotional/affective symptom validity (SVT-E). Groups were compared on PCRS total and index scores (i.e., cognitive, activities of daily living, emotional, and interpersonal competencies) using ANOVAs. Receiver operating characteristic (ROC) curve analyses assessed the classification accuracy of the PCRS total and index scores for SVT-C and SVT-E.

RESULTS

In ANOVAs, SVT-C was significantly associated with all PCRS indices, while SVT-E was only significantly associated with the PCRS total, emotional, and interpersonal competency indices. Although the PCRS-T ≤ 90 had the strongest classification of SVT-C and SVT-E (specificities: .90, sensitivities: .44 to .50), PCRS index scores showed suggestive evidence of domain specificity, with PCRS-ADL ≤22, PCRS-C ≤ 20, and PCRS-CADL ≤45 best classifying SVT-C (specificities: .92, sensitivities: .33) and the PCRS-E ≤ 18 best classifying the SVT-E group (specificity: .93, sensitivity: .40).

CONCLUSION

Results suggest the PCRS may be used to obtain clinically useful information while including embedded indicators that can assess cognitive and/or emotional/affective symptom invalidity.

Utility of the D-KEFS color word interference test as an embedded measure of performance validity after traumatic brain injury

ObjectiveWe sought to determine the accuracy of embedded performance measures for the D-KEFS Color Word Interference Test that were recently proposed by Eglit et al. In particular, we wanted to determine if these indices would be associated with a specificity of at least .90, an Area Under the Curve of at least .70 and a positive likelihood ratio of at least 2. Method: We used logistic regression to investigate how well these indices could distinguish persons with traumatic brain injury (n = 169) who were evaluated within 1-12 months after injury. All participants had been classified on the basis of at least three independent performance validity tests as valid performance (n = 145) or invalid performance (n = 24). Results: None of the three indices that Eglit et al. had proposed as embedded performance measures for the D-KEFS Color Word Interference Test achieved the a priori defined minimally acceptable level of specificity. One of them did meet the criteria for Area Under the Curve as well as positive likelihood ratio. Conclusion: The current results do not support the application of the Eglit et al. embedded performance validity measures for the D-KEFS Color Word Interference Test in the clinical evaluation of patients with traumatic brain injury.

All of the accuracy in half of the time: Assessing abbreviated versions of the Test of Memory Malingering in the context of verbal and visual memory impairment

ObjectiveThe Test of Memory Malingering (TOMM) Trial 1 (T1) and errors on the first 10 items of T1 (T1-e10) were developed as briefer versions of the TOMM to minimize evaluation time and burden, although the effect of genuine memory impairment on these indices is not well established. This study examined whether increasing material-specific verbal and visual memory impairment affected T1 and T1-e10 performance and accuracy for detecting invalidity. Method: Data from 155 neuropsychiatric patients administered the TOMM, Rey Auditory Verbal Learning Test (RAVLT), and Brief Visuospatial Memory Test-Revised (BVMT-R) during outpatient evaluation were examined. Valid (N = 125) and invalid (N = 30) groups were established by four independent criterion performance validity tests. Verbal/visual memory impairment was classified as ≥37 T (normal memory); 30 T-36T (mild impairment); and ≤29 T (severe impairment). Results: Overall, T1 had outstanding accuracy, with 77% sensitivity/90% specificity. T1-e10 was less accurate but had excellent discriminability, with 60% sensitivity/87% specificity. T1 maintained excellent accuracy regardless of memory impairment severity, with 77% sensitivity/≥88% specificity and a relatively invariant cut-score even among those with severe verbal/visual memory impairment. T1-e10 had excellent classification accuracy among those with normal memory and mild impairment, but accuracy and sensitivity dropped with severe impairment and the optimal cut-score had to be increased to maintain adequate specificity. Conclusion: TOMM T1 is an effective performance validity test with strong psychometric properties regardless of material-specificity and severity of memory impairment. By contrast, T1-e10 functions relatively well in the context of mild memory impairment but has reduced discriminability with severe memory impairment.

Response time measures on the Word Memory Test do not add incremental validity to accuracy scores in predicting noncredible neurocognitive dysfunction in mild traumatic brain injury litigants

The objective of the current study was to investigate whether response time measures on the Word Memory Test (WMT) increase predictive validity on determining noncredible neurocognitive dysfunction in a large sample of mild traumatic brain injury (MTBI) litigants. Participants included 203 adults who underwent a comprehensive neuropsychological examination. Criterion groups were formed based upon their performance on stand-alone measures of cognitive performance validity (PVT). Participants failing PVTs exhibited significantly slower response times and lower accuracy on the WMT compared to participants who passed PVTs. Response time measures did not add significant incremental validity beyond that afforded by WMT accuracy measures alone. The best predictor of PVT status was the WMT Consistency Score (CNS) which was associated with an extremely large effect size (d = 16.44), followed by Immediate Recognition (IR: d = 10.68) and Delayed Recognition (DR: d = 10.10).

Cross-Cultural Performance Validity Testing: Managing False Positives in Examinees with Limited English Proficiency

Base rates of failure (BR_Fail) on performance validity tests (PVTs) were examined in university students with limited English proficiency (LEP). BR_Fail was calculated for several free-standing and embedded PVTs. All free-standing PVTs and certain embedded indicators were robust to LEP. However, LEP was associated with unacceptably high BR_Fail (20-50%) on several embedded PVTs with high levels of verbal mediation (even multivariate models of PVT could not contain BR_Fail). In conclusion, failing free-standing/dedicated PVTs cannot be attributed to LEP. However, the elevated BR_Fail on several embedded PVTs in university students suggest an unacceptably high overall risk of false positives associated with LEP.

BNT-15: Revised Performance Validity Cutoffs and Proposed Clinical Classification Ranges

BACKGROUND

Abbreviated neurocognitive tests offer a practical alternative to full-length versions but often lack clear interpretive guidelines, thereby limiting their clinical utility.

OBJECTIVE

To replicate validity cutoffs for the Boston Naming Test-Short Form (BNT-15) and to introduce a clinical classification system for the BNT-15 as a measure of object-naming skills.

METHOD

We collected data from 43 university students and 46 clinical patients. Classification accuracy was computed against psychometrically defined criterion groups. Clinical classification ranges were developed using a z -score transformation.

RESULTS

Previously suggested validity cutoffs (≤11 and ≤12) produced comparable classification accuracy among the university students. However, a more conservative cutoff (≤10) was needed with the clinical patients to contain the false-positive rate (0.20-0.38 sensitivity at 0.92-0.96 specificity). As a measure of cognitive ability, a perfect BNT-15 score suggests above average performance; ≤11 suggests clinically significant deficits. Demographically adjusted prorated BNT-15 T-scores correlated strongly (0.86) with the newly developed z -scores.

CONCLUSION

Given its brevity (<5 minutes), ease of administration and scoring, the BNT-15 can function as a useful and cost-effective screening measure for both object-naming/English proficiency and performance validity. The proposed clinical classification ranges provide useful guidelines for practitioners.