Expert beliefs and practices regarding neuropsychological validity testing

Psychometric properties of the Portuguese version of the Personality Assessment Inventory: normative data and reliability

Introduction

Originally published in the United States of America in 1991, the Personality Assessment Inventory (PAI) has been translated and adapted to a growing number of countries, but Portugal had yet to study its adequacy to the Portuguese population.

Methods

The current study aimed to investigate the Portuguese normative data, the predictive effect of sociodemographic variables on the PAI scores, and the reliability of the Portuguese version of the PAI. Additionally, results were compared with other international versions of the PAI. The sample was comprised of 900 participants (age: M = 43.13, SD = 14.28, range = 18-75), recruited from various regions of Portugal.

Results

Findings showed that the Portuguese sample scored higher than the U.S. and other international versions of the PAI in most scales. Sociodemographic variables (e.g., gender, age, and educational level) were significant predictors on PAI scores. The internal consistency of the Portuguese sample revealed lower values on the validity scales, but adequate on the clinical, treatment, and interpersonal scales. Overall, the Portuguese PAI revealed adequate psychometric properties, with normative results often superior to other international versions of the inventory.

Discussion

It is a crucial step into the Portuguese adaptation and validation of this instrument, a measure with considerable potential in clinical, forensic, and research contexts. This adaptation may lead to the growth and development of the psychological assessment field in Portugal, and the opportunity to develop future cross-cultural studies with other international versions of the PAI.

Ability of the Wisconsin Card-Sorting Test-64 as an embedded measure to identify noncredible neurocognitive performance in mild traumatic brain injury litigants

OBJECTIVE

To investigate the ability of selective measures on the Wisconsin Card Sorting Test-64 (WCST-64) to predict noncredible neurocognitive dysfunction in a large sample of mild traumatic brain injury (mTBI) litigants.

METHOD

Participants included 114 adults who underwent a comprehensive neuropsychological examination. Criterion groups were formed based upon their performance on stand-alone measures of cognitive performance validity (PVT).

RESULTS

Participants failing PVTs performed worse across all WCST-64 dependent variables of interest compared to participants who passed PVTs. Receiver operating curve analysis revealed that only categories completed was a significant predictors of PVT status. Multivariate logistic regression did not add to classification accuracy.

CONCLUSION

Consideration of noncredible executive functioning may be warranted in mild traumatic brain injury (mTBI) litigants who complete ≤ 1 category on the WCST-64.

The impact of race and other demographic factors on the false positive rates of five embedded Performance Validity Tests (PVTs) in a Veteran sample

INTRODUCTION

It is common to use normative adjustments based on race to maintain accuracy when interpreting cognitive test results during neuropsychological assessment. However, embedded performance validity tests (PVTs) do not adjust for these racial differences and may result in elevated rates of false positives in African American/Black (AA) samples compared to European American/White (EA) samples.

METHODS

Veterans without Major Neurocognitive Disorder completed an outpatient neuropsychological assessment and were deemed to be performing in a valid manner (e.g., passing both the Test of Memory Malingering Trial 1 (TOMM1) and the Medical Symptom Validity Test (MSVT), (n = 531, EA = 473, AA = 58). Five embedded PVTs were administered to all patients: WAIS-III/IV Processing Speed Index (PSI), Brief Visuospatial Memory Test-Revised: Discrimination Index (BVMT-R), TMT-A (secs), California Verbal Learning Test-II (CVLT-II) Forced Choice, and WAIS-III/IV Digit Span Scaled Score. Individual PVT false positive rates, as well as the rate of failing two or more embedded PVTs, were calculated.

RESULTS

Failure rates of two embedded PVTs (PSI, TMT-A), and the total number of PVTs failed, were higher in the AA sample. The PSI and TMT-A remained significantly impacted by race after accounting for age, education, sex, and presence of Mild Neurocognitive Disorder. There were PVT failure rates greater than 10% (and considered false positives) in both groups (AA: PSI, TMT-A, and BVMT-R, 12-24%; EA: BVMT-R, 17%). Failing 2 or more PVTs (AA = 9%, EA = 4%) was impacted by education and Mild Neurocognitive Disorder but not by race.

CONCLUSIONS

Individual (timed) PVTs showed higher false positive rates in the AA sample even after accounting for demographic factors and diagnosis of Mild Neurocognitive Disorder. Requiring failure on 2 or more embedded PVTs reduced false positive rates to acceptable levels across both groups (10% or less) and was not significantly influenced by race.

Detection of noncredible cognitive performance with Wechsler Memory Scale-IV measures in mild traumatic brain injury litigants

OBJECTIVE

To investigate the operating characteristics of selective measures on the Wechsler Memory Scale-IV (WMS-IV) to predict noncredible neurocognitive dysfunction in a sample of mild traumatic brain injury (mTBI) litigants.

METHOD

Participants included 110 adults who underwent a comprehensive neuropsychological examination. Criterion groups were formed based upon their performance on stand-alone measures of cognitive performance validity testing (PVT).

RESULTS

Participants failing two stand-alone PVTs exhibited significantly lower scores across all WMS-IV dependent variables of interest compared to participants who passed both PVTs. Participants who failed one PVT were excluded. Bivariate logistic regression revealed that all six dependent variables were significant predictors of PVT status. The best prediction model consisted of three WMS-IV variables including Logical Memory Delayed Recall (LM2), Logical Memory Recognition (LMR), and Visual Reproduction Recognition (VRR). This model demonstrated an accuracy of 90.2%, 0.89 sensitivity, 0.92 specificity, and a Receiver Operating Curve (ROC) of 0.957.

CONCLUSION

The current empirically-derived cutscores and logit equation for the WMS-IV may be an additional consideration in analyzing database validity and noncredible performance in mTBI personal injury litigants ages 18-69.

Quo Vadis Forensic Neuropsychological Malingering Determinations? Reply to Drs. Bush, Faust, and Jewsbury

The thoughtful commentaries in this volume of Drs. Bush, Jewsbury, and Faust add to the impact of the two reviews in this volume of statistical and methodological issues in the forensic neuropsychological determination of malingering based on performance and symptom validity tests (PVTs and SVTs). In his commentary, Dr. Bush raises, among others, the important question of whether such malingering determinations can still be considered as meeting the legal Daubert standard which is the basis for neuropsychological expert testimony. Dr. Jewsbury focuses mostly on statistical issues and agrees with two key points of the statistical review: Positive likelihood chaining is not a mathematically tenable method to combine findings of multiple PVTs and SVTs, and the Simple Bayes method is not applicable to malingering determinations. Dr. Faust adds important narrative texture to the implications for forensic neuropsychological practice and points to a need for research into factors other than malingering that may explain PVT and SVT failures. These commentaries put into even sharper focus the serious questions raised in the reviews about the scientific basis of present practices in the forensic neuropsychological determination of malingering.

Ruling out feigned crime-related amnesia? A response to Acklin (2022)

In a recent paper, Acklin discussed a case of possible amnesia for a murder in terms of neurobiology, psychoanalysis, and personality assessment. Acklin accepted the defendant's claim of amnesia for the crime as genuine. The considerable literature that takes a skeptical view of crime-related amnesia was not cited, and the possibility of feigning or malingering was "ruled out" with a single sentence that does not withstand scrutiny. A review of the literature on feigned amnesia suggests that it may not be possible to rule out malingering even if the best available tools are used: There has been minimal investigation of most validity tests and estimates of base rates of feigned amnesia for a crime vary widely and make estimates of Negative Predictor Power highly unreliable. Although one cannot know from the information presented if Acklin's defendant legitimately experienced amnesia, feigning could not be ruled out using an interview and the test data cited by Acklin. I call for a moratorium on publication of further articles on crime specific amnesia that do not conscientiously examine other potential explanations and do not use current best practices for assessing negative response bias.

Best Practices and Methodological Strategies for Addressing Generalizability in Neuropsychological Assessment

Generalizability considerations are widely discussed and a core foundation for understanding when and why treatment effects will replicate across sample demographics. However, guidelines on assessing and reporting generalizability-related factors differ across fields and are inconsistently applied. This paper synthesizes obstacles and best practices to apply recent work on measurement and sample diversity. We present a brief history of how knowledge in psychology has been constructed, with implications for who has been historically prioritized in research. We then review how generalizability remains a contemporary threat to neuropsychological assessment and outline best practices for researchers and clinical neuropsychologists. In doing so, we provide concrete tools to evaluate whether a given assessment is generalizable across populations and assist researchers in effectively testing and reporting treatment differences across sample demographics.

Time is money: Examining the time cost and associated charges of common performance validity tests

Objective: This study presents data on the time cost and associated charges for common performance validity tests (PVTs). It also applies an approach from cost effectiveness research to comparison of tests that incorporates cost and classification accuracy. Method: A recent test usage survey was used to identify PVTs in common use among adult neuropsychologists. Data on test administration and scoring time were aggregated. Charges per test were calculated. A cost effectiveness approach was applied to compare pairs of tests from three studies using data on test administration time and classification accuracy operationalized as improvement in posterior probability beyond base rate. Charges per unit increase in posterior probability over base rate were calculated for base rates of invalidity ranging from 10 to 40%. Results: Ten commonly used PVTs measures showed a wide range in test administration and scoring time from 1 to 3 minutes to over 40 minutes with associated charge estimates from $4 to $284. Cost effectiveness comparisons illustrated the nuance in test selection and benefit of considering cost in relation to outcome rather than prioritizing time (i.e. cost minimization) classification accuracy alone. Conclusions: Findings extend recent research efforts to fill knowledge gaps related to the cost of neuropsychological evaluation. The cost effectiveness approach warrants further study in other samples with different neuropsychological and outcome measures.

The Pain Disability Index: Effects of performance and symptom validity in mild traumatic brain injury litigants with persistent Post-Concussion pain complaints

OBJECTIVE

Objectives of the current study were to cross validate the Pain Disability Index (PDI) as a measure of symptom validity in a large sample of mild traumatic brain injury (MTBI) litigants with persistent post-concussive pain complaints, and investigate the effects of performance and symptom validity testing on PDI scores.

METHODS

Participants included 91 adults who underwent a comprehensive neuropsychological examination. Criterion groups were formed based upon their performance on stand-alone measures of cognitive performance validity (PVT), and the MMPI-2-RF Symptom Validity Scale (FBS-r) as a measure of symptom validity (SVT).

RESULTS

Participants who failed PVT and SVT scored significantly higher on the PDI compared to participants who passed both. Failing both was associated with a large effect size. Failing PVT, but passing SVT, was associated with a medium effect on PDI scores, while passing PVT, but failing SVT demonstrated a small effect. A PDI cutscore of 49 was associated with .90 specificity and .47 sensitivity.

CONCLUSION

The PDI demonstrates external validity as a self-report measure of symptom validity in MTBI litigants with persistent post-concussive pain complaints. A dose response relationship exists between PVT, SVT and PDI scores. Forensic examiners should include both PVT and SVT to optimize clinical decision making when evaluating MTBI litigants with complaints of pain-related disability years post-incident.

The Continuous Visual Memory Test: Update and extension on the operating characteristics as an embedded measure of cognitive performance validity

Objective: To compare and update predictive models comprised of embedded measures from the Continuous Visual Memory Test (CVMT) in their ability to predict performance validity in personal injury litigants. Methods: Ninety-two personal injury litigants underwent a comprehensive neuropsychological examination. Criterion groups were formed, i.e. PVT-Pass and PVT-Fail, based upon their performance on stand-alone measures of performance validity (PVT). Independent-samples t-tests investigated group differences on dependent variables of interest while logistic regression analyses, as well as a decision tree classification procedure, were employed to identify the best predictive model. Results: The PVT-Fail group scored significantly lower on the 20-item Larrabee Index (LI), and three CVMT variables comprising the Henry-Enders Index (HEI) including Hits, Total Score, and Delayed Recall, but significantly higher on False Alarm Errors. Although the Total score was the best single predictor of PVT status, the addition of LI improved sensitivity. The best predictive model was derived via a classification and regression tree analysis which selected LI and CVMT-FA resulting in .91 specificity, .60 sensitivity, and ROC = 0.832. Conclusion: In the current study total CVMT scores < 70, and LI scores < 18 were rare for PI litigants with MTBI and not seen in litigants with moderate and severe brain injury who passed PVTs. Three predictive CVMT models were derived. When failure on one of the models is observed then concerns about the credibility of visual memory performance should be considered with particular attention to other stand-alone and embedded measures of performance validity.

Benefits and challenges of using logistic regression to assess neuropsychological performance validity: Evidence from a simulation study

Logistic regression (LR) is recognized as a promising method for making decisions about neuropsychological performance validity by integrating information across multiple measures. However, this method has yet to be widely adopted in clinical practice, likely because several open questions remain about its utility relative to simpler methods, its effectiveness across different clinical contexts, and its feasibility at sample sizes common in the field. The current study addresses these questions by assessing classification performance of logistic regression and alternative methods across an array of simulated data sets.

We simulated scores of valid and invalid performers on 6 tests designed to mimic the psychometric and distributional properties of real performance validity measures. Out-of-sample predictive performance of LR and a commonly used alternative ("vote counting") was assessed across different base rates, validity measure properties, and sample sizes.

LR improved classification accuracy by 2%-12% across simulation conditions, primarily by improving sensitivity. False positives and negatives can be further reduced when LR predictions are interpreted as continuous, rather than binary. LR made robust predictions at sample sizes feasible for neuropsychology research (N = 307) and when as few as 2 tests with good psychometric properties were used.

Although training and test data sets of at least several hundred individuals may be required to develop and evaluate LR models for use in clinical practice, LR promises to be an efficient and powerful tool for improving judgements about performance validity. We offer several recommendations for model development and LR interpretation in a clinical setting.

Failing The b Test: The influence of cutoff scores and criterion group approaches in a sample of adults referred for psychoeducational evaluation

OBJECTIVE

Previous research has shown that both criterion grouping approaches and cutoff scores can impact PVT classification accuracy statistics. This study aimed to examine the influence of cutoff scores and criterion grouping approaches on The b Test, a measure designed to identify feigned impairment in visual scanning, processing speed, and letter identification.

METHOD

Two hundred ninety-seven adults referred for psychoeducational testing were included with the majority of individuals seeking academic accommodations (n = 215). Cutoff scores of ≥82, ≥90, and ≥120 were utilized along with two different criterion group approaches, 0 PVT failures vs. ≥2 PVT failures and 0 PVT failures versus ≥ 1 PVT failures.

RESULTS

Failure rates for The b Test in the overall sample ranged from 12.5% to 16.2%. Subgroup analyses in those referred specifically for ADHD revealed failure rates for The b Test ranging from 10.5% to 14.2%. ROC curves within the full sample and ADHD subsample demonstrated significant AUCs utilizing both criterion group approaches (AUC = .66 - .78). Sensitivity and specificity varied as a function of criterion group approach and cutoff score, with 0 PVT failures vs. ≥ 2 PVT failures resulting in the greatest sensitivity when maximizing specificity at ≥.90 in the full sample and ADHD sample.

CONCLUSIONS

The results demonstrate that criterion approaches and cutoff scores impact classification accuracy of The b Test with 0 PVT vs. ≥ 2 PVT failures demonstrating the greatest classification accuracy. Special considerations should be made with regard to clinical decision making in the context of psychoeducational evaluations given that a large portion of individuals seeking accommodations fail only one PVT. Limitations of this study are also discussed.

The rate of psychiatric disorders in veterans undergoing intensive EEG monitoring is associated with symptom and performance invalidity

To determine if the number of participants with psychiatric disorders increased in association with failures on symptom validity tests (SVTs) and a performance validity test (PVT) in Veterans admitted for evaluation of possible seizures.

The 254 participants were Veterans undergoing inpatient video-EEG monitoring for the diagnosis of possible seizures. DSM-IV psychiatric disorders were diagnosed with the SCID IV. Symptom exaggeration was assessed with the MMPI-2-RF and performance validity with the TOMM.

On the MMPI-2-RF, 27.6%-32.7% showed symptom exaggeration. Participants who exaggerated on the MMPI-2-RF were more often diagnosed with psychiatric disorders. The TOMM was failed by 15.4% of the sample. Participants who failed the TOMM were more often diagnosed with an Axis I disorder but not with a personality disorder. The MMPI-2-RF was invalid in more cases than the TOMM, but 7.9% of the sample generated a valid MMPI-2-RF and an invalid TOMM.

The correlational design does not allow conclusions about cause and effect. The invalid groups may have had a higher rate of psychopathology. The number of participants with psychiatric disorders increased in association with symptom exaggeration and performance invalidity. Symptom exaggeration was more frequent than performance invalidity, but the TOMM made a unique contribution to identification of invalidity. The routine clinical use of SVTs and PVTs is supported. The results also suggest the need for caution in diagnosing psychiatric disorders when there is symptom exaggeration or performance invalidity, because diagnostic validity is dependent on the accuracy of symptom reporting.

Base rates of invalidity when patients undergoing routine clinical evaluations have social security disability as an external incentive

Objective: Social Security Disability is a common external incentive in neuropsychological evaluations. This study determined base rates of invalidity when patients referred for routine clinical evaluations have Social Security Disability as an external incentive. Method: Patients (n = 242) were grouped as validly or invalidly performing based on the use of multiple performance validity tests. Frequency analyses were then conducted. Results: As a whole, 46.0% of clinically referred patients with Social Security Disability as an external incentive produced invalid data. When divided by disability pursuit status, 58.6% of individuals already receiving Social Security Disability, 44.6% of individuals actively seeking Social Security Disability, and 39.3% of individuals considering seeking Social Security Disability produced invalid data. By comparison, only 8.5% of clinically referred patients without known external incentives produced invalid data. Conclusions: Beyond establishing base rates, these data indicate that the external incentive, not necessarily the evaluation setting, increases the rate of invalidity, as obtained base rates mirror those observed in independent medical examinations. In addition, this study highlights that even patients who report that they are considering but have not committed themselves to pursuing an external incentive frequently invalidate testing.

Response time measures on the Word Memory Test do not add incremental validity to accuracy scores in predicting noncredible neurocognitive dysfunction in mild traumatic brain injury litigants

The objective of the current study was to investigate whether response time measures on the Word Memory Test (WMT) increase predictive validity on determining noncredible neurocognitive dysfunction in a large sample of mild traumatic brain injury (MTBI) litigants. Participants included 203 adults who underwent a comprehensive neuropsychological examination. Criterion groups were formed based upon their performance on stand-alone measures of cognitive performance validity (PVT). Participants failing PVTs exhibited significantly slower response times and lower accuracy on the WMT compared to participants who passed PVTs. Response time measures did not add significant incremental validity beyond that afforded by WMT accuracy measures alone. The best predictor of PVT status was the WMT Consistency Score (CNS) which was associated with an extremely large effect size (d = 16.44), followed by Immediate Recognition (IR: d = 10.68) and Delayed Recognition (DR: d = 10.10).

Preliminary findings from reevaluating the MMPI Response Bias Scale items in veterans undergoing neuropsychological evaluation

The Response Bias Scale (RBS) was developed to predict non-credible cognitive presentations among disability claimants without head injury. Developers used empirical keying, which is independent of apparent content, to select items from the Minnesota Multiphasic Personality Inventory-2 (MMPI-2) item pool that distinguished between individuals passing or failing performance validity tests (PVTs). No study has examined which of these items would have psychometric value when used in clinical neuropsychological evaluations. This study reexamined items comprising RBS with reference to manifest item content, internal consistency, PVTs, and a symptom validity test (SVT) in a sample of 173 predominately White male veterans (MAGE = 50.70, MEDU = 13.73) in a VA outpatient neuropsychology clinic. Participants completed the MMPI-2 Restructured Form (MMPI-2-RF), PVTs, and an SVT. The 28-item RBS appears to contain three types of items: those that manifestly address cognitive functioning, those that are supported but do not appear to address cognitive functioning, and nine items that were unrelated to cognition and not statistically supported. The 19 empirically supported items, or RBS-19, predicted PVT and SVT failures marginally better than the RBS. Both the RBS and RBS-19 had stronger relationships with SVTs relative to PVTs. Although the removal of the nine problematic items improved the diagnostic accuracy of the scale, it still did not reach the level that is generally considered to be clinically optimal. The RBS-19 offers a measure with improved internal consistency and predictive validity compared to the RBS and warrants additional research.

Considerations for Reliable Digit Span as a performance validity test for long-term survivors of childhood brain tumors

The Reliable Digit Span (RDS) is a performance validity test (PVT) used widely within non-clinical samples, but its utility is in question in clinical groups with cognitive impairment. To investigate, RDS scores were calculated and correlated with the Neurological Predictor Scale, an informant-reported Activities of Daily Living score, and a proxy measure of intelligence (Vocabulary) for 83 adult survivors of childhood brain tumors and 105 healthy controls. Analyses were covaried for age at examination. Participants were divided into passing and failing groups at each RDS cutoff, and ANCOVAs for each of the three variables of interest covaried for age at the examination were run. RDS was correlated with all three variables of interest in survivors but only Vocabulary in controls. At the ≤7 cutoff, passing and failing survivors demonstrated significant differences across all variables of interest, while passing and failing controls differed only on Vocabulary. Differences were also found between passing and failing survivors at lower cutoffs. RDS is related to and likely impacted by various neurological and cognitive challenges faced by brain tumor survivors. Using the standard RDS cutoff of ≤7 may result in inaccurate interpretation of valid performance in this population; therefore, the use of other PVTs is recommended.

Psychometric implications of failure on one performance validity test: a cross-validation study to inform criterion group definition

Introduction: Research to date has supported the use of multiple performance validity tests (PVTs) for determining validity status in clinical settings. However, the implications of including versus excluding patients failing one PVT remains a source of debate, and methodological guidelines for PVT research are lacking. This study evaluated three validity classification approaches (i.e. 0 vs. ≥2, 0-1 vs. ≥2, and 0 vs. ≥1 PVT failures) using three reference standards (i.e. criterion PVT groupings) to recommend approaches best suited to establishing validity groups in PVT research methodology.Method: A mixed clinical sample of 157 patients was administered freestanding (Medical Symptom Validity Test, Dot Counting Test, Test of Memory Malingering, Word Choice Test), and embedded PVTs (Reliable Digit Span, RAVLT Effort Score, Stroop Word Reading, BVMT-R Recognition Discrimination) during outpatient neuropsychological evaluation. Three reference standards (i.e. two freestanding and three embedded PVTs from the above list) were created. Rey 15-Item Test and RAVLT Forced Choice were used solely as outcome measures in addition to two freestanding PVTs not employed in the reference standard. Receiver operating characteristic curve analyses evaluated classification accuracy using the three validity classification approaches for each reference standard.Results: When patients failing only one PVT were excluded or classified as valid, classification accuracy ranged from acceptable to excellent. However, classification accuracy was poor to acceptable when patients failing one PVT were classified as invalid. Sensitivity/specificity across two of the validity classification approaches (0 vs. ≥2; 0-1 vs. ≥2) remained reasonably stable.Conclusions: These results reflect that both inclusion and exclusion of patients failing one PVT are acceptable approaches to PVT research methodology and the choice of method likely depends on the study rationale. However, including such patients in the invalid group yields unacceptably poor classification accuracy across a number of psychometrically robust outcome measures and therefore is not recommended.

Normal cognitive test scores cannot be interpreted as accurate measures of ability in the context of failed performance validity testing: A symptom- and detection-coached simulation study

Introduction: While use of performance validity tests (PVTs) has become a standard of practice in neuropsychology, there are differing opinions regarding whether to interpret cognitive test data when standard scores fall within normal limits despite PVTs being failed. This study is the first to empirically determine whether normal cognitive test scores underrepresent functioning when PVTs are failed.Method: Participants, randomly assigned to either a simulated malingering group (n = 50) instructed to mildly suppress test performances or a best-effort/control group (n = 50), completed neuropsychological tests which included the North American Adult Reading Test (NAART), California Verbal Learning Test - 2^nd Edition (CVLT-II), and Test of Memory Malingering (TOMM).Results: Groups were not significantly different in age, sex, education, or NAART predicted intellectual ability, but simulators performed significantly worse than controls on the TOMM, CVLT-II Forced Choice Recognition, and CVLT-II Short Delay Free Recall. The groups did not significantly differ on other examined CVLT-II measures. Of simulators who failed validity testing, 36% scored no worse than average and 73% scored no worse than low average on any of the examined CVLT-II indices.Conclusions: Of simulated malingerers who failed validity testing, nearly three-fourths were able to produce cognitive test scores that were within normal limits, which indicates that normal cognitive performances cannot be interpreted as accurately reflecting an individual's capabilities when obtained in the presence of validity test failure. At the same time, only 2 of 50 simulators were successful in passing validity testing while scoring within an impaired range on cognitive testing. This latter finding indicates that successfully feigning cognitive deficits is difficult when PVTs are utilized within the examination.

Detection of Feigned ADHD Using the MOXO-d-CPT

Objective: The objective of this study was to assess the MOXO-d-CPT utility in detecting feigned ADHD and establish cutoffs with adequate specificity and sensitivity. Method: The study had two phases. First, using a prospective design, healthy adults who simulated ADHD were compared with healthy controls and ADHD patients who performed the tasks to the best of their ability (n = 47 per group). Participants performed the MOXO-d-CPT and an established performance validity test (PVT). Second, the MOXO-d-CPT classification accuracy, employed in Phase 1, was retrospectively compared with archival data of 47 ADHD patients and age-matched healthy controls. Results: Simulators performed significantly worse on all MOXO-d-CPT indices than healthy controls and ADHD patients. Three MOXO-d-CPT indices (attention, hyperactivity, impulsivity) and a scale combining these indices showed adequate discriminative capacity. Conclusion: The MOXO-d-CPT showed promise for the detection of feigned ADHD and, pending replication, can be employed for this aim in clinical practice and ADHD research.

Patterns of Performance and Symptom Validity Test Findings After Mild Traumatic Brain Injury

OBJECTIVE

The purpose of this study was to evaluate the presence of demographic, injury and neuropsychological correlates of distinct patterns of performance validity test and symptom validity test results in persons with mild traumatic brain injury (mTBI).

METHOD

One hundred and seventy-eight persons with mTBI completed the Test of Memory Malingering (TOMM; performance validity) and the Minnesota Multiphasic Personality Inventory-2-Restructured Form (MMPI-2-RF; symptom validity) within 1-12 months postinjury. Four groups were compared: (a) pass both TOMM and MMPI-2-RF validity criteria, (b) pass TOMM and fail MMPI-2-RF, (c) fail TOMM and pass MMPI-2-RF, and (d) fail both TOMM and MMPI-2-RF.

RESULTS

Compared to Group a, participants in combined Groups b-d were more than twice as likely to be engaged in financial compensation-seeking and about four times less likely to have neuroimaging evidence of an intracranial lesion. The average performance of Group d on an independent test of verbal learning was more than 1.5 standard deviations below that of Group a. Participants in Group b were more likely to have intracranial lesions on neuroimaging than participants in Group c.

CONCLUSION

Performance and symptom validity tests provide complementary and non-redundant information in persons with mTBI. Whereas financial compensation-seeking is associated with increased risk of failure of either PVT or SVT, or both, the presence of intracranial findings on neuroimaging is associated with decreased risk of such.

American Academy of Clinical Neuropsychology (AACN) 2021 consensus statement on validity assessment: Update of the 2009 AACN consensus conference statement on neuropsychological assessment of effort, response bias, and malingering

Objective: Citation and download data pertaining to the 2009 AACN consensus statement on validity assessment indicated that the topic maintained high interest in subsequent years, during which key terminology evolved and relevant empirical research proliferated. With a general goal of providing current guidance to the clinical neuropsychology community regarding this important topic, the specific update goals were to: identify current key definitions of terms relevant to validity assessment; learn what experts believe should be reaffirmed from the original consensus paper, as well as new consensus points; and incorporate the latest recommendations regarding the use of validity testing, as well as current application of the term 'malingering.' Methods: In the spring of 2019, four of the original 2009 work group chairs and additional experts for each work group were impaneled. A total of 20 individuals shared ideas and writing drafts until reaching consensus on January 21, 2021. Results: Consensus was reached regarding affirmation of prior salient points that continue to garner clinical and scientific support, as well as creation of new points. The resulting consensus statement addresses definitions and differential diagnosis, performance and symptom validity assessment, and research design and statistical issues. Conclusions/Importance: In order to provide bases for diagnoses and interpretations, the current consensus is that all clinical and forensic evaluations must proactively address the degree to which results of neuropsychological and psychological testing are valid. There is a strong and continually-growing evidence-based literature on which practitioners can confidently base their judgments regarding the selection and interpretation of validity measures.

Victoria Symptom Validity Test: A Systematic Review and Cross-Validation Study

The Victoria Symptom Validity Test (VSVT) is a performance validity test (PVT) with over two decades of empirical backing, although methodological limitations within the extant literature restrict its clinical and research generalizability. Chief among these constraints includes limited consensus on the most accurate index within the VSVT and the most appropriate cut-scores within each VSVT validity index. The current systematic review synthesizes existing VSVT validation studies and provides additional cross-validation in an independent sample using a known-groups design. We completed a systematic search of the literature, identifying 17 peer-reviewed studies for synthesis (7 simulation designs, 7 differential prevalence designs, and 3 known-groups designs). The independent cross-validation sample consisted of 200 mixed clinical neuropsychiatric patients referred for outpatient neuropsychological evaluation. Across all indices, Total item accuracy produced the strongest psychometric properties at an optimal cut-score of ≤ 40 (62% sensitivity/88% specificity). However, ROC curve analyses for all VSVT indices yielded statistically significant areas under the curve (AUCs; .73-81), suggestive of moderate classification accuracy. Cut-scores derived using the independent cross-validation sample converged with some previous findings supporting cut-scores of ≤ 22 for Easy item accuracy and ≤ 40 for Total item accuracy, although divergent findings were noted for Difficult item accuracy. Overall, VSVT validity indicators have adequate diagnostic accuracy across populations, with the current study providing additional support for its use as a psychometrically sound PVT in clinical settings. However, caution is recommended among patients with certain verified clinical conditions (e.g., dementia) and those with pronounced working memory deficits due to concerns for increased risk of false positives.

Professional Practices, Beliefs, and Incomes of Postdoctoral Trainees: The AACN, NAN, SCN 2020 Practice and ‘Salary Survey’

Objective

Within a portion of the 2020 professional practice and “salary survey,” to update key information regarding neuropsychology postdoctoral trainees.

Methods

Postdoctoral trainees were contacted via a variety of membership listings, including the listserv used by the program directors of the Association of Postdoctoral Programs in Clinical Neuropsychology (APPCN). Invitations sent in multiple waves to members of numerous neuropsychological organizations via e-messages and physical postcards included the request that postdoctoral trainees participate. The survey website was opened on January 17, 2020 and closed on April 2, 2020, during which time a total of 178 postdoctoral trainees in the USA and 3 in Canada participated.

Results

Response rate was estimated to be 56.4%, which adequately represents the target sample. The modal postdoctoral trainee is a woman whose internship was American Psychological Association (APA)-accredited and whose postdoctoral training is in an APPCN program that adheres to Houston Conference training guidelines. Extensive clinical experiences in neuropsychology in the form of externship practica and during internship were reported by the majority of trainees prior to postdoctoral training. There are few differences between APPCN and non-APPCN trainees and reported training experiences. Job satisfaction is high. Salaries appear to have increased substantially in recent years. There is universal interest in pursuing board certification. Support for the empirical foundations justifying assessment of response validity is high.

Conclusions

Surveys of postdoctoral trainees continue to provide valuable perspectives regarding training background, clinical experiences, practice beliefs, and incomes of individuals who will soon launch their careers in clinical neuropsychology.

Deception is different: Negative validity test findings do not provide "evidence" for "good effort"

OBJECTIVE

The purpose of this paper is to determine whether negative validity test findings should be used in the Bayesian aggregate along with positive test findings for the determination of malingering as the condition of interest (COI). Method: Evidence-based diagnostic methods for conditions in neuropsychology and medicine were reviewed for comparison with their use in cases of malingering. Logical and Bayesian analyses of these cases were applied. A case study showed that negative validity test findings did not indicate "good effort". Results: Deception about illness is fundamentally different from other constructs/diseases in evidence-based medicine and neuropsychology. This is because deception involves a deliberate process that may involve coaching, claimant research, and/or focusing the deception on one aspect (e.g., slowness) as opposed to other neurocognitive problems (e.g., memory). Comparatively, other conditions in medicine and neuropsychology are unlikely to be manipulated by the patient. Conclusions: The assertion by Frederick (2015) and Black, Necrason, and Omasta (2016) that both positive and negative validity test findings must be used together in the aggregate does not stand up to this comparative scrutiny. The fundamental assumption by these authors that a negative test finding concerning malingering represents "good effort" is flawed; it simply represents lack of evidence of malingering, which cannot be construed as evidence of lack of malingering. We recommend that in forensic determination of malingering, negative validity test findings should not be used in a Bayesian aggregation. This conclusion is consistent with current practices in the field.

Gender Differences in Veterans Referred for Neuropsychological Evaluation in an Outpatient Neuropsychology Consultation Service

OBJECTIVE

The number of women veterans seeking Veterans Health Administration services has substantially increased over the past decade. Neuropsychology remains an understudied area in the examination of gender differences. The present study sought to delineate similarities and differences in men and women veterans presenting for neuropsychological evaluation in terms of demographics, referral, medical conditions, effort, and outcome diagnosis.

METHOD

A database collected from an outpatient VA neuropsychology clinic from 2013 to 2019 was analyzed (n = 232 women, 2642 men). Additional analyses examined younger (n = 836 men, 155 women) and older (n = 1805 men, 77 women) age cohorts.

RESULTS

Women veterans were younger and more educated than men, whereas men had higher prevalence of vascular risk factors. Both groups were most often referred from mental health clinics and memory was the most common referral question. Although men performed worse on performance validity measures, clinicians rated women as evidencing poorer effort on a cumulative rating based on formal and embedded performance validity measures, behavioral observations, and inconsistent test patterns. Older women reported more depressive symptoms than older men and were more commonly diagnosed with depression.

CONCLUSIONS

This exploratory study fills a gap in the understanding of gender differences in veterans presenting for neuropsychological evaluations. Findings emphasize consideration for the intersection of gender with demographics, medical factors, effort, and psychological symptoms by VA neuropsychologists. A better understanding of relationships between gender and these factors may inform neuropsychologists' test selection, interpretation of behavioral observations, and diagnostic considerations to best treat women veterans.

A Systematic Review and Meta-Analysis of the Utility of the Test of Memory Malingering in Pediatric Examinees

Objective

This is the first systematic review and meta-analysis of the Test of Memory Malingering (TOMM) in pediatric examinees. It adheres to Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.

Method

A systematic literature search was conducted using PsycINFO and PubMed, reviewing articles from January 1997 to July 2019. Books providing data on pediatric validity testing were also reviewed for references to relevant articles. Eligibility criteria included publication in a peer-reviewed journal, utilizing a pediatric sample, providing sufficient data to calculate specificity and/or sensitivity, and providing a means for evaluating validity status external to the TOMM. After selection criteria were applied, 9 articles remained for meta-analysis. Samples included clinical patients and healthy children recruited for research purposes; ages ranged from 5 to 18. Fixed and random effects models were used to calculate classification accuracy statistics.

Results

Traditional adult-derived cutoffs for Trial 2 and Retention were highly specific (0.96–0.99) in pediatric examinees for both clinical and research samples. Sensitivity was relatively strong (0.68–0.70), although only two studies reported sensitivity rates. A supplemental review of the literature corroborated these findings, revealing that traditional adult-based TOMM cutoffs are supported in most pediatric settings. However, limited research exists on the impact of very young age, extremely low cognitive functioning, and varying clinical diagnoses.

Conclusions

The TOMM, at traditional adult cutoffs, has strong specificity as a performance validity test in pediatric neuropsychological evaluations. This meta-analysis found that specificity values in children are comparable to those of adults. Areas for further research are discussed.

Delayed matching to sample task 48: assessment of malingering with simulating design

The results of neuropsychological tests may be distorted by patients who exaggerate cognitive deficits. Eighty-three patients with cognitive deficit [Amnestic Mild Cognitive Impairment (aMCI), n = 53; Alzheimer's disease (AD) dementia, n = 30], 44 healthy older adults (HA), and 30 simulators of AD (s-AD) underwent comprehensive neuropsychological assessment. Receiver Operating Characteristic (ROC) analysis revealed high specificity but low sensitivity of the Delayed Matching to Sample Task (DMS48) in differentiating s-AD from AD dementia (87 and 53%, respectively) and from aMCI (96 and 57%). The sensitivity was considerably increased by using the DMS48/Rey Auditory Verbal Learning Test (RAVLT) ratio (specificity and sensitivity 93% and 93% for AD dementia and 96% and 80% for aMCI). The DMS48 differentiates s-AD from both aMCI and AD dementia with high specificity but low sensitivity. Its predictive value greatly increased when evaluated together with the RAVLT.

Multidimensional Malingering Criteria for Neuropsychological Assessment: A 20-Year Update of the Malingered Neuropsychological Dysfunction Criteria

OBJECTIVES

Empirically informed neuropsychological opinion is critical for determining whether cognitive deficits and symptoms are legitimate, particularly in settings where there are significant external incentives for successful malingering. The Slick, Sherman, and Iversion (1999) criteria for malingered neurocognitive dysfunction (MND) are considered a major milestone in the field's operationalization of neurocognitive malingering and have strongly influenced the development of malingering detection methods, including serving as the criterion of malingering in the validation of several performance validity tests (PVTs) and symptom validity tests (SVTs) (Slick, D.J., Sherman, E.M.S., & Iverson, G. L. (1999). Diagnostic criteria for malingered neurocognitive dysfunction: Proposed standards for clinical practice and research. The Clinical Neuropsychologist, 13(4), 545-561). However, the MND criteria are long overdue for revision to address advances in malingering research and to address limitations identified by experts in the field.

METHOD

The MND criteria were critically reviewed, updated with reference to research on malingering, and expanded to address other forms of malingering pertinent to neuropsychological evaluation such as exaggeration of self-reported somatic and psychiatric symptoms.

RESULTS

The new proposed criteria simplify diagnostic categories, expand and clarify external incentives, more clearly define the role of compelling inconsistencies, address issues concerning PVTs and SVTs (i.e., number administered, false positives, and redundancy), better define the role of SVTs and of marked discrepancies indicative of malingering, and most importantly, clearly define exclusionary criteria based on the last two decades of research on malingering in neuropsychology. Lastly, the new criteria provide specifiers to better describe clinical presentations for use in neuropsychological assessment.

CONCLUSIONS

The proposed multidimensional malingering criteria that define cognitive, somatic, and psychiatric malingering for use in neuropsychological assessment are presented.

The Modified Somatic Perception Questionnaire: Operating characteristics in non-pain forensic samples

The objective of the current archival study was to investigate the ability of the Modified Somatic Perception Questionnaire (MSPQ) to discriminate between noncredible and credible neurocognitive dysfunction in a large mixed non-pain forensic sample comprised of personal injury litigants and disability claimants. Participants included 149 adults who underwent comprehensive neuropsychological examination. Criterion groups were formed, i.e., Credible Group (CG), or Noncredible Group (NCG) based upon their performance on stand-alone performance validity tests (PVT) including the Word Memory Test (WMT), and/or Test of Memory Malingering (TOMM), and Victoria Symptom Validity Test (VSVT). After excluding examinees with evidence of somatization participants in the NCG scored significantly higher on the MSPQ compared to participants in the CG. Scores on the MSPQ were not only related to cognitive performance validity but also amplified by the co-existence of somatization. There were no significant effects of gender, age, or race on MSPQ scores. We conclude that the MSPQ demonstrates external validity as a self-report measure of symptom validity that is sensitive to somatic and autonomic complaints in non-pain forensic samples. Future research is needed to develop MSPQ cutscores with external validity pertinent to other non-pain populations.

Base Rates of Invalid Test Performance Across Clinical Non-forensic Contexts and Settings

Objective

Base rates of invalidity in forensic neuropsychological contexts are well explored and believed to approximate 40%, whereas base rates of invalidity across clinical non-forensic contexts are relatively less known.

Methods

Adult-focused neuropsychologists (n = 178) were surveyed regarding base rates of invalidity across various clinical non-forensic contexts and practice settings. Median values were calculated and compared across contexts and settings.

Results

The median estimated base rate of invalidity across clinical non-forensic evaluations was 15%. When examining specific clinical contexts and settings, base rate estimates varied from 5% to 50%. Patients with medically unexplained symptoms (50%), external incentives (25%–40%), and oppositional attitudes toward testing (37.5%) were reported to have the highest base rates of invalidity. Patients with psychiatric illness, patients evaluated for attention deficit hyperactivity disorder, and patients with a history of mild traumatic brain injury were also reported to invalidate testing at relatively high base rates (approximately 20%). Conversely, patients presenting for dementia evaluation and patients with none of the previously mentioned histories and for whom invalid testing was unanticipated were estimated to produce invalid testing in only 5% of cases. Regarding practice setting, Veterans Affairs providers reported base rates of invalidity to be nearly twice that of any other clinical settings.

Conclusions

Non-forensic clinical patients presenting with medically unexplained symptoms, external incentives, or oppositional attitudes are reported to invalidate testing at base rates similar to that of forensic examinees. The impact of context-specific base rates on the clinical evaluation of invalidity is discussed.

Feedback with patients who produce invalid testing: Professional values and reported practices

OBJECTIVE

Providing feedback to clinical patients who produce invalid neuropsychological test data presents with several potential challenges. Given the limited guidance available on the topic, neuropsychologists most likely utilize approaches that idiosyncratically incorporate professional values related to both assessment and patient care to address overarching feedback goals. The current article discusses professional values believed to inform feedback decisions and presents survey results regarding how neuropsychologists would approach feedback across various clinical scenarios where testing is invalid.

METHOD

Participants were 209 adult-focused clinical neuropsychologists recruited via professional listservs, the majority of whom reported being board certified. Respondents were provided three case vignettes of clinical patients who produced invalid test data and were asked how they would provide feedback to each patient. Open-ended responses were coded to determine the relative frequency of feedback approaches, explicit statements, and implied goals.

RESULTS

Nearly all respondents (98%) indicated that feedback would include some description of the invalid findings, and most respondents indicated that they would provide explanations for the reasons of invalidity (67%) and statements regarding the impact of invalidity on test interpretation (75%). There was little agreement across respondents, however, regarding specific feedback approach. Feedback goals included to Inform, Investigate, Treat, Educate, and Correct, with the presence of these implied goals also varying across respondents.

CONCLUSIONS

The findings indicate that there is minimal consensus regarding feedback approaches provided to patients who produce invalid test data and underscore a need for further development and validation of specific feedback methods. The results are discussed within the context of the potentially competing professional values of evidence-based assessment, patient-doctor collaboration, and aspirational principles of beneficence and fidelity.

Cross-validation of the Invalid Forgetting Frequency Index (IFFI) from the Test of Memory Malingering

Objective

To increase sensitivity of the Test of Memory Malingering (TOMM), adjustments have been proposed, including adding consistency indices. The Invalid Forgetting Frequency Index (IFFI) is the most recently developed consistency index. While strong classification accuracy rates were originally reported, it currently lacks cross-validation.

Method

A sample of 184 outpatients was utilized. Valid performers passed all criterion performance validity tests (PVTs) and invalid performers failed two or more PVTs. Classification accuracy statistics were calculated.

Results

AUC for the IFFI was 0.80, demonstrating adequate discrimination between valid and invalid groups. A score of 3 or more inconsistent responses resulted in sensitivity and specificity rates of 63% and 92%, respectively.

Conclusions

This is the first article to cross-validate the IFFI. In both the original IFFI study and the current study, the same cut-off was found to maintain at least 90% specificity while producing higher sensitivity rates than those achieved by traditional TOMM indices.

Effort Testing in Dementia Assessment: A Systematic Review

Objective

Interpretation of neuropsychological test data is only valid when appropriate effort has been exerted. Research, however, suggests that neuropsychologists do not always formally test for effort and that this may especially be the case in the context of dementia assessment. This review systematically examined the literature that has investigated the use of both purpose-built and embedded effort-sensitive indices in dementia, mild cognitive impairment (MCI), and healthy control samples. The aim was to determine which tests of effort are most sensitive to suboptimal effort and least sensitive to the type of cognitive impairment seen in dementia.

Methods

A systematic search of databases was conducted to October 2017. There was no start date.

Results

Twenty-five studies were included for review. The studies were divided into two categories according to methodology. One category of studies (n = 5) was reviewed using a tailored methodological quality rating checklist whilst the remaining studies (n = 20) were reviewed using the Crowe Critical Appraisal Tool (CCAT).

Conclusions

The results of this review suggest that PVTs which take a hierarchical approach to effort testing such as the WMT, MSVT, and NV-MSVT are preferable for use with older adults who are under investigation for possible dementia. These tests go above and beyond the traditional pass/fail approach of more traditional tests of effort because they allow the examiner to analyze the examinee's profile of scores. The methodological limitations and challenges involved in this field of research are discussed.

The Minnesota Multiphasic Personality Inventory-2-RF in Treatment-Seeking Veterans with History of Mild Traumatic Brain Injury

OBJECTIVE

This study examined the Minnesota Multiphasic Personality Inventory-Second Edition-Restructured Form (MMPI-2-RF) to better understand symptom presentation in a sample of treatment-seeking Operation Enduring Freedom/Operation Iraqi Freedom (OEF/OIF) Veterans with self-reported history of mild traumatic brain injury (mTBI).

METHOD

Participants underwent a comprehensive clinical neuropsychological battery including performance and symptom validity measures and self-report measures of depressive, posttraumatic, and post-concussive symptomatology. Those with possible symptom exaggeration (SE+) on the MMPI-2-RF were compared with those without (SE-) with regard to injury, psychiatric, validity, and cognitive variables.

RESULTS

Between 50% and 87% of participants demonstrated possible symptom exaggeration on one or more MMPI-2-RF validity scales, and a large majority were elevated on content scales related to cognitive, somatic, and emotional complaints. The SE+ group reported higher depressive, posttraumatic, and post-concussive symptomatology, had higher scores on symptom validity measures, and performed more poorly on neuropsychological measures compared with the SE- group. There were no group differences with regard to injury variables or performance validity measures. Participants were more likely to exhibit possible symptom exaggeration on cognitive/somatic compared with traditional psychopathological validity scales.

CONCLUSIONS

A sizable portion of treatment-seeking OEF/OIF Veterans demonstrated possible symptom exaggeration on MMPI-2-RF validity scales, which was associated with elevated scores on self-report measures and poorer cognitive performance, but not higher rates of performance validity failure, suggesting symptom and performance validity are distinct concepts. These findings have implications for the interpretation of clinical data in the context of possible symptom exaggeration and treatment in Veterans with persistent post-concussive symptoms.

Cross-validation of three Advanced Clinical Solutions performance validity tests: Examining combinations of measures to maximize classification of invalid performance

Use of multiple performance validity tests (PVTs) may best identify invalid performance, though few studies have examined the utility and accuracy of combining PVTs. This study examined the following PVTs in the Advanced Clinical Solutions (ACS) package to determine their utility alone and in concert: Word Choice Test (WCT), Reliable Digit Span (RDS), and Logical Memory Recognition (LMR). Ninety-three veterans participated in clinical neuropsychological evaluations to determine presence of cognitive impairment; 25% of the performances were deemed invalid via criterion PVTs. Classification accuracy of the ACS measures was assessed via receiver operating characteristic curves, while logistic regressions determined utility of combining these PVTs. The WCT demonstrated superior classification accuracy compared to the two embedded measures of the ACS, even in veterans with cognitive impairment. The two embedded measures (even when used in concert) exhibited inadequate classification accuracy. A combined model with all three ACS PVTs similarly demonstrated little benefit of the embedded indicators over the WCT alone. Results suggest the ACS WCT has utility for detecting invalid performance in a clinical sample with likely cognitive impairment, though the embedded ACS measures (RDS and LMR) may have limited incremental utility, particularly in individuals with cognitive impairment.

"Must try harder." Is effort and performance validity testing a necessary part of pediatric neuropsychological assessment?

Neuropsychological assessments results have significant implications for pediatric populations, based upon the assumption that the young person has adopted an effortful approach and has engaged in assessment. There is a commonly-accepted risk to assuming the validity of neuropsychological assessment results with adults, and, therefore, performance validity testing (PVT) has become a major topic of research and investigation and has become an accepted part of routine assessment. The same approach has not been adopted in assessment with children and a paucity of studies has focused on PVT in children. We review studies that demonstrate that children are equal to adults in their ability to use deception and that clinicians cannot detect false-effort without use of validity tests. We explore how frequently such tests are used and how well they work in assessment with children, and the limits, complexities, and constraints of adapting adult tests. We advocate that adequate performance validity testing is essential in order to maximize confidence in the results and we hypothesize that assessment with pediatric populations should take into account a range of influences, such as neuro-developmental factors associated with age of the child and suitability of proposed measures according to the evidence-base.

Wait, There’s a Baby in this Bath Water! Update on Quantitative and Qualitative Cut-Offs for Rey 15-Item Recall and Recognition

Objective

Evaluate the effectiveness of Rey 15-item plus recognition data in a large neuropsychological sample.

Method

Rey 15-item plus recognition scores were compared in credible (n = 138) and noncredible (n = 353) neuropsychology referrals.

Results

Noncredible patients scored significantly worse than credible patients on all Rey 15-item plus recognition scores. When cut-offs were selected to maintain at least 89.9% specificity, cut-offs could be made more stringent, with the highest sensitivity found for recognition correct (cut-off ≤11; 62.6% sensitivity) and the combination score (recall + recognition – false positives; cut-off ≤22; 60.6% sensitivity), followed by recall correct (cut-off ≤11; 49.3% sensitivity), and recognition false positive errors (≥3; 17.9% sensitivity). A cut-off of ≥4 applied to a summed qualitative error score for the recall trial resulted in 19.4% sensitivity. Approximately 10% of credible subjects failed either recall correct or recognition correct, whereas two-thirds of noncredible patients (67.7%) showed this pattern. Thirteen percent of credible patients failed either recall correct, recognition correct, or the recall qualitative error score, whereas nearly 70% of noncredible patients failed at least one of the three. Some individual qualitative recognition errors had low false positive rates (&lt;2%) indicating that their presence was virtually pathognomonic for noncredible performance. Older age (&gt;50) and IQ &lt; 80 were associated with increased false positive rates in credible patients.

Conclusions

Data on a larger sample than that available in the 2002 validation study show that Rey 15-item plus recognition cut-offs can be made more stringent, and thereby detect up to 70% of noncredible test takers, but the test should be used cautiously in older individuals and in individuals with lowered IQ.

A meta-analysis of the accuracy of embedded performance validity indicators from the repeatable battery for the assessment of neuropsychological status

Objective: Meta-analyze the embedded performance validity tests of the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS), emphasizing two topics: diagnostic accuracy and relationships with demographics. Method: Literature was identified through a review of PsycINFO, PubMed, and Google Scholar. Bivariate linear mixed models were used for diagnostic accuracy analyses. Random-effects analysis of correlations with demographic variables was also used for the Effort Index. Results: Diagnostic accuracy analyses utilized 12 studies of the Effort Index (EI; N = 1469) and 6 studies of the Effort Scale (ES; N = 854). The EI was found to have a sensitivity of .44 (95% CI: .28 - .62), specificity of .87 (95% CI: .78 - .93), and diagnostic odds ratio (DOR) of 5.41 (95% CI: 3.74 - 7.58). The ES produced a sensitivity of .67 (95% CI: .50 - .81), specificity of .72 (95% CI: .53 - .85), and DOR of 5.97 (95% CI: 1.70 - 15.30). Meta-analysis of correlations utilized 14 samples of the EI (N = 3781), but insufficient data were available for the ES. Correlations between the EI and age (r = .10, 95% CI: .05 - .16), education (r = -.11, 95% CI: -.18 to -.04), and the RBANS Total Scale (r = -.45, 95% CI: -.62 to -.24) were significant. These results appeared to be robust to publication bias. Conclusion: Results support use of the EI over the ES; however, the latter is comparatively less studied and did not have similar variability in reported cut-off scores. ES accuracy statistics were significantly predicted by sample variables, which may be related to its relatively fewer studies. Both measures should be used cautiously in the context of genuine cognitive impairment.