Specificity rates for non-clinical, bilingual, Mexican Americans on three popular performance validity measures

Examining the cross-cultural validity of the test of memory malingering and the Rey 15-item test

OBJECTIVE

This study was designed to investigate the cross-cultural validity of two freestanding performance validity tests (PVTs), the Test of Memory Malingering - Trial 1 (TOMM-1) and the Rey Fifteen Item Test (Rey-15) in Romanian-speaking patients.

METHODS

The TOMM-1 and Rey-15 free recall (FR) and the combination score incorporating the recognition trial (COMB) were administered to a mixed clinical sample of 61 adults referred for cognitive evaluation, 24 of whom had external incentives to appear impaired. Average scores on PVTs were compared between the two groups. Classification accuracies were computed using one PVT against another.

RESULTS

Patients with identifiable external incentives to appear impaired produced significantly lower scores and more errors on validity indicators. The largest effect sizes emerged on TOMM-1 (Cohen's d = 1.00-1.19). TOMM-1 was a significant predictor of the Rey-15 COMB ≤20 (AUC = .80; .38 sensitivity; .89 specificity at a cutoff of ≤39). Similarly, both Rey-15 indicators were significant predictors of TOMM-1 at ≤39 as the criterion (AUCs = .73-.76; .33 sensitivity; .89-.90 specificity).

CONCLUSION

Results offer a proof of concept for the cross-cultural validity of the TOMM-1 and Rey-15 in a Romanian clinical sample.

Validity assessment in Eastern Europe: cross-validation of the Dot Counting Test and MODEMM against the TOMM-1 and Rey-15 in a Romanian mixed clinical sample

OBJECTIVE

This study investigated performance validity in the understudied Romanian clinical population by exploring classification accuracies of the Dot Counting Test (DCT) and the first Romanian performance validity test (PVT) (Memory of Objects and Digits and Evaluation of Memory Malingering/MODEMM) in a heterogeneous clinical sample.

METHODS

We evaluated 54 outpatients (26 females; MAge = 62.02; SDAge = 12.3; MEducation = 2.41, SDEducation = 2.82) with the Test of Memory Malingering 1 (TOMM-1), Rey Fifteen Items Test (Rey-15) (free recall and recognition trials), DCT, MODEMM, and MMSE/MoCA as part of their neuropsychological assessment. Accuracy parameters and base failure rates were computed for the DCT and MODEMM indicators against the TOMM-1 and Rey-15. Two patient groups were constructed according to psychometrically defined credible/noncredible performance (i.e., pass/fail both TOMM-1 and Rey-15).

RESULTS

Similar to other cultures, a cutoff of ≥18 on the DCT E score produced the best combination between sensitivity (0.50-0.57) and specificity (≥0.90). MODEMM indicators based on recognition accuracy, inconsistencies, and inclusion false positives generated 0.75-0.86 sensitivities at ≥0.90 specificities. Multivariable models of MODEMM indicators reached perfect sensitivities at ≥0.90 specificities against two PVTs. Patients who failed the TOMM-1 and Rey-15 were significantly more likely to fail the DCT and MODEMM than patients who passed both PVTs.

CONCLUSIONS

Our results offer proof of concept for the DCT's cross-cultural validity and the applicability of the MODEMM on Romanian clinical examinees, further recommending the use of heterogeneous validity indicators in clinical assessments.

Reliable Digit Span Scores Among Latin American Monolingual Spanish Speakers

OBJECTIVE

The Reliable Digit Span (RDS) is an embedded measure used to assess performance validity. The purpose of this study was to determine whether the RDS can be used with Spanish speakers from Latin America.

METHOD

The Digit Span subtest of the Wechsler Adult Intelligence Scale-IV was administered to 66 Spanish-speaking patients, 66 English-speaking patients, and 30 Spanish-speaking normal controls. The Spanish-speaking patients had a mean age of 65.82 (standard deviation [SD] = 12.44) and mean education of 6.62 years (SD = 4.91), whereas the controls had a mean age of 48.27 (SD = 10.97) and mean education of 8.33 (SD = 3.04). The English-speaking patients had a mean age of 56.44 (SD = 13.53) and mean education of 12.14 (SD = 2.37).

RESULTS

The average RDS among the Spanish-speaking patients was 5.36 (SD = 1.27) and the average RDS among the controls was 5.90 (SD = 1.06). The mean RDS of the English-speaking patients was 7.83 (SD = 1.72). The modal RDS among Spanish-speaking patients was 5. The modal RDS among Spanish-speaking controls was also 5, though the distribution was bimodal in nature with nearly as many 7 s as 5 s. The modal RDS among the English-speaking patients was 7. Among patients with ≥9 years of education and controlling for age, there remained a significant difference between both language patient groups on the RDS (F(1, 79) = 30.85, p < .0001).

CONCLUSIONS

These data suggest that the RDS may not be a valuable measure in assessing performance validity among monolingual Spanish speakers from Latin America, particularly among those with low levels of education.

Performance validity of the Dot Counting Test in a dementia clinic setting

OBJECTIVE

This study examined the utility of a performance validity test (PVT), the Dot Counting Test (DCT), in individuals undergoing neuropsychological evaluations for dementia. We investigated specificity rates of the DCT Effort Index score (E-Score) and various individual DCT scores (based on completion time/errors) to further establish appropriate cutoff scores.

METHOD

This cross-sectional study included 56 non-litigating, validly performing older adults with no/minimal, mild, or major cognitive impairment. Cutoffs associated with ≥90% specificity were established for 7 DCT scoring methods across impairment severity subgroups.

RESULTS

Performance on 5 of 7 DCT scoring methods significantly differed based on impairment severity. Overall, more severely impaired participants had significantly higher E-Scores and longer completion times but demonstrated comparable errors to their less impaired counterparts. Contrary to the previously established E-Score cutoff of ≥17, a cutoff of ≥22 was required to maintain adequate specificity in our total sample, with significantly higher adjustments required in the Mild and Major Neurocognitive Disorder subgroups (≥27 and ≥40, respectively). A cutoff of >3 errors achieved adequate specificity in our sample, suggesting that error scores may produce lower false positive rates than E-Scores and completion time scores, both of which overemphasize speed and could inadvertently penalize more severely impaired individuals.

CONCLUSIONS

In a dementia clinic setting, error scores on the DCT may have greater utility in detecting non-credible performance than E-Scores and completion time scores, particularly among more severely impaired individuals. Future research should establish and cross-validate the sensitivity and specificity of the DCT for assessing performance validity.

Cross-Cultural Performance Validity Testing: Managing False Positives in Examinees with Limited English Proficiency

Base rates of failure (BR_Fail) on performance validity tests (PVTs) were examined in university students with limited English proficiency (LEP). BR_Fail was calculated for several free-standing and embedded PVTs. All free-standing PVTs and certain embedded indicators were robust to LEP. However, LEP was associated with unacceptably high BR_Fail (20-50%) on several embedded PVTs with high levels of verbal mediation (even multivariate models of PVT could not contain BR_Fail). In conclusion, failing free-standing/dedicated PVTs cannot be attributed to LEP. However, the elevated BR_Fail on several embedded PVTs in university students suggest an unacceptably high overall risk of false positives associated with LEP.

The BNT-15 provides an accurate measure of English proficiency in cognitively intact bilinguals - a study in cross-cultural assessment

This study was designed to replicate earlier reports of the utility of the Boston Naming Test - Short Form (BNT-15) as an index of limited English proficiency (LEP). Twenty-eight English-Arabic bilingual student volunteers were administered the BNT-15 as part of a brief battery of cognitive tests. The majority (23) were women, and half had LEP. Mean age was 21.1 years. The BNT-15 was an excellent psychometric marker of LEP status (area under the curve: .990-.995). Participants with LEP underperformed on several cognitive measures (verbal comprehension, visuomotor processing speed, single word reading, and performance validity tests). Although no participant with LEP failed the accuracy cutoff on the Word Choice Test, 35.7% of them failed the time cutoff. Overall, LEP was associated with an increased risk of failing performance validity tests. Previously published BNT-15 validity cutoffs had unacceptably low specificity (.33-.52) among participants with LEP. The BNT-15 has the potential to serve as a quick and effective objective measure of LEP. Students with LEP may need academic accommodations to compensate for slower test completion time. Likewise, LEP status should be considered for exemption from failing performance validity tests to protect against false positive errors.

Cultural accommodations for cutoff scores of embedded performance validity tests in a Spanish college population

OBJECTIVE

The objective of this study was to identify specific cutoff scores for three commonly used embedded performance validity tests (PVTs) for a Spanish speaking population. Culturally adapted cutoff scores for embedded PVTs were established using an analog study design. In addition, the psychometric properties of these measures when applying culturally adapted scores as compared to non-adapted scores were analyzed.

METHOD

Participants (N = 114) were administered three embedded PVTs (Reliable Digit Span, Phonetic Fluency Test, and Animal Semantic Fluency Test) in a randomized order. Following an analog design, control participants were instructed to perform to the best of their abilities and the analog group was instructed to simulate cognitive impairment.

RESULTS

In keeping with guidelines for specificity and sensitivity, the most culturally appropriate scores of ≤6, ≤27, and ≤16 were determined for the Reliable Digit Span, Phonetic Fluency Test, and the Semantic Fluency Test, respectively.

CONCLUSIONS

This the first study addressing culturally sensitive cutoffs for commonly used embedded validity measures using a European Spanish population. While these findings cannot be generalized to forensic or clinical populations at the present time, they support the claim that specific cutoff scores that are sensitive to cultural variables are necessary in addressing embedded validity measures of the Reliable Digit Span, Phonetic Fluency Test, and Semantic Fluency Test.

The Coin in Hand-Extended Version: Development and Validation of a Multicultural Performance Validity Test

The number of computerized and reliable performance validity tests are scarce. This study aims to address this issue by validating a free and computerized performance validity test: the Coin in Hand-Extended Version (CIH-EV). The CIH-EV test was administered in four countries (Colombia, Spain, Portugal, and the United States) and performance was compared with other commonly used validated tests. Results showed that the CIH-EV has at least 95% specificity and 62% sensitivity, and performance was highly correlated with scores on the Test of Memory Malingering, Victoria Symptom Validity Test, and Digit Span of the Wechsler Adult Intelligence Scale. There were no significant differences in scores across countries, suggesting that the CIH-EV performs similarly in a variety of cultures. Our findings suggest that the CIH-EV has the potential to serve as a valid validity test either alone or as a supplement to other commonly used validity tests.