A signal detection-item response theory model for evaluating neuropsychological measures

Recent Advances in Neuropsychological Test Interpretation for Clinical Practice

Much attention in the field of clinical neuropsychology has focused on adapting to the modern healthcare environment by advancing telehealth and promoting technological innovation in assessment. Perhaps as important (but less discussed) are advances in the development and interpretation of normative neuropsychological test data. These techniques can yield improvement in diagnostic decision-making and treatment planning with little additional cost. Brooks and colleagues (Can Psychol 50: 196-209, 2009) eloquently summarized best practices in normative data creation and interpretation, providing a practical overview of norm development, measurement error, the base rates of low scores, and methods for assessing change. Since the publication of this seminal work, there have been several important advances in research on development and interpretation of normative neuropsychological test data, which may be less familiar to the practicing clinician. Specifically, we provide a review of the literature on regression-based normed scores, item response theory, multivariate base rates, summary/factor scores, cognitive intraindividual variability, and measuring change over time. For each topic, we include (1) an overview of the method, (2) a rapid review of the recent literature, (3) a relevant case example, and (4) a discussion of limitations and controversies. Our goal was to provide a primer for use of normative neuropsychological test data in neuropsychological practice.

Face identification proficiency test designed using item response theory

Measures of face-identification proficiency are essential to ensure accurate and consistent performance by professional forensic face examiners and others who perform face-identification tasks in applied scenarios. Current proficiency tests rely on static sets of stimulus items and so cannot be administered validly to the same individual multiple times. To create a proficiency test, a large number of items of "known" difficulty must be assembled. Multiple tests of equal difficulty can be constructed then using subsets of items. We introduce the Triad Identity Matching (TIM) test and evaluate it using item response theory (IRT). Participants view face-image "triads" (N = 225) (two images of one identity, one image of a different identity) and select the different identity. In Experiment 3, university students (N = 197) showed wide-ranging accuracy on the TIM test, and IRT modeling demonstrated that the TIM items span various difficulty levels. In Experiment 3, we used IRT-based item metrics to partition the test into subsets of specific difficulties. Simulations showed that subsets of the TIM items yielded reliable estimates of subject ability. In Experiments 3a and b, we found that the student-derived IRT model reliably evaluated the ability of non-student participants and that ability generalized across different test sessions. In Experiment 3c, we show that TIM test performance correlates with other common face-recognition tests. In summary, the TIM test provides a starting point for developing a framework that is flexible and calibrated to measure proficiency across various ability levels (e.g., professionals or populations with face-processing deficits).

Developing and Validating a Facial Emotion Recognition Task With Graded Intensity

Facial emotion recognition (FER) tasks are often digitally altered to vary expression intensity; however, such tasks have unknown psychometric properties. In these studies, an FER task was developed and validated-the Graded Emotional Face Task (GEFT)-which provided an opportunity to examine the psychometric properties of such tasks. Facial expressions were altered to produce five intensity levels for six emotions (e.g., 40% anger). In Study 1, 224 undergraduates viewed subsets of these faces and labeled the expressions. An item selection algorithm was used to maximize internal consistency and balance gender and ethnicity. In Study 2, 219 undergraduates completed the final GEFT and a multimethod battery of validity measures. Finally, in Study 3, 407 undergraduates oversampled for borderline personality disorder (BPD) completed the GEFT and a self-report BPD measure. Broad FER scales (e.g., overall anger) demonstrated evidence of reliability and validity; however, more specific subscales (e.g., 40% anger) had more variable psychometric properties. Notably, ceiling/floor effects appeared to decrease both internal consistency and limit external validity correlations. The findings are discussed from the perspective of measurement issues in the social cognition literature.

Construction of a computerized adaptive test (CAT-CCNB) for efficient neurocognitive and clinical psychopathology assessment

BACKGROUND

Traditional paper-and-pencil neurocognitive evaluations and semi-structured mental health interviews can take hours to administer and score. Computerized assessment has decreased that burden substantially, and contemporary psychometric tools such as item response theory and computerized adaptive testing (CAT) allow even further abbreviation.

NEW METHOD

The goal of this paper was to describe the application of CAT and related methods to the Penn Computerized Neurocognitive Battery (CNB) and a well-validated clinical assessment in order to increase efficiency in assessment and relevant domain coverage. To calibrate item banks for CAT, N = 5053 participants (63% female; mean age 45 years, range 18-80) were collected from across the United States via crowdsourcing, providing item parameters that were then linked to larger item banks and used in individual test construction. Tests not amenable to CAT were abbreviated using complementary short-form methods.

RESULTS

The final "CAT-CCNB" battery comprised 21 cognitive tests (compared to 14 in the original) and five adaptive clinical scales (compared to 16 in the original).

COMPARISON WITH EXISTING METHODS

This new battery, derived with contemporary psychometric approaches, provides further improvements over existing assessments that use collections of fixed-length tests developed for stand-alone administration. The CAT-CCNB provides an improved version of the CNB that shows promise as a maximally efficient tool for neuropsychiatric assessment.

CONCLUSIONS

We anticipate CAT-CCNB will help satisfy the clear need for broad yet efficient measurement of cognitive and clinical domains, facilitating implementation of large-scale, "big science" approaches to data collection, and potential widespread clinical implementation.

Problematic Shopping Behavior: An Item Response Theory Examination of the Seven-Item Bergen Shopping Addiction Scale

There has been an increasing amount of research examining problematic shopping behavior (PSB), often referred to in the psychological literature as "compulsive buying" or "shopping addiction." A popular scale for assessing the risk of PSB is the seven-item Bergen Shopping Addiction Scale (BSAS). To expand our knowledge of the psychometric properties of this instrument, the present study employed Item Response Theory (IRT) and differential item functioning analyses (DIF) while concurrently attempting to determine a preliminary cut-off point. A relatively large community sample completed the BSAS online (N = 968, M age = 29.5 years, SD age = 9.36, 32.5% women). IRT analyses showed differences regarding the BSAS items' discrimination, difficulty, and precision, with a raw score exceeding 23 (out of 28) indicating a higher risk of shopping addiction. Finally, while most BSAS items operated equally among males and females, Item 2 (mood modification) required a higher level of shopping addiction behaviors to be endorsed by males. The BSAS functions as a reliable assessment of the risk of shopping addiction, particularly between average and high levels of the trait. Clinical implications are discussed in light of these findings.

Detecting the Inverted-U in fMRI Studies of Schizophrenia: A Comparison of Three Analysis Methods

OBJECTIVE

Cognitive tasks are used to probe neuronal activity during functional magnetic resonance imaging (fMRI) to detect signs of aberrant cognitive functioning in patients diagnosed with schizophrenia (SZ). However, nonlinear (inverted-U-shaped) associations between neuronal activity and task difficulty can lead to misinterpretation of group differences between patients and healthy comparison subjects (HCs). In this paper, we evaluated a novel method for correcting these misinterpretations based on conditional performance analysis.

METHOD

Participants included 25 HCs and 27 SZs who performed a working memory (WM) task (N-back) with 5 load conditions while undergoing fMRI. Neuronal activity was regressed onto: 1) task load (i.e., parametric task levels), 2) marginal task performance (i.e., performance averaged over all load conditions), or 3) conditional task performance (i.e., performance within each load condition).

RESULTS

In most regions of interest, conditional performance analysis uniquely revealed inverted-U-shaped neuronal activity in both SZs and HCs. After accounting for conditional performance differences between groups, we observed few difference in both the pattern and level of neuronal activity between SZs and HCs within regions that are classically associated with WM functioning (e.g., posterior dorsolateral prefrontal and parietal association cortices). However, SZs did show aberrant activity within the anterior dorsolateral prefrontal cortex.

CONCLUSIONS

Interpretations of differences in neuronal activity between groups, and of associations between neuronal activity and performance, should be considered within the context of task performance. Whether conditional performance-based differences reflect compensation, dedifferentiation, or other processes is not a question that is easily resolved by examining activation and performance data alone.

Bayesian modeling of item heterogeneity in dichotomous recognition memory data and prospects for computerized adaptive testing

Most current models of recognition memory fail to separately model item and person heterogeneity which makes it difficult to assess ability at the latent construct level and prevents the administration of adaptive tests. Here we propose to employ a General Condorcet Model for Recognition (GCMR) in order to estimate ability, response bias and item difficulty in dichotomous recognition memory tasks. Using a Bayesian modeling framework and MCMC inference, we perform 3 separate validation studies comparing GCMR to the Rasch model from IRT and the 2-High-Threshold (2HT) recognition model. First, two simulations demonstrate that recovery of GCMR ability estimates with varying sparsity and test difficulty is more robust and that estimates improve from the two other models under common test scenarios. Then, using a real dataset, face validity is confirmed by replicating previous findings of general and domain-specific age effects (Güsten et al. in Cortex 137:138-148, https://doi.org/10.1016/j.cortex.2020.12.017 , 2021). Using cross-validation we show better out-of-sample prediction for the GCMR as compared to Rasch and 2HT model. In addition, we present a hierarchical extension of the model that is able to estimate age- and domain-specific effects directly, without recurring to a two-stage procedure. Finally, an adaptive test using the GCMR is simulated, showing that the test length necessary to obtain reliable ability estimates can be significantly reduced compared to a non-adaptive procedure. The GCMR allows to model trial-by-trial performance and to increase the efficiency and reliability of recognition memory assessments.

Neurocognitive Assessment Tools for Military Personnel With Mild Traumatic Brain Injury: Scoping Literature Review

BACKGROUND

Mild traumatic brain injury (mTBI) occurs at a higher frequency among military personnel than among civilians. A common symptom of mTBIs is cognitive dysfunction. Health care professionals use neuropsychological assessments as part of a multidisciplinary and best practice approach for mTBI management. Such assessments support clinical diagnosis, symptom management, rehabilitation, and return-to-duty planning. Military health care organizations currently use computerized neurocognitive assessment tools (NCATs). NCATs and more traditional neuropsychological assessments present unique challenges in both clinical and military settings. Many research gaps remain regarding psychometric properties, usability, acceptance, feasibility, effectiveness, sensitivity, and utility of both types of assessments in military environments.

OBJECTIVE

The aims of this study were to explore evidence regarding the use of NCATs among military personnel who have sustained mTBIs; evaluate the psychometric properties of the most commonly tested NCATs for this population; and synthesize the data to explore the range and extent of NCATs among this population, clinical recommendations for use, and knowledge gaps requiring future research.

METHODS

Studies were identified using MEDLINE, Embase, American Psychological Association PsycINFO, CINAHL Plus with Full Text, Psych Article, Scopus, and Military & Government Collection. Data were analyzed using descriptive analysis, thematic analysis, and the Randolph Criteria. Narrative synthesis and the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-analyses extension for Scoping Reviews) guided the reporting of findings. The psychometric properties of NCATs were evaluated with specific criteria and summarized.

RESULTS

Of the 104 papers, 33 met the inclusion criteria for this scoping review. Thematic analysis and NCAT psychometrics were reported and summarized.

CONCLUSIONS

When considering the psychometric properties of the most commonly used NCATs in military populations, these assessments have yet to demonstrate adequate validity, reliability, sensitivity, and clinical utility among military personnel with mTBIs. Additional research is needed to further validate NCATs within military populations, especially for those living outside of the United States and individuals experiencing other conditions known to adversely affect cognitive processing. Knowledge gaps remain, warranting further study of psychometric properties and the utility of baseline and normative testing for NCATs.

Latent Variable Modeling and Adaptive Testing for Experimental Cognitive Psychopathology Research

The adaptation of experimental cognitive tasks into measures that can be used to quantify neurocognitive outcomes in translational studies and clinical trials has become a key component of the strategy to address psychiatric and neurological disorders. Unfortunately, while most experimental cognitive tests have strong theoretical bases, they can have poor psychometric properties, leaving them vulnerable to measurement challenges that undermine their use in applied settings. Item response theory-based computerized adaptive testing has been proposed as a solution but has been limited in experimental and translational research due to its large sample requirements. We present a generalized latent variable model that, when combined with strong parametric assumptions based on mathematical cognitive models, permits the use of adaptive testing without large samples or the need to precalibrate item parameters. The approach is demonstrated using data from a common measure of working memory-the N-back task-collected across a diverse sample of participants. After evaluating dimensionality and model fit, we conducted a simulation study to compare adaptive versus nonadaptive testing. Computerized adaptive testing either made the task 36% more efficient or score estimates 23% more precise, when compared to nonadaptive testing. This proof-of-concept study demonstrates that latent variable modeling and adaptive testing can be used in experimental cognitive testing even with relatively small samples. Adaptive testing has the potential to improve the impact and replicability of findings from translational studies and clinical trials that use experimental cognitive tasks as outcome measures.

When CAT is not an option: complementary methods of test abbreviation for neurocognitive batteries

INTRODUCTION

There is an obvious need for efficient measurement of neuropsychiatric phenomena. A proven method-computerized adaptive testing (CAT)-is not feasible for all tests, necessitating alternatives for increasing test efficiency.

METHODS

We combined/compared two methods for abbreviating rapid tests using two tests unamenable to CAT (a Continuous Performance Test [CPT] and n-back test [NBACK]). N=9,498 (mean age 14.2 years; 52% female) were administered the tests, and abbreviation was accomplished using methods answering two questions: what happens to measurement error as items are removed, and what happens to correlations with validity criteria as items are removed. The first was investigated using quasi-CAT simulation, while the second was investigated using bootstrapped confidence intervals around full-form-short-form comparisons.

RESULTS

Results for the two methods overlapped, suggesting that the CPT could be abbreviated to 57% of original and NBACK could be abbreviated to 87% of original with the max-acceptable loss of precision and min-acceptable relationships with validity criteria.

CONCLUSIONS

This method combination shows promise for use in other test types, and the divergent results for the CPT/NBACK demonstrate the methods' abilities to detect when a test should not be shortened. The methods should be used in combination because they emphasize complementary measurement qualities: precision/validity..

Paradigm Shift Toward Digital Neuropsychology and High-Dimensional Neuropsychological Assessments: Review

Neuropsychologists in the digital age have increasing access to emerging technologies. The National Institutes of Health (NIH) initiatives for behavioral and social sciences have emphasized these developing scientific and technological potentials (eg, novel sensors) for augmented characterization of neurocognitive, behavioral, affective, and social processes. Perhaps these innovative technologies will lead to a paradigm shift from disintegrated and data-poor behavioral science to cohesive and data-rich science that permits improved translation from bench to bedside. The 4 main advances influencing the scientific priorities of a recent NIH Office of Behavioral and Social Sciences Research strategic plan include the following: integration of neuroscience into behavioral and social sciences, transformational advances in measurement science, digital intervention platforms, and large-scale population cohorts and data integration. This paper reviews these opportunities for novel brain-behavior characterizations. Emphasis is placed on the increasing concern of neuropsychology with these topics and the need for development in these areas to maintain relevance as a scientific discipline and advance scientific developments. Furthermore, the effects of such advancements necessitate discussion and modification of training as well as ethical and legal mandates for neuropsychological research and praxes.

Studying Individual Differences in Spatial Cognition Through Differential Item Functioning Analysis

Background: In the field of spatial cognition, the study of individual differences represents a typical research topic. Gender and age have been prominently investigated. A promising statistical technique used to identify the different responses to items in relation to different group memberships is the Differential Item Functioning Analysis (DIF). The aim of the present study was to investigate the DIF of the Landmark positioning on a Map (LPM) task, across age groups (young and elderly) and gender, in a sample of 400 healthy human participants. Methods: LPM is a hometown map completion test based on well-known and familiar landmarks used to assess allocentric mental representations. DIF was assessed on LPM items two times: on categorical (i.e., positions) and coordinate (i.e., distances) scores, separately. Results: When positions and distances were difficult to assess with respect to the intended reference point, the probability to endorse the items seemed to get worse for the elderly compared to the younger participants. Instead other features of landmarks (high pleasantness, restorativeness) seemed to improve the elderly performance. A gender-related improvement of probability to endorse distance estimation of some landmarks, favoring women, emerged, probably associated with their repeated experiences with those landmarks. Overall, the complexity of the task seemed to have a differential impact on young and elderly people while gender-oriented activities and places seemed to have a differential impact on men and women. Conclusions: For the first time DIF was applied to a spatial mental representation task, based on the schematic sketch maps of the participants. The application of DIF to the study of individual differences in spatial cognition should become a systematic routine to early detect differential items, improving knowledge, as well as experimental control, on individual differences.

An efficient and adaptive test of auditory mental imagery

The ability to silently hear music in the mind has been argued to be fundamental to musicality. Objective measurements of this subjective imagery experience are needed if this link between imagery ability and musicality is to be investigated. However, previous tests of musical imagery either rely on self-report, rely on melodic memory, or do not cater in range of abilities. The Pitch Imagery Arrow Task (PIAT) was designed to address these shortcomings; however, it is impractically long. In this paper, we shorten the PIAT using adaptive testing and automatic item generation. We interrogate the cognitive processes underlying the PIAT through item response modelling. The result is an efficient online test of auditory mental imagery ability (adaptive Pitch Imagery Arrow Task: aPIAT) that takes 8 min to complete, is adaptive to participant's individual ability, and so can be used to test participants with a range of musical backgrounds. Performance on the aPIAT showed positive moderate-to-strong correlations with measures of non-musical and musical working memory, self-reported musical training, and general musical sophistication. Ability on the task was best predicted by the ability to maintain and manipulate tones in mental imagery, as well as to resist perceptual biases that can lead to incorrect responses. As such, the aPIAT is the ideal tool in which to investigate the relationship between pitch imagery ability and musicality.