Empirical Evaluation of Computer-Adaptive Alternate Short Forms for the Assessment of Anomia Severity

Assessing Relative Linguistic Impairment With Model-Based Item Selection

PURPOSE

A picture naming test is presented that reveals impairment to specific mechanisms involved in the naming process, using accuracy scores on curated item sets. A series of psychometric validation experiments are reported.

METHOD

Using a computational model that enables estimation of item difficulty at the lexical and sublexical stages of word retrieval, two complimentary sets of items were constructed that challenge the respective psycholinguistic levels of representation. The difference in accuracy between these item sets yields the relative linguistic impairment (RLI) score. In a cohort of 91 people with chronic left-hemisphere stroke who enrolled in a clinical trial for anomia, we assessed psychometric properties of the RLI score and then used the new scale to make predictions about other language behaviors, lesion distributions, and functional activation during naming.

RESULTS

RLI scores had adequate psychometric properties for clinical significance. RLI scores contained predictive information about spontaneous speech fluency, over and above accuracy. A dissociation was observed between performance on the RLI item sets and performance on the subtests of an independent language battery. Sublexical RLI was significantly associated with apraxia of speech and with lesions encompassing perisylvian regions, while lexical RLI was associated with lesions to deep white matter. The RLI construct was reflected in functional brain activity during naming, independent of overall accuracy, with a respective shift of activation between dorsal and ventral networks responsible for different aspects of word retrieval.

CONCLUSION

The RLI assessment satisfies the psychometric requirements to serve as a useful clinical measure.

Response Latencies During Confrontation Picture Naming in Aphasia: Are Proxy Measurements Sufficient?

PURPOSE

Anomia, or word-finding difficulty, is a prevalent and persistent feature of aphasia, a neurogenic language disorder affecting millions of people in the United States. Anomia assessments are essential for measuring performance and monitoring outcomes in clinical settings. This study aims to evaluate the reliability of response time (RT) annotation based on spectrograms and assess the predictive utility of proxy RTs collected during computerized naming tests.

METHOD

Archival data from 10 people with aphasia were used. Trained research assistants phonemically transcribed participants' responses, and RTs were generated from the onset of picture stimulus to the initial phoneme of the first complete attempt. RTs were measured in two ways: hand-generated RTs (from spectrograms) and proxy RTs (automatically extracted online). Interrater agreement was evaluated based on interclass correlation coefficients and generalizability theory tools including variance partitioning and the φ-coefficient. The predictive utility of proxy RTs was evaluated within a linear mixed-effects framework.

RESULTS

RT annotation reliability showed near-perfect agreement across research assistants (φ-coefficient = .93), and the variance accounted for by raters was negligible. Furthermore, proxy RTs significantly and strongly predicted hand-annotated RTs (R2 = ~0.82), suggesting their utility as an alternative measure.

CONCLUSIONS

The study confirms the reliability of RT annotation and demonstrates the predictive utility of proxy RTs in estimating RTs during computerized naming tests. Incorporating proxy RTs can enhance clinical assessments, providing additional information for cognitive measurement. Further research with larger samples and exploring the impact of using proxy RTs in different psychometric models could optimize clinical protocols and improve communication interventions for individuals with aphasia.

Item Response Theory Modeling of the Verb Naming Test

PURPOSE

Item response theory (IRT) is a modern psychometric framework with several advantageous properties as compared with classical test theory. IRT has been successfully used to model performance on anomia tests in individuals with aphasia; however, all efforts to date have focused on noun production accuracy. The purpose of this study is to evaluate whether the Verb Naming Test (VNT), a prominent test of action naming, can be successfully modeled under IRT and evaluate its reliability.

METHOD

We used responses on the VNT from 107 individuals with chronic aphasia from AphasiaBank. Unidimensionality and local independence, two assumptions prerequisite to IRT modeling, were evaluated using factor analysis and Yen's Q 3 statistic (Yen, 1984), respectively. The assumption of equal discrimination among test items was evaluated statistically via nested model comparisons and practically by using correlations of resulting IRT-derived scores. Finally, internal consistency, marginal and empirical reliability, and conditional reliability were evaluated.

RESULTS

The VNT was found to be sufficiently unidimensional with the majority of item pairs demonstrating adequate local independence. An IRT model in which item discriminations are constrained to be equal demonstrated fit equivalent to a model in which unique discrimination parameters were estimated for each item. All forms of reliability were strong across the majority of IRT ability estimates.

CONCLUSIONS

Modeling the VNT using IRT is feasible, yielding ability estimates that are both informative and reliable. Future efforts are needed to quantify the validity of the VNT under IRT and determine the extent to which it measures the same construct as other anomia tests.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.22329235.

Accuracy of Naming Error Profiles Elicited From Adaptive Short Forms of the Philadelphia Naming Test

PURPOSE

The purpose of this study was to evaluate whether a short-form computerized adaptive testing (CAT) version of the Philadelphia Naming Test (PNT) provides error profiles and model-based estimates of semantic and phonological processing that agree with the full test.

METHOD

Twenty-four persons with aphasia took the PNT-CAT and the full version of the PNT (hereinafter referred to as the "full PNT") at least 2 weeks apart. The PNT-CAT proceeded in two stages: (a) the PNT-CAT30, in which 30 items were selected to match the evolving ability estimate with the goal of producing a 50% error rate, and (b) the PNT-CAT60, in which an additional 30 items were selected to produce a 75% error rate. Agreement was evaluated in terms of the root-mean-square deviation of the response-type proportions and, for individual response types, in terms of agreement coefficients and bias. We also evaluated agreement and bias for estimates of semantic and phonological processing derived from the semantic-phonological interactive two-step model (SP model) of word production.

RESULTS

The results suggested that agreement was poorest for semantic, formal, mixed, and unrelated errors, all of which were underestimated by the short forms. Better agreement was observed for correct and nonword responses. SP model weights estimated by the short forms demonstrated no substantial bias but generally inadequate agreement with the full PNT, which itself showed acceptable test-retest reliability for SP model weights and all response types except for formal errors.

DISCUSSION

Results suggest that the PNT-CAT30 and the PNT-CAT60 are generally inadequate for generating naming error profiles or model-derived estimates of semantic and phonological processing ability. Post hoc analyses suggested that increasing the number of stimuli available in the CAT item bank may improve the utility of adaptive short forms for generating error profiles, but the underlying theory also suggests that there are limitations to this approach based on a unidimensional measurement model.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.22320814.

ParAlg: A Paraphasia Algorithm for Multinomial Classification of Picture Naming Errors

PURPOSE

A preliminary version of a paraphasia classification algorithm (henceforth called ParAlg) has previously been shown to be a viable method for coding picture naming errors. The purpose of this study is to present an updated version of ParAlg, which uses multinomial classification, and comprehensively evaluate its performance when using two different forms of transcribed input.

METHOD

A subset of 11,999 archival responses produced on the Philadelphia Naming Test were classified into six cardinal paraphasia types using ParAlg under two transcription configurations: (a) using phonemic transcriptions for responses exclusively (phonemic-only) and (b) using phonemic transcriptions for nonlexical responses and orthographic transcriptions for lexical responses (orthographic-lexical). Agreement was quantified by comparing ParAlg-generated paraphasia codes between configurations and relative to human-annotated codes using four metrics (positive predictive value, sensitivity, specificity, and F1 score). An item-level qualitative analysis of misclassifications under the best performing configuration was also completed to identify the source and nature of coding discrepancies.

RESULTS

Agreement between ParAlg-generated and human-annotated codes was high, although the orthographic-lexical configuration outperformed phonemic-only (weighted-average F1 scores of .78 and .87, respectively). A qualitative analysis of the orthographic-lexical configuration revealed a mix of human- and ParAlg-related misclassifications, the former of which were related primarily to phonological similarity judgments whereas the latter were due to semantic similarity assignment.

CONCLUSIONS

ParAlg is an accurate and efficient alternative to manual scoring of paraphasias, particularly when lexical responses are orthographically transcribed. With further development, it has the potential to be a useful software application for anomia assessment.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.22087763.

The Severity-Calibrated Aphasia Naming Test

PURPOSE

We present a 20-item naming test, the Severity-Calibrated Aphasia Naming Test (SCANT), that can serve as a proxy measure for an aphasia severity scale that is derived from a thorough test battery of connected speech production, single-word production, speech repetition, and auditory verbal comprehension.

METHOD

We use lasso regression and cross-validation to identify an optimal subset from a set of 174 pictures to be named for prediction of aphasia severity, based on data from 200 participants with left-hemisphere stroke who were quasirandomly selected to represent the full impairment scale. Data from 20 healthy controls (i.e., participant caretakers/spouses) were also analyzed. We examine interrater reliability, test-retest reliability, sensitivity and specificity to the presence of aphasia, sensitivity to therapy gains, and external validity (i.e., correlation with aphasia severity measures) for the SCANT.

RESULTS

The SCANT has extremely high interrater reliability, and it is sensitive and specific to the presence of aphasia. We demonstrate the superiority of predictions based on the SCANT over those based on the full set of naming items. We estimate a 15% reduction in power when using the SCANT score versus the full test battery's aphasia severity score as an outcome measure; for example, to maintain the same power to detect a significant group average change in aphasia severity, a study with 25 participants using the full test battery to measure treatment effectiveness would require 30 participants if the SCANT were to be used as the testing instrument instead.

CONCLUSION

We provide a linear model to convert SCANT scores to aphasia severity scores, and we identify a change score cutoff of four SCANT items to obtain a high degree of confidence based on test-retest SCANT data and the modeled relation between SCANT and aphasia severity scores.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.21476871.

Beyond Percent Correct: Measuring Change in Individual Picture Naming Ability

PURPOSE

Meaningful changes in picture naming responses may be obscured when measuring accuracy instead of quality. A statistic that incorporates information about the severity and nature of impairments may be more sensitive to the effects of treatment.

METHOD

We analyzed data from repeated administrations of a naming test to 72 participants with stroke aphasia in a clinical trial for anomia therapy. Participants were divided into two groups for analysis to demonstrate replicability. We assessed reliability among response type scores from five raters. We then derived four summary statistics of naming ability and their changes over time for each participant: (a) the standard accuracy measure, (b) an accuracy measure adjusted for item difficulty, (c) an accuracy measure adjusted for item difficulty for specific response types, and (d) a distance measure adjusted for item difficulty for specific response types. While accuracy measures address the likelihood of a correct response, the distance measure reflects that different response types range in their similarity to the target. Model fit was assessed. The frequency of significant improvements and the average magnitude of improvements for each summary statistic were compared between treatment groups and a control group. Effect sizes for each model-based statistic were compared with the effect size for the standard accuracy measure.

RESULTS

Interrater and intrarater reliability were near perfect, on average, though compromised somewhat by phonological-level errors. The effects of treatment were more evident, in terms of both frequency and magnitude, when using the distance measure versus the other accuracy statistics.

CONCLUSIONS

Consideration of item difficulty and response types revealed additional effects of treatment on naming scores beyond those observed for the standard accuracy measure. The results support theories that assume naming ability is decomposable into subabilities rather than being monolithic, suggesting new opportunities for measuring treatment outcomes.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.17019515.

Computer Adaptive Testing for the Assessment of Anomia Severity

Anomia assessment is a fundamental component of clinical practice and research inquiries involving individuals with aphasia, and confrontation naming tasks are among the most commonly used tools for quantifying anomia severity. While currently available confrontation naming tests possess many ideal properties, they are ultimately limited by the overarching psychometric framework they were developed within. Here, we discuss the challenges inherent to confrontation naming tests and present a modern alternative to test development called item response theory (IRT). Key concepts of IRT approaches are reviewed in relation to their relevance to aphasiology, highlighting the ability of IRT to create flexible and efficient tests that yield precise measurements of anomia severity. Empirical evidence from our research group on the application of IRT methods to a commonly used confrontation naming test is discussed, along with future avenues for test development.

The Aphasia Communication Outcome Measure: Motivation, Development, Validity Evidence, and Interpretation of Change Scores

The Aphasia Communication Outcome Measure (ACOM) is a patient-reported measure of communicative functioning developed for persons with stroke-induced aphasia. It was motivated by the desire to include the perspective of persons with aphasia in the measurement of treatment outcomes and to apply newly accessible psychometric tools to improve the quality and usefulness of available outcome measures for aphasia. The ACOM was developed within an item response theory framework, and the validity of the score estimates it provides is supported by evidence based on its content, internal structure, relationships with other variables, stability over time, and responsiveness to treatment. This article summarizes the background and motivation for the ACOM, the steps in its initial development, evidence supporting its validity as a measure of patient-reported communication functioning, and current recommendations for interpreting change scores.

Disentangling the Psycholinguistic Loci of Anomia with Cognitive Psychometric Models

This article reviews advanced statistical techniques for measuring impairments in object naming, particularly in the context of stroke-induced aphasia. Traditional testing strategies can be challenged by the multifaceted nature of impairments that arise due to the complex relationships between localized brain damage and disruption to the cognitive processes required for successful object naming. Cognitive psychometric models can combine response-type analysis with item-response theory to yield accurate estimates of multiple abilities using data collected from a single task. The models also provide insights about how the test items can be challenging in different ways. Although more work is needed to fully optimize their clinical utility in practice, these formal concepts can guide thoughtful selection of stimuli used in treatment or assessment, as well as providing a framework to interpret response-type data.