Beyond Percent Correct: Measuring Change in Individual Picture Naming Ability

Assessing Relative Linguistic Impairment With Model-Based Item Selection

PURPOSE

A picture naming test is presented that reveals impairment to specific mechanisms involved in the naming process, using accuracy scores on curated item sets. A series of psychometric validation experiments are reported.

METHOD

Using a computational model that enables estimation of item difficulty at the lexical and sublexical stages of word retrieval, two complimentary sets of items were constructed that challenge the respective psycholinguistic levels of representation. The difference in accuracy between these item sets yields the relative linguistic impairment (RLI) score. In a cohort of 91 people with chronic left-hemisphere stroke who enrolled in a clinical trial for anomia, we assessed psychometric properties of the RLI score and then used the new scale to make predictions about other language behaviors, lesion distributions, and functional activation during naming.

RESULTS

RLI scores had adequate psychometric properties for clinical significance. RLI scores contained predictive information about spontaneous speech fluency, over and above accuracy. A dissociation was observed between performance on the RLI item sets and performance on the subtests of an independent language battery. Sublexical RLI was significantly associated with apraxia of speech and with lesions encompassing perisylvian regions, while lexical RLI was associated with lesions to deep white matter. The RLI construct was reflected in functional brain activity during naming, independent of overall accuracy, with a respective shift of activation between dorsal and ventral networks responsible for different aspects of word retrieval.

CONCLUSION

The RLI assessment satisfies the psychometric requirements to serve as a useful clinical measure.

Reliability, Measurement Error, and Responsiveness of the Voice Handicap Index: A Systematic Review and Meta-analysis

OBJECTIVE

To analyze the reliability, measurement error, and responsiveness of the Voice Handicap Index (VHI) for measuring voice handicap in individuals with voice disorders.

METHODS

This systematic review followed the recommendations of the COnsensus-based Standards for the Selection of Health Measurement INstruments. Studies that validated the VHI, analyzed the measurement properties of reliability, measurement error, or responsiveness, and had dysphonic individuals as the study population were included. The following electronic databases were searched: Cochrane Library, EMBASE, LILACS, PubMed, Scopus, and Web of Science. The manual search was carried out on gray literature in the Biblioteca Digital Brasileira de Teses e Dissertações and ProQuest Dissertation & Theses, in addition to mapping citations and consulting an expert in the field. Evidence selection, data extraction, risk of bias analysis, certainty of evidence, and good psychometric measurements were performed by two blinded and independent reviewers. A meta-analysis was performed using Fisher's transformed r-to-z correlation coefficient and standardized mean difference. Heterogeneity was calculated using Tau² and I² statistical tests in JAMOVI 2.3.2 software.

RESULTS

Eighty studies were included in the meta-analysis. When assessing the risk of bias, most studies were classified as inadequate during the reliability and measurement error stages. In the responsiveness stage, they were classified as doubtful in the sensitivity and specificity sub-boxes and good in the comparison sub-box with a gold standard instrument. For psychometric properties, most studies were classified as indeterminate in terms of both reliability and responsiveness. In test-retest reliability and responsiveness, the average outcome differed significantly from zero, indicating agreement between the test and retest moments and a significant reduction in the VHI score after intervention. Considering certainty of the evidence, the level of evidence was very low in both the reliability and responsiveness stages. The analyzed studies did not evaluate observational errors, and hence, they were not considered in this review.

CONCLUSION

The VHI proved to be reliable and responsive in measuring voice handicap. However, it should be used with caution, as there is heterogeneity, a risk of bias, and no adherence to the analysis of observational error.

Analyzing direct effects of education level and estimated IQ between cognitively intact Mexican Americans and Non-Hispanic whites on a confrontational naming task

Confrontational naming is widely used in diagnosing neurodegenerative disorders like MCI and dementia, and previous research indicates that healthy Non-Hispanic Whites outperform Hispanics in such tasks. However, understanding the factors contributing to score differences among ethnic groups remains limited. This study focuses on cognitively intact Mexican Americans and Non-Hispanic White older adults from the TARCC Hispanic Cohort project. Hierarchical regression analyses reveal that sex, age, ethnicity, education level, and estimated IQ significantly predict performance on the Boston Naming Test (BNT). Notably, education level and estimated IQ more strongly influence BNT performance in Mexican Americans than in Non-Hispanic Whites. When controlling for education level, estimated IQ has a more pronounced impact on BNT performance in aging Mexican Americans compared to Non-Hispanic Whites. Conversely, after controlling for estimated IQ, the influence of education level is weaker for Mexican Americans than Non-Hispanic Whites. These findings emphasize the need for careful evaluation of confrontational naming task scores in diverse ethnic groups, emphasizing the critical role of education and estimated IQ in understanding performance disparities.

Defining aphasia: Content analysis of six aphasia diagnostic batteries

Clear operational definitions of constructs are necessary to ensure that research findings are meaningful and interpretable. In the field of aphasiology, aphasia is often defined to the effect of "aphasia is an acquired language disorder often due to brain injury that affects expressive and receptive language." To contribute to our understanding of the construct of aphasia, we conducted a content analysis of six diagnostic aphasia tests: the Minnesota Test for the Differential Diagnosis of Aphasia, the Porch Index of Communicative Ability, the Boston Diagnostic Aphasia Examination, the Western Aphasia Battery, the Comprehensive Aphasia Test, and the Quick Aphasia Battery. These chosen tests have historical prominence, with several in regular clinical and research use today. We hypothesized that the content of the aphasia tests should be very similar since they all purport to identify and characterize (if present) aphasia, with recognition that there may be some subtle differences in test content stemming in large part to epistemological differences in the test makers' views of aphasia. Instead, we found predominantly weak Jaccard indices, a similarity correlation coefficient, between test targets. Only five test targets were found in all six aphasia tests: auditory comprehension of words and sentences, repetition of words, confrontation naming of nouns, and reading comprehension of words. The qualitative and quantitative results suggest that the content across aphasia tests may be more disparate than expected. We conclude by discussing the implication of our results for the field, including the importance of updating, if necessary, the operational definition of aphasia through conversation with a broad audience of interested and affected people.

Item Response Theory Modeling of the Verb Naming Test

PURPOSE

Item response theory (IRT) is a modern psychometric framework with several advantageous properties as compared with classical test theory. IRT has been successfully used to model performance on anomia tests in individuals with aphasia; however, all efforts to date have focused on noun production accuracy. The purpose of this study is to evaluate whether the Verb Naming Test (VNT), a prominent test of action naming, can be successfully modeled under IRT and evaluate its reliability.

METHOD

We used responses on the VNT from 107 individuals with chronic aphasia from AphasiaBank. Unidimensionality and local independence, two assumptions prerequisite to IRT modeling, were evaluated using factor analysis and Yen's Q 3 statistic (Yen, 1984), respectively. The assumption of equal discrimination among test items was evaluated statistically via nested model comparisons and practically by using correlations of resulting IRT-derived scores. Finally, internal consistency, marginal and empirical reliability, and conditional reliability were evaluated.

RESULTS

The VNT was found to be sufficiently unidimensional with the majority of item pairs demonstrating adequate local independence. An IRT model in which item discriminations are constrained to be equal demonstrated fit equivalent to a model in which unique discrimination parameters were estimated for each item. All forms of reliability were strong across the majority of IRT ability estimates.

CONCLUSIONS

Modeling the VNT using IRT is feasible, yielding ability estimates that are both informative and reliable. Future efforts are needed to quantify the validity of the VNT under IRT and determine the extent to which it measures the same construct as other anomia tests.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.22329235.

ParAlg: A Paraphasia Algorithm for Multinomial Classification of Picture Naming Errors

PURPOSE

A preliminary version of a paraphasia classification algorithm (henceforth called ParAlg) has previously been shown to be a viable method for coding picture naming errors. The purpose of this study is to present an updated version of ParAlg, which uses multinomial classification, and comprehensively evaluate its performance when using two different forms of transcribed input.

METHOD

A subset of 11,999 archival responses produced on the Philadelphia Naming Test were classified into six cardinal paraphasia types using ParAlg under two transcription configurations: (a) using phonemic transcriptions for responses exclusively (phonemic-only) and (b) using phonemic transcriptions for nonlexical responses and orthographic transcriptions for lexical responses (orthographic-lexical). Agreement was quantified by comparing ParAlg-generated paraphasia codes between configurations and relative to human-annotated codes using four metrics (positive predictive value, sensitivity, specificity, and F1 score). An item-level qualitative analysis of misclassifications under the best performing configuration was also completed to identify the source and nature of coding discrepancies.

RESULTS

Agreement between ParAlg-generated and human-annotated codes was high, although the orthographic-lexical configuration outperformed phonemic-only (weighted-average F1 scores of .78 and .87, respectively). A qualitative analysis of the orthographic-lexical configuration revealed a mix of human- and ParAlg-related misclassifications, the former of which were related primarily to phonological similarity judgments whereas the latter were due to semantic similarity assignment.

CONCLUSIONS

ParAlg is an accurate and efficient alternative to manual scoring of paraphasias, particularly when lexical responses are orthographically transcribed. With further development, it has the potential to be a useful software application for anomia assessment.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.22087763.

The Severity-Calibrated Aphasia Naming Test

PURPOSE

We present a 20-item naming test, the Severity-Calibrated Aphasia Naming Test (SCANT), that can serve as a proxy measure for an aphasia severity scale that is derived from a thorough test battery of connected speech production, single-word production, speech repetition, and auditory verbal comprehension.

METHOD

We use lasso regression and cross-validation to identify an optimal subset from a set of 174 pictures to be named for prediction of aphasia severity, based on data from 200 participants with left-hemisphere stroke who were quasirandomly selected to represent the full impairment scale. Data from 20 healthy controls (i.e., participant caretakers/spouses) were also analyzed. We examine interrater reliability, test-retest reliability, sensitivity and specificity to the presence of aphasia, sensitivity to therapy gains, and external validity (i.e., correlation with aphasia severity measures) for the SCANT.

RESULTS

The SCANT has extremely high interrater reliability, and it is sensitive and specific to the presence of aphasia. We demonstrate the superiority of predictions based on the SCANT over those based on the full set of naming items. We estimate a 15% reduction in power when using the SCANT score versus the full test battery's aphasia severity score as an outcome measure; for example, to maintain the same power to detect a significant group average change in aphasia severity, a study with 25 participants using the full test battery to measure treatment effectiveness would require 30 participants if the SCANT were to be used as the testing instrument instead.

CONCLUSION

We provide a linear model to convert SCANT scores to aphasia severity scores, and we identify a change score cutoff of four SCANT items to obtain a high degree of confidence based on test-retest SCANT data and the modeled relation between SCANT and aphasia severity scores.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.21476871.