Rasch modeling of revised token test performance: validity and sensitivity to change

Diagnosis of aphasia in stroke populations: A systematic review of language tests

Background and purpose

Accurate aphasia diagnosis is important in stroke care. A wide range of language tests are available and include informal assessments, tests developed by healthcare institutions and commercially published tests available for purchase in pre-packaged kits. The psychometrics of these tests are often reported online or within the purchased test manuals, not the peer-reviewed literature, therefore the diagnostic capabilities of these measures have not been systematically evaluated. This review aimed to identify both commercial and non-commercial language tests and tests used in stroke care and to examine the diagnostic capabilities of all identified measures in diagnosing aphasia in stroke populations.

Methods

Language tests were identified through a systematic search of 161 publisher databases, professional and resource websites and language tests reported to be used in stroke care. Two independent reviewers evaluated test manuals or associated resources for cohort or cross-sectional studies reporting the tests’ diagnostic capabilities (sensitivity, specificity, likelihood ratios or diagnostic odds ratios) in differentiating aphasic and non-aphasic stroke populations.

Results

Fifty-six tests met the study eligibility criteria. Six “non-specialist” brief screening tests reported sensitivity and specificity information, however none of these measures reported to meet the specific diagnostic needs of speech pathologists. The 50 remaining measures either did not report validity data (n = 7); did not compare patient test performance with a comparison group (n = 17); included non-stroke participants within their samples (n = 23) or did not compare stroke patient performance against a language reference standard (n = 3). Diagnostic sensitivity analysis was completed for six speech pathology measures (WAB, PICA, CADL-2, ASHA-FACS, Adult FAVRES and EFA-4), however all studies compared aphasic performance with that of non-stroke healthy controls and were consequently excluded from the review.

Conclusions

No speech pathology test was found which reported diagnostic data for identifying aphasia in stroke populations. A diagnostically validated post-stroke aphasia test is needed.

Applying Item Response Theory to the Development of a Screening Adaptation of the Goldman-Fristoe Test of Articulation-Second Edition

PURPOSE

Item response theory (IRT) is a psychometric approach to measurement that uses latent trait abilities (e.g., speech sound production skills) to model performance on individual items that vary by difficulty and discrimination. An IRT analysis was applied to preschoolers' productions of the words on the Goldman-Fristoe Test of Articulation-Second Edition (GFTA-2) to identify candidates for a screening measure of speech sound production skills.

METHOD

The phoneme accuracies from 154 preschoolers, with speech skills on the GFTA-2 ranging from the 1st to above the 90th percentile, were analyzed with a 2-parameter logistic model.

RESULTS

A total of 108 of the 232 phonemes from stimuli in the sounds-in-words subtest fit the IRT model. These phonemes, and subgroups of the most difficult of these phonemes, correlated significantly with the children's overall percentile scores on the GFTA-2. Regression equations calculated for the 5 and 10 most difficult phonemes predicted overall percentile score at levels commensurate with other screening measures.

CONCLUSIONS

These results suggest that speech production accuracy can be screened effectively with a small number of sounds. They motivate further research toward the development of a screening measure of children's speech sound production skills whose stimuli consist of a limited number of difficult phonemes.

Development and Simulation Testing of a Computerized Adaptive Version of the Philadelphia Naming Test

PURPOSE

The purpose of this study was to develop a computerized adaptive test (CAT) version of the Philadelphia Naming Test (PNT; Roach, Schwartz, Martin, Grewal, & Brecher, 1996), to reduce test length while maximizing measurement precision. This article is a direct extension of a companion article (Fergadiotis, Kellough, & Hula, 2015), in which we fitted the PNT to a 1-parameter logistic item-response-theory model and examined the validity and precision of the resulting item parameter and ability score estimates.

METHOD

Using archival data collected from participants with aphasia, we simulated two PNT-CAT versions and two previously published static PNT short forms, and compared the resulting ability score estimates to estimates obtained from the full 175-item PNT. We used a jackknife procedure to maintain independence of the samples used for item estimation and CAT simulation.

RESULTS

The PNT-CAT recovered full PNT scores with equal or better accuracy than the static short forms. Measurement precision was also greater for the PNT-CAT than the static short forms, though comparison of adaptive and static nonoverlapping alternate forms showed minimal differences between the two approaches.

CONCLUSION

These results suggest that CAT assessment of naming in aphasia has the potential to reduce test burden while maximizing the accuracy and precision of score estimates.

A tutorial on aphasia test development in any language: Key substantive and psychometric considerations

BACKGROUND

There are a limited number of aphasia language tests in the majority of the world's commonly spoken languages. Furthermore, few aphasia tests in languages other than English have been standardized and normed, and few have supportive psychometric data pertaining to reliability and validity. The lack of standardized assessment tools across many of the world's languages poses serious challenges to clinical practice and research in aphasia.

AIMS

The current review addresses this lack of assessment tools by providing conceptual and statistical guidance for the development of aphasia assessment tools and establishment of their psychometric properties.

MAIN CONTRIBUTION

A list of aphasia tests in the 20 most widely spoken languages is included. The pitfalls of translating an existing test into a new language versus creating a new test are outlined. Factors to consider in determining test content are discussed. Further, a description of test items corresponding to different language functions is provided, with special emphasis on implementing important controls in test design. Next, a broad review of principal psychometric properties relevant to aphasia tests is presented, with specific statistical guidance for establishing psychometric properties of standardized assessment tools.

CONCLUSIONS

This article may be used to help guide future work on developing, standardizing and validating aphasia language tests. The considerations discussed are also applicable to the development of standardized tests of other cognitive functions.

Evaluating language comprehension in Alzheimer's disease: the use of the Token test

OBJECTIVE

To analyze the psychometric properties of the Token test (TT), a verbal comprehension test, and its applicability to the diagnosis of mild Alzheimer's disease (AD).

METHODS

One hundred and sixty participants (80 AD and 80 controls) performed the TT and a short battery of neuropsychological tests designed to evaluate general cognitive status, working memory and executive functions. Internal consistency, factor structure, correlation with other measures and group comparisons were evaluated.

RESULTS

The test evinced good internal consistency and a bi-factorial structure (related to comprehension and attention). Differences between AD and controls were significant, however the TT presented only moderate sensitivity and specificity for the AD diagnosis.

CONCLUSION

The TT showed evidence of good psychometric properties and adequacy for characterizing comprehension deficits in AD, but it was not an appropriate test for the AD detection and diagnosis.

Model choice and sample size in item response theory analysis of aphasia tests

PURPOSE

The purpose of this study was to identify the most appropriate item response theory (IRT) measurement model for aphasia tests requiring 2-choice responses and to determine whether small samples are adequate for estimating such models.

METHOD

Pyramids and Palm Trees (Howard & Patterson, 1992) test data that had been collected from individuals with aphasia were analyzed, and the resulting item and person estimates were used to develop simulated test data for 3 sample size conditions. The simulated data were analyzed using a standard 1-parameter logistic (1-PL) model and 3 models that accounted for the influence of guessing: augmented 1-PL and 2-PL models and a 3-PL model. The model estimates obtained from the simulated data were compared to their known true values.

RESULTS

With small and medium sample sizes, an augmented 1-PL model was the most accurate at recovering the known item and person parameters; however, no model performed well at any sample size. Follow-up simulations confirmed that the large influence of guessing and the extreme easiness of the items contributed substantially to the poor estimation of item difficulty and person ability.

CONCLUSION

Incorporating the assumption of guessing into IRT models improves parameter estimation accuracy, even for small samples. However, caution should be exercised in interpreting scores obtained from easy 2-choice tests, regardless of whether IRT modeling or percentage correct scoring is used.

Item-level psychometrics and predictors of performance for Spanish/English bilingual speakers on an object and action naming battery

PURPOSE

There is a pressing need for psychometrically sound naming materials for Spanish/English bilingual adults. To address this need, in this study the authors examined the psychometric properties of An Object and Action Naming Battery (An O&A Battery; Druks & Masterson, 2000) in bilingual speakers.

METHOD

Ninety-one Spanish/English bilinguals named O&A Battery items in English and Spanish. Responses underwent a Rasch analysis. Using correlation and regression analyses, the authors evaluated the effect of psycholinguistic (e.g., imageability) and participant (e.g., proficiency ratings) variables on accuracy.

RESULTS

Rasch analysis determined unidimensionality across English and Spanish nouns and verbs and robust item-level psychometric properties, evidence for content validity. Few items did not fit the model, there were no ceiling or floor effects after uninformative and misfit items were removed, and items reflected a range of difficulty. Reliability coefficients were high, and the number of statistically different ability levels provided indices of sensitivity. Regression analyses revealed significant correlations between psycholinguistic variables and accuracy, providing preliminary construct validity. The participant variables that contributed most to accuracy were proficiency ratings and time of language use.

CONCLUSIONS

Results suggest adequate content and construct validity of O&A items retained in the analysis for Spanish/English bilingual adults and support future efforts to evaluate naming in older bilinguals and persons with bilingual aphasia.

Psychometric data for the revised token test in normally developing Mexican children ages 4-12 years

Language comprehension is vital to social and educational development but few pediatric tests are available for its assessment. To approach this problem, two versions of the Token Test (TT), "TT short form" (DeRenzi & Faglioni, 1978) and "Revised Token Test" (RTT), were first compared. Using a sample of 88 normally developing Spanish-speaking children, the tests were compared on their: (a) established psychometric development and (b) internal consistency. The RTT was judged to be superior and was selected for additional experimentation. The RTT was compared with a developmental measure of lexical knowledge on a cross-sectional sample of 250 4-12-year-old normally developing Spanish-speaking children. A significant positive and high correlation supports its concurrent validity. Significant differences across the age groups, along with a principal component analysis that yielded a three-factor structure, support its construct validity. Preliminary normative data across the nine age groups are provided.

Rasch models of aphasic performance on syntactic comprehension tests

Responses of 42 people with aphasia to 11 sentence types in enactment and sentence-picture matching tasks were characterized using Rasch models that varied in the inclusion of the factors of task, sentence type, and patient group. The best fitting models required the factors of task and patient group but not sentence type. The results provide evidence that aphasic syntactic comprehension is best accounted for by models that include different estimates of patient ability in different tasks and different difficulty of all sentences in different groups of patients, but that do not include different estimates of patient ability for different types of sentences.

Developing the communicative participation item bank: Rasch analysis results from a spasmodic dysphonia sample

PURPOSE

The purpose of this study was to conduct the initial psychometric analyses of the Communicative Participation Item Bank-a new self-report instrument designed to measure the extent to which communication disorders interfere with communicative participation. This item bank is intended for community-dwelling adults across a range of communication disorders.

METHOD

A set of 141 candidate items was administered to 208 adults with spasmodic dysphonia. Participants rated the extent to which their condition interfered with participation in various speaking communication situations. Questionnaires were administered online or in a paper version per participant preference. Participants also completed the Voice Handicap Index (B. H. Jacobson et al., 1997) and a demographic questionnaire. Rasch analyses were conducted using Winsteps software (J. M. Linacre, 1991).

RESULTS

The results show that items functioned better when the 5-category response format was recoded to a 4-category format. After removing 8 items that did not fit the Rasch model, the remaining 133 items demonstrated strong evidence of sufficient unidimensionality, with the model accounting for 89.3% of variance. Item location values ranged from -2.73 to 2.20 logits.

CONCLUSIONS

Preliminary Rasch analyses of the Communicative Participation Item Bank show strong psychometric properties. Further testing in populations with other communication disorders is needed.

A psychometric analysis of functional category production in English agrammatic narratives

Hierarchical models of agrammatism propose that sentence production deficits can be accounted for in terms of clausal syntactic structure [Friedmann, N., & Grodzinsky, Y. (1997). Tense and agreement in agrammatic production: Pruning the syntactic tree. Brain and Language, 56, 397-425; Hagiwara, H. (1995). The breakdown of functional categories and the economy of derivation. Brain and Language, 50, 92-116]. Such theories predict that morpho-syntactic elements associated with higher nodes in the syntactic tree (complementizers and verb inflections) will be more impaired than elements associated with lower structural positions (negation markers and aspectual verb forms). While this hypothesis has been supported by the results of several studies [Benedet, M. J., Christiansen, J. A., & Goodglass, H. (1998). A cross-linguistic study of grammatical morphology in Spanish- and English-speaking agrammatic patients. Cortex, 34, 309-336; Friedmann, N. (2001). Agrammatism and the psychological reality of the syntactic tree. Journal of Psycholinguistic Research, 30, 71-88; Friedmann, N. (2002). Question production in agrammatism: The tree pruning hypothesis. Brain and Language, 80, 160-187], it has also been challenged on several grounds [Burchert, F., Swoboda-Moll, M., & De Bleser, R. (2005a). Tense and agreement dissociations in German agrammatic speakers: Underspecification vs. hierarchy. Brain and Language, 94, 188-199; Lee, M. (2003). Dissociations among functional categories in Korean agrammatism. Brain and Language, 84, 170-188; Lee, J., Milman, L. H., & Thompson, C. K. (2005). Functional category production in agrammatic speech. Brain and Language, 95, 123-124]. In this paper the question of hierarchical structure was re-examined within the framework of Item Response Theory [IRT, Rasch, G. (1980). Probabilistic models for some intelligence and attainment tests (Expanded ed.). Chicago: University of Chicago Press]. IRT is a probabilistic model widely used in the field of psychometrics to model behavioral constructs as numeric variables. In this study we examined production of functional categories (complementizers, verb inflections, negation markers, and aspectual verb forms) in narrative samples elicited from 18 individuals diagnosed with nonfluent aphasia and 18 matched controls. Data from the aphasic participants were entered into an IRT analysis to test (1) whether production of clausal functional categories can be represented as a variable on a numeric scale; and (2) whether production patterns were consistent with hierarchical syntactic structure. Pearson r correlation coefficients were also computed to determine whether there was a relation between functional category production and other indices of language performance. Results indicate that functional category production can be modeled as a numeric variable using IRT. Furthermore, although variability was observed across individuals, consistent patterns were evident when the data were interpreted within a probabilistic framework. Although functional category production was moderately correlated with a second measure of clausal structure (clause length), it was not correlated with more distant language constructs (noun/verb ratio and WAB A.Q.). These results suggest that functional category production is related to some, but not all, measures of agrammatic language performance.

Initial validity and reliability of the SCCAN: using tailored testing to assess adult cognition and communication

PURPOSE

The Scales of Cognitive and Communicative Ability for Neurorehabilitation (SCCAN; L. Milman & A. Holland, 2007) was developed in the hospital setting to address changes in assessment practice. The SCCAN was designed to provide an overview of impairment and activity limitations across 8 cognitive scales (Speech Comprehension, Oral Expression, Reading, Writing, Orientation, Attention, Memory, and Problem Solving). The scales were developed using item response theory so that tailored testing could be implemented to reduce test administration time. This research investigated the validity and reliability of the SCCAN.

METHOD

A battery of neuropsychological tests was administered to 40 neurologically healthy control participants and 51 participants diagnosed with left-hemisphere pathology, right-hemisphere pathology, or probable Alzheimer's disease. Analyses were performed to assess test sensitivity and specificity, construct validity, administration time, and reliability.

RESULTS

The test accurately classified 95% of the control participants and 98% of the participants diagnosed with neurological disorders. Results indicate that the test also differentiated the performance profiles of the 3 clinical populations. In addition, test scores correlated significantly with external measures of the same cognitive areas. Mean administration time was 34 min. Test-retest stability (r = .96, p < .001) and internal consistency (r = .99, p < .001) coefficients were both significant, indicating that tailored testing procedures generated reliable test scores.