An Item Response Theory-Based, Computerized Adaptive Testing Version of the MacArthur-Bates Communicative Development Inventory: Words & Sentences (CDI:WS)

Should We Stop Using Lexical Diversity Measures in Children's Language Sample Analysis?

PURPOSE

Prior work has identified weaknesses in commonly used indices of lexical diversity in spoken language samples, such as type-token ratio (TTR) due to sample size and elicitation variation, we explored whether TTR and other diversity measures, such as number of different words/100 (NDW), vocabulary diversity (VocD), and the moving average TTR would be more sensitive to child age and clinical status (typically developing [TD] or developmental language disorder [DLD]) if samples were obtained from standardized prompts.

METHOD

We utilized archival data from the norming samples of the Test of Narrative Language and the Edmonton Narrative Norms Instrument. We examined lexical diversity and other linguistic properties of the samples, from a total of 1,048 children, ages 4-11 years; 798 of these were considered TD, whereas 250 were categorized as having a language learning disorder.

RESULTS

TTR was the least sensitive to child age or diagnostic group, with good potential to misidentify children with DLD as TD and TD children as having DLD. Growth slopes of NDW were shallow and not very sensitive to diagnostic grouping. The strongest performing measure was VocD. Mean length of utterance, TNW, and verbs/utterance did show both good growth trajectories and ability to distinguish between clinical and typical samples.

CONCLUSIONS

This study, the largest and best controlled to date, re-affirms that TTR should not be used in clinical decision making with children. A second popular measure, NDW, is not measurably stronger in terms of its psychometric properties. Because the most sensitive measure of lexical diversity, VocD, is unlikely to gain popularity because of reliance on computer-assisted analysis, we suggest alternatives for the appraisal of children's expressive vocabulary skill.

oREV: An item response theory-based open receptive vocabulary task for 3- to 8-year-old children

Individual differences in early language abilities are an important predictor of later life outcomes. High-quality, easy-access measures of language abilities are rare, especially in the preschool and primary school years. The present study describes the construction of a new receptive vocabulary task for children between 3 and 8 years of age. The task was implemented as a browser-based web application, allowing for both in-person and remote data collection via the internet. Based on data from N = 581 German-speaking children, we estimated the psychometric properties of each item in a larger initial item pool via item response modeling. We then applied an automated item selection procedure to select an optimal subset of items based on item difficulty and discrimination. The so-constructed task has 22 items and shows excellent psychometric properties with respect to reliability, stability, and convergent and discriminant validity. The construction, implementation, and item selection process described here makes it easy to extend the task or adapt it to different languages. All materials and code are freely accessible to interested researchers. The task can be used via the following website: https://ccp-odc.eva.mpg.de/orev-demo .

Measurement of Lexical Diversity in Children's Spoken Language: Computational and Conceptual Considerations

Background

Type-Token Ratio (TTR), given its relatively simple hand computation, is one of the few LSA measures calculated by clinicians in everyday practice. However, it has significant well-documented shortcomings; these include instability as a function of sample size, and absence of clear developmental profiles over early childhood. A variety of alternative measures of lexical diversity have been proposed; some, such as Number of Different Words/100 (NDW) can also be computed by hand. However, others, such as Vocabulary Diversity (VocD) and the Moving Average Type Token Ratio (MATTR) rely on complex resampling algorithms that cannot be conducted by hand. To date, no large-scale study of all four measures has evaluated how well any capture typical developmental trends over early childhood, or whether any reliably distinguish typical from atypical profiles of expressive child language ability.

Materials and Methods

We conducted linear and non-linear regression analyses for TTR, NDW, VocD, and MATTR scores for samples taken from 946 corpora from typically developing preschool children (ages 2-6 years), engaged in adult-child toy play, from the Child Language Data Exchange System (CHILDES). These were contrasted with 504 samples from children known to have delayed expressive language skills (total n = 1,454 samples). We also conducted a separate sub-analysis which examined possible contextual effects of sampling environment on lexical diversity.

Results

Only VocD showed significantly different mean scores between the typically -developing children and delayed developing children group. Using TTR would actually misdiagnose typical children and miss children with known language impairment. However, computation of VocD as a function of toy interactions was significant and emerges as a further caution in use of lexical diversity as a valid proxy index of children's expressive vocabulary skill.

Discussion

This large scale statistical comparison of computer-implemented algorithms for expressive lexical profiles in young children with traditional, hand-calculated measures showed that only VocD met criteria for evidence-based use in LSA. However, VocD was impacted by sample elicitation context, suggesting that non-linguistic factors, such as engagement with elicitation props, contaminate estimates of spoken lexical skill in young children. Implications and suggested directions are discussed.

Online Computerized Adaptive Tests of Children's Vocabulary Development in English and Mexican Spanish

PURPOSE

Measuring the growth of young children's vocabulary is important for researchers seeking to understand language learning as well as for clinicians aiming to identify early deficits. The MacArthur-Bates Communicative Development Inventories (CDIs) are parent report instruments that offer a reliable and valid method for measuring early productive and receptive vocabulary across a number of languages. CDI forms typically include hundreds of words, however, and so the burden of completion is significant. We address this limitation by building on previous work using item response theory (IRT) models to create computer adaptive test (CAT) versions of the CDIs. We created CDI-CATs for both comprehension and production vocabulary, for both American English and Mexican Spanish.

METHOD

Using a data set of 7,633 English-speaking children ages 12-36 months and 1,692 Spanish-speaking children ages 12-30 months, across three CDI forms (Words & Gestures, Words & Sentences, and CDI-III), we found that a 2-parameter logistic IRT model fits well for a majority of the 680 pooled vocabulary items. We conducted CAT simulations on this data set, assessing simulated tests of varying length (25-400 items).

RESULTS

Even very short CATs recovered participant abilities very well with little bias across ages. An empirical validation study with N = 204 children ages 15-36 months showed a correlation of r = .92 between language ability estimated from full CDI versus CDI-CAT forms.

CONCLUSION

We provide our item bank along with fitted parameters and other details, offer recommendations for how to construct CDI-CATs in new languages, and suggest when this type of assessment may or may not be appropriate.

Parental Report via a Mobile App in the Context of Early Language Trajectories: StarWords Study Protocol

Social sciences researchers emphasize that new technologies can overcome the limitations of small and homogenous samples. In research on early language development, which often uses parental reports, taking the testing online might be particularly compelling. Due to logistical limitations, previous studies on bilingual children have explored the language development trajectories in general (e.g., by including few and largely set apart timepoints), or focused on small, homogeneous samples. The present study protocol presents a new, on-going study which uses new technologies to collect longitudinal data continuously from parents of multilingual, bilingual, and monolingual children. Our primary aim is to establish the developmental trajectories in Polish-British English and Polish-Norwegian bilingual children and Polish monolingual children aged 0-3 years with the use of mobile and web-based applications. These tools allow parents to report their children's language development as it progresses, and allow us to characterize children's performance in each language (the age of reaching particular language milestones). The project's novelty rests on its use of mobile technologies to characterize the bilingual and monolingual developmental trajectory from the very first words to broader vocabulary and multiword combinations.

Computer Adaptive Testing for the Assessment of Anomia Severity

Anomia assessment is a fundamental component of clinical practice and research inquiries involving individuals with aphasia, and confrontation naming tasks are among the most commonly used tools for quantifying anomia severity. While currently available confrontation naming tests possess many ideal properties, they are ultimately limited by the overarching psychometric framework they were developed within. Here, we discuss the challenges inherent to confrontation naming tests and present a modern alternative to test development called item response theory (IRT). Key concepts of IRT approaches are reviewed in relation to their relevance to aphasiology, highlighting the ability of IRT to create flexible and efficient tests that yield precise measurements of anomia severity. Empirical evidence from our research group on the application of IRT methods to a commonly used confrontation naming test is discussed, along with future avenues for test development.

A shortened version of Raven's standard progressive matrices for children and adolescents

Numerous developmental studies assess general cognitive ability, not as the primary variable of interest, but rather as a background variable. Raven's Progressive Matrices is an easy to administer non-verbal test that is widely used to measure general cognitive ability. However, the relatively long administration time (up to 45 min) is still a drawback for developmental studies as it often leaves little time to assess the primary variable of interest. Therefore, we used a machine learning approach - regularized regression in combination with cross-validation - to develop a short 15-item version. We did so for two age groups, namely 9 to 12 years and 13 to 16 years. The short versions predicted the scores on the standard full 60-item versions to a very high degree r = 0.89 (9-12 years) and r = 0.93 (13-16 years). We, therefore, recommend using the short version to measure general cognitive ability as a background variable in developmental studies.

A Bayesian-Inspired Item Response Theory-Based Framework to Produce Very Short Versions of MacArthur-Bates Communicative Development Inventories

Purpose This study introduces a framework to produce very short versions of the MacArthur-Bates Communicative Development Inventories (CDIs) by combining the Bayesian-inspired approach introduced by Mayor and Mani (2019) with an item response theory-based computerized adaptive testing that adapts to the ability of each child, in line with Makransky et al. (2016). Method We evaluated the performance of our approach-dynamically selecting maximally informative words from the CDI and combining parental response with prior vocabulary data-by conducting real-data simulations using four CDI versions having varying sample sizes on Wordbank-the online repository of digitalized CDIs: American English (a very large data set), Danish (a large data set), Beijing Mandarin (a medium-sized data set), and Italian (a small data set). Results Real-data simulations revealed that correlations exceeding .95 with full CDI administrations were reached with as few as 15 test items, with high levels of reliability, even when languages (e.g., Italian) possessed few digitalized administrations on Wordbank. Conclusions The current approach establishes a generic framework that produces very short (less than 20 items) adaptive early vocabulary assessments-hence considerably reducing their administration time. This approach appears to be robust even when CDIs have smaller samples in online repositories, for example, with around 50 samples per month-age.

A short version of the MacArthur-Bates Communicative Development Inventories with high validity

The MacArthur-Bates Communicative Development Inventories (CDIs) are among the most widely used evaluation tools for early language development. CDIs are filled in by the parents or caregivers of young children by indicating which of a prespecified list of words and/or sentences their child understands and/or produces. Despite the success of these instruments, their administration is time-consuming and can be of limited use in clinical settings, multilingual environments, or when parents possess low literacy skills. We present a new method through which an estimation of the full-CDI score can be obtained, by combining parental responses on a limited set of words sampled randomly from the full CDI with vocabulary information extracted from the WordBank database, sampled from age-, gender-, and language-matched participants. Real-data simulations using versions of the CDI-WS for American English, German, and Norwegian as examples revealed the high validity and reliability of the instrument, even for tests having just 25 words, effectively cutting administration time to a couple of minutes. Empirical validations with new German-speaking participants confirmed the robustness of the test.

Empirical Evaluation of Computer-Adaptive Alternate Short Forms for the Assessment of Anomia Severity

Purpose The purpose of this study was to verify the equivalence of 2 alternate test forms with nonoverlapping content generated by an item response theory (IRT)-based computer-adaptive test (CAT). The Philadelphia Naming Test (PNT; Roach, Schwartz, Martin, Grewal, & Brecher, 1996)was utilized as an item bank in a prospective, independent sample of persons with aphasia. Method Two alternate CAT short forms of the PNT were administered to a sample of 25 persons with aphasia who were at least 6 months postonset and received no treatment for 2 weeks before or during the study. The 1st session included administration of a 30-item PNT-CAT, and the 2nd session, conducted approximately 2 weeks later, included a variable-length PNT-CAT that excluded items administered in the 1st session and terminated when the modeled precision of the ability estimate was equal to or greater than the value obtained in the 1st session. The ability estimates were analyzed in a Bayesian framework. Results The 2 test versions correlated highly (r = .89) and obtained means and standard deviations that were not credibly different from one another. The correlation and error variance between the 2 test versions were well predicted by the IRT measurement model. Discussion The results suggest that IRT-based CAT alternate forms may be productively used in the assessment of anomia. IRT methods offer advantages for the efficient and sensitive measurement of change over time. Future work should consider the potential impact of differential item functioning due to person factors and intervention-specific effects, as well as expanding the item bank to maximize the clinical utility of the test. Supplemental Material https://doi.org/10.23641/asha.11368040.

Post hemorrhagic hydrocephalus and neurodevelopmental outcomes in a context of neonatal intraventricular hemorrhage: an institutional experience in 122 preterm children

BACKGROUND

Intraventricular hemorrhage (IVH) is a frequent complication in extreme and very preterm births. Despite a high risk of death and impaired neurodevelopment, the precise prognosis of infants with IVH remains unclear. The objective of this study was to evaluate the rate and predictive factors of evolution to post hemorrhagic hydrocephalus (PHH) requiring a shunt, in newborns with IVH and to report their neurodevelopmental outcomes at 2 years of age.

METHODS

Among all preterm newborns admitted to the department of neonatalogy at Rouen University Hospital, France between January 2000 and December 2013, 122 had an IVH and were included in the study. Newborns with grade 1 IVH according to the Papile classification were excluded.

RESULTS

At 2-year, 18% (n = 22) of our IVH cohort required permanent cerebro spinal fluid (CSF) derivation. High IVH grade, low gestational age at birth and increased head circumference were risk factors for PHH. The rate of death of IVH was 36.9% (n = 45). The rate of cerebral palsy was 55.9% (n = 43) in the 77 surviving patients (49.4%). Risk factors for impaired neurodevelopment were high grade IVH and increased head circumference.

CONCLUSION

High IVH grade was strongly correlated with death and neurodevelopmental outcome. The impact of an increased head circumference highlights the need for early management. CSF biomarkers and new medical treatments such as antenatal magnesium sulfate have emerged and could predict and improve the prognosis of these newborns with PHH.

Accurate Autism Screening at the 18-Month Well-Child Visit Requires Different Strategies than at 24 Months

Accuracy of autism screening using M-CHAT plus the follow-up interview (M-CHAT/F) for children screened positive at 18-months was compared to screening at 24-months. Formal ASD testing was criterion for a community sample of M-CHAT positive children (n = 98), positive predictive value (PPV) was 0.40 for the M-CHAT and 0.58 for the M-CHAT/F. MCHAT/F PPV was 0.69 among children 20+ months compared to 0.36 for <20 months. Multivariate analyses incorporating data from the Ages and Stages Questionnaire, MacArthur-Bates Communicative Development Inventory, M-CHAT and M-CHAT/F results, and M-CHAT items suggest language variables carry greatest relative importance in contributing to an age-based algorithm with potential to improve PPV for toddlers <20 months to the same level as observed in older toddlers.

Applying Item Response Theory to the Development of a Screening Adaptation of the Goldman-Fristoe Test of Articulation-Second Edition

PURPOSE

Item response theory (IRT) is a psychometric approach to measurement that uses latent trait abilities (e.g., speech sound production skills) to model performance on individual items that vary by difficulty and discrimination. An IRT analysis was applied to preschoolers' productions of the words on the Goldman-Fristoe Test of Articulation-Second Edition (GFTA-2) to identify candidates for a screening measure of speech sound production skills.

METHOD

The phoneme accuracies from 154 preschoolers, with speech skills on the GFTA-2 ranging from the 1st to above the 90th percentile, were analyzed with a 2-parameter logistic model.

RESULTS

A total of 108 of the 232 phonemes from stimuli in the sounds-in-words subtest fit the IRT model. These phonemes, and subgroups of the most difficult of these phonemes, correlated significantly with the children's overall percentile scores on the GFTA-2. Regression equations calculated for the 5 and 10 most difficult phonemes predicted overall percentile score at levels commensurate with other screening measures.

CONCLUSIONS

These results suggest that speech production accuracy can be screened effectively with a small number of sounds. They motivate further research toward the development of a screening measure of children's speech sound production skills whose stimuli consist of a limited number of difficult phonemes.