Machine-Scored Syntax: Comparison of the CLAN Automatic Scoring Program to Manual Scoring

How to Use the Index of Productive Syntax to Select Goals and Monitor Progress in Preschool Children

PURPOSE

Syntax assessment and treatment is a major focus of speech-language pathologists who work with young children with language needs. The Index of Productive Syntax (IPSyn) provides an efficient method for the assessment of syntax of preschool-age children. The purpose of this tutorial is to illustrate how to hand-score an IPSyn from language sample transcripts and use this information to formulate appropriate syntactic goals.

METHOD

We describe general methods and tips for facilitating hand-scoring. Two case study transcripts of a late talker at 36 and 48 months are provided. One case study transcript is fully scored in IPSyn with detailed explanations; the second case study is provided as a practice example with an IPSyn answer key. Relevant sample probes are provided to assist in the goal of obtaining a representative set of language structures.

RESULTS

We show how the IPSyn can be scored, subscale by subscale, and can highlight developmental stages of syntax following Brown's stages of grammatical development. We illustrate how the visual snapshot developmental nature of the instrument makes it ideal for deriving goals.

CONCLUSION

The IPSyn is a practical and valuable tool for the assessment of syntax of preschool-age children.

The Index of Productive Syntax: Psychometric Properties and Suggested Modifications

PURPOSE

The Index of Productive Syntax (IPSyn) is a well-known language sample analysis tool. However, its psychometric properties have not been assessed across a wide sample of typically developing preschool-age children and children with language disorders. We sought to determine the profile of IPSyn scores by age over early childhood. We additionally explored whether the IPSyn could be shortened to fewer items without loss of information and whether the required language sample could be shortened from a current required number of 100 utterances to 50.

METHOD

We used transcripts from the Child Language Data Exchange System, including 1,051 samples of adult-child conversational play with toys within the theoretical framework of item response theory. Samples included those from typically developing children as well as children with hearing loss, Down syndrome, and late language emergence.

RESULTS

The Verb Phrase and Sentence Structure subscales showed more stable developmental trajectories over the preschool years and greater differentiation between typical and atypical cohorts than did the Noun Phrase and Question/Negation subscales. A number of current IPSyn scoring items can be dropped without loss of information, and 50-utterance samples demonstrate most of the same psychometric properties of longer samples.

DISCUSSION

Our findings suggest ways in which the IPSyn can be automated and streamlined (proposed IPSyn-C) so as to provide useful clinical guidance with fewer items and a shorter required language sample. Reference values for the IPSyn-C are provided. Trajectories for one subscale (Question/Negation) appear inherently unstable and may require structured elicitation. Potential limitations, ramifications, and future directions are discussed.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.16915690.

Tracking Child Language Development With Neural Network Language Models

Recent work on the application of neural networks to language modeling has shown that models based on certain neural architectures can capture syntactic information from utterances and sentences even when not given an explicitly syntactic objective. We examine whether a fully data-driven model of language development that uses a recurrent neural network encoder for utterances can track how child language utterances change over the course of language development in a way that is comparable to what is achieved using established language assessment metrics that use language-specific information carefully designed by experts. Given only transcripts of child language utterances from the CHILDES Database and no pre-specified information about language, our model captures not just the structural characteristics of child language utterances, but how these structures reflect language development over time. We establish an evaluation methodology with which we can examine how well our model tracks language development compared to three known approaches: Mean Length of Utterance, the Developmental Sentence Score, and the Index of Productive Syntax. We discuss the applicability of our model to data-driven assessment of child language development, including how a fully data-driven approach supports the possibility of increased research in multilingual and cross-lingual issues.

Using Free Computer-Assisted Language Sample Analysis to Evaluate and Set Treatment Goals for Children Who Speak African American English

Purpose Spoken language sample analysis (LSA) is widely considered to be a critical component of assessment for child language disorders. It is our best window into a preschool child's everyday expressive communicative skills. However, historically, the process can be cumbersome, and reference values against which LSA findings can be "benchmarked" are based on surprisingly little data. Moreover, current LSA protocols potentially disadvantage speakers of nonmainstream English varieties, such as African American English (AAE), blurring the line between language difference and disorder. Method We provide a tutorial on the use of free software (Computerized Language Analysis [CLAN]) enabled by the ongoing National Institute on Deafness and Other Communication Disorders-funded "Child Language Assessment Project." CLAN harnesses the advanced computational power of the Child Language Data Exchange System archive (www.childes.talkbank.org), with an aim to develop and test fine-grained and potentially language variety-sensitive benchmarks for a range of LSA measures. Using retrospective analysis of data from AAE-speaking children, we demonstrate how CLAN LSA can facilitate dialect-fair assessment and therapy goal setting. Results Using data originally collected to norm the Diagnostic Evaluation of Language Variation, we suggest that Developmental Sentence Scoring does not appear to bias against children who speak AAE but does identify children who have language impairment (LI). Other LSA measure scores were depressed in the group of AAE-speaking children with LI but did not consistently differentiate individual children as LI. Furthermore, CLAN software permits rapid, in-depth analysis using Developmental Sentence Scoring and the Index of Productive Syntax that can identify potential intervention targets for children with developmental language disorder.

Improving Automatic IPSyn Coding

Purpose The Computerized Language ANalysis-Index of Productive Syntax (CLAN-IPSyn) system is designed to facilitate automatic computation of the IPSyn measure of productive child syntax. Roberts et al. (2020) conducted a thorough comparison of hand-generated and automatic scores on the Index of Productive Syntax (IPSyn) measure (Scarborough, 1990) and found a high level of error for CLAN-IPSyn. We report on the use of the Roberts et al. analysis to reduce and eliminate errors in CLAN-IPSyn, to improve its accuracy. Method Scores provided by manual and machine scoring of the 20 transcripts used in Roberts et al. (2020) were compared. Divergences in point assignment were examined and significant modifications made to the CLAN-IPSyn program to increase its accuracy. Conclusion The currently available, free version of CLAN at https://talkbank.org is now significantly more correct in terms of exemplars produced, and should assist clinicians and researchers in using the revised IPSyn (Altenberg et al. 2018).