Using Computerized Language Analysis to Evaluate Grammatical Skills

Syntactic Growth of Adolescent Boys With Fragile X Syndrome or Down Syndrome: A Longitudinal Study

PURPOSE

The current study addresses a gap in the literature regarding syntactic development of adolescent boys with fragile X syndrome (FXS) and Down syndrome (DS). Specifically, we ask whether syntactic skills plateau or continue to change during adolescence for these groups and whether the profile of syntactic change differs between boys with FXS and those with DS.

METHOD

Participants were 38 boys with FXS (with and without autism) and 20 boys with DS between the ages of 10 and 16 years, as well as 33 boys who were neurotypical between the ages of 3 and 8 years at study entry. Trained examiners evaluated the participants annually for four consecutive years. The evaluation included standardized language assessments and a conversational language sample, which was analyzed using mean length of utterance-morphemes and the Index of Productive Syntax. For each measure, we fit a series of candidate models, including the intercept-only model and models with nonverbal cognition and maternal IQ as moderators. We then used Akaike's information criteria-corrected to determine which model in a candidate set had the most empirical evidence.

RESULTS

Our between-groups results indicated that FXS and DS have distinct syntactic profiles. However, our growth analyses and moderator analyses yielded mixed results. For most measures, the most likely models suggest that there is no plateau in the growth of syntactic skills for boys with FXS or DS and that nonverbal cognition is associated with the rate of change.

CONCLUSIONS

These results suggest that syntactic change continues to occur throughout adolescence for boys with FXS or DS. The results also indicate that the growth profiles are distinct between the two groups. Future research with more participants from more diverse backgrounds would add more clarity to these findings.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.27984548.

Updates on Clinical Language Sampling Practices: A Survey of Speech-Language Pathologists Practicing in the United States

PURPOSE

Language sample analysis (LSA) provides many benefits for assessing, identifying therapy goals, and monitoring the progress of children with language disorders. Despite these widely recognized advantages, previous surveys suggest the declining use of LSA by speech-language pathologists (SLPs). This study aimed to provide updates on clinical LSA use following the recent introduction of two new LSA protocols, namely, the Sampling Utterances and Grammatical Analysis Revised (SUGAR) protocol and the Computerized Language Analysis KIDEVAL program.

METHOD

Survey data from SLPs practicing in the United States (N = 337) were used to examine rates of LSA use, methods, and protocols. Factors predicting LSA use and reported facilitators and barriers were also examined.

RESULTS

Results indicated that 60% of SLPs used LSA in the past year. LSA skill level, training, and serving preschool or elementary school children predicted LSA use, whereas workplace, caseload, and years of experience were not significant predictors. Most SLPs reported using self-designed LSA protocols (62%), followed by Systematic Analysis of Language Transcripts (23%) and SUGAR (12%) protocols. SLPs who did not use LSA reported limited time (74%), limited resources (59%), and limited expertise (41%) as barriers and identified additional training on LSA computer programs (52%) and access to automatic speech recognition programs (49%) as facilitators to their adoption of LSA.

CONCLUSIONS

Reported rates of LSA use and methods were consistent with previous survey findings. This study's findings highlight the ongoing needs for more extensive preprofessional training in LSA.

Using language sample analyses across English dialects: A case-based approach for preschoolers

This study compared language samples from typically developing 4-year-olds who spoke African American English (AAE), Jamaican English (JE), or Mainstream American English (MAE) to assess the value of using language sample analysis (LSA) measures for characterising language use across dialects of English. Specific LSA metrics included mean length of utterance (MLU) in morphemes and in words, the Index of Productive Syntax (IPSyn), Developmental Sentence Scoring (DSS) and measures of lexical diversity. Children demonstrated diverse linguistic patterns across dialects, but a Kruskal-Wallis H test did not reveal significant differences in scores obtained through LSA measures. Notably, the IPSyn captured morphosyntactic structures in each category across dialects where prior research has highlighted limitations. This preliminary study uses a case-based approach to illustrate the applicability of LSAs in describing linguistic variations across children who speak different dialects of English. Moreover, the findings from this study underscore the potential use of LSAs in describing linguistic patterns to support the characterisation of communication profiles for culturally and linguistically diverse children.

Should We Stop Using Lexical Diversity Measures in Children's Language Sample Analysis?

PURPOSE

Prior work has identified weaknesses in commonly used indices of lexical diversity in spoken language samples, such as type-token ratio (TTR) due to sample size and elicitation variation, we explored whether TTR and other diversity measures, such as number of different words/100 (NDW), vocabulary diversity (VocD), and the moving average TTR would be more sensitive to child age and clinical status (typically developing [TD] or developmental language disorder [DLD]) if samples were obtained from standardized prompts.

METHOD

We utilized archival data from the norming samples of the Test of Narrative Language and the Edmonton Narrative Norms Instrument. We examined lexical diversity and other linguistic properties of the samples, from a total of 1,048 children, ages 4-11 years; 798 of these were considered TD, whereas 250 were categorized as having a language learning disorder.

RESULTS

TTR was the least sensitive to child age or diagnostic group, with good potential to misidentify children with DLD as TD and TD children as having DLD. Growth slopes of NDW were shallow and not very sensitive to diagnostic grouping. The strongest performing measure was VocD. Mean length of utterance, TNW, and verbs/utterance did show both good growth trajectories and ability to distinguish between clinical and typical samples.

CONCLUSIONS

This study, the largest and best controlled to date, re-affirms that TTR should not be used in clinical decision making with children. A second popular measure, NDW, is not measurably stronger in terms of its psychometric properties. Because the most sensitive measure of lexical diversity, VocD, is unlikely to gain popularity because of reliance on computer-assisted analysis, we suggest alternatives for the appraisal of children's expressive vocabulary skill.

Effect of sample length on MLU in Mandarin-speaking hard-of-hearing children

This study investigated the impact of language sample length on mean length of utterance (MLU) and aimed to determine the minimum number of utterances required for a reliable MLU. Conversations were collected from Mandarin-speaking, hard-of-hearing and typical-hearing children aged 16-81 months. The MLUs were calculated using sample sizes ranging from 25 to 200 utterances. The results showed that for an MLU between 1.0 and 2.5, 25 and 50 utterances were sufficient for reliable MLU calculations for hard-of-hearing and typical-hearing children, respectively. For an MLU between 2.5 and 3.75, 125 utterances were required for both groups. For an MLU greater than 3.75, 150 and 125 utterances were required for hard-of-hearing and typical-hearing children, respectively. These findings suggest that a greater number of utterances are required for a reliable MLU as language complexity increases. Professionals working with hard-of-hearing children should consider collecting different numbers of utterances based on the children's language complexity levels.

Assessment and Therapy Goal Planning Using Free Computerized Language Analysis Software

Background

We discuss a free software system (Computerized Language Analysis [CLAN]) that can enable fast, thorough, and informative language sample analysis (LSA).

Method

We describe methods for eliciting, transcribing, analyzing, and interpreting language samples. Using a hypothetical child speaker, we illustrate use KidEval to generate a diagnostic report.

Results

Given LSA results suggestive of expressive language delay, we analyze further using CLAN's Developmental Sentence Score and Index of Productive Syntax routines, and outline the child's use of Brown's morphemes.

Discussion

This tutorial provides an introduction to the use of free CLAN software. We discuss how LSA results can be used to structure therapy goals that address specific aspects of grammatical structure that the child may not yet demonstrate in their spoken language. Finally, we provide answers to common questions, including user support.

Measurement of Lexical Diversity in Children's Spoken Language: Computational and Conceptual Considerations

Background

Type-Token Ratio (TTR), given its relatively simple hand computation, is one of the few LSA measures calculated by clinicians in everyday practice. However, it has significant well-documented shortcomings; these include instability as a function of sample size, and absence of clear developmental profiles over early childhood. A variety of alternative measures of lexical diversity have been proposed; some, such as Number of Different Words/100 (NDW) can also be computed by hand. However, others, such as Vocabulary Diversity (VocD) and the Moving Average Type Token Ratio (MATTR) rely on complex resampling algorithms that cannot be conducted by hand. To date, no large-scale study of all four measures has evaluated how well any capture typical developmental trends over early childhood, or whether any reliably distinguish typical from atypical profiles of expressive child language ability.

Materials and Methods

We conducted linear and non-linear regression analyses for TTR, NDW, VocD, and MATTR scores for samples taken from 946 corpora from typically developing preschool children (ages 2-6 years), engaged in adult-child toy play, from the Child Language Data Exchange System (CHILDES). These were contrasted with 504 samples from children known to have delayed expressive language skills (total n = 1,454 samples). We also conducted a separate sub-analysis which examined possible contextual effects of sampling environment on lexical diversity.

Results

Only VocD showed significantly different mean scores between the typically -developing children and delayed developing children group. Using TTR would actually misdiagnose typical children and miss children with known language impairment. However, computation of VocD as a function of toy interactions was significant and emerges as a further caution in use of lexical diversity as a valid proxy index of children's expressive vocabulary skill.

Discussion

This large scale statistical comparison of computer-implemented algorithms for expressive lexical profiles in young children with traditional, hand-calculated measures showed that only VocD met criteria for evidence-based use in LSA. However, VocD was impacted by sample elicitation context, suggesting that non-linguistic factors, such as engagement with elicitation props, contaminate estimates of spoken lexical skill in young children. Implications and suggested directions are discussed.

Automated Progress-Monitoring for Literate Language Use in Narrative Assessment (LLUNA)

Language sample analysis (LSA) is an important practice for providing a culturally sensitive and accurate assessment of a child's language abilities. A child's usage of literate language devices in narrative samples has been shown to be a critical target for evaluation. While automated scoring systems have begun to appear in the field, no such system exists for conducting progress-monitoring on literate language usage within narratives. The current study aimed to develop a hard-coded scoring system called the Literate Language Use in Narrative Assessment (LLUNA), to automatically evaluate six aspects of literate language in non-coded narrative transcripts. LLUNA was designed to individually score six literate language elements (e.g., coordinating and subordinating conjunctions, meta-linguistic and meta-cognitive verbs, adverbs, and elaborated noun phrases). The interrater reliability of LLUNA with an expert scorer, as well as its' reliability compared to certified undergraduate scorers was calculated using a quadratic weighted kappa (K_qw). Results indicated that LLUNA met strong levels of interrater reliability with an expert scorer on all six elements. LLUNA also surpassed the reliability levels of certified, but non-expert scorers on four of the six elements and came close to matching reliability levels on the remaining two. LLUNA shows promise as means for automating the scoring of literate language in LSA and narrative samples for the purpose of assessment and progress-monitoring.

The Reliability of Short Conversational Language Sample Measures in Children With and Without Developmental Language Disorder

PURPOSE

Language sample analysis (LSA) represents an ecologically valid method for diagnosing, identifying goals, and measuring progress in children with developmental language disorder (DLD). LSA is, however, time consuming. The purpose of this study was to determine the length of sample needed to obtain reliable LSA measures for children in kindergarten and first grade with typical language (TL) and DLD using automated analyses from the Systematic Analysis of Language Transcripts software.

METHOD

Play-based conversational language samples collected on kindergarten to first-grade children with TL (n = 21) and DLD (n = 21) from a community-based sample were analyzed. Eight LSA measures were calculated from 1-, 3-, 5-, 7-, and 10-min sample cuts and compared to 20-min samples for reliability.

RESULTS

Reliability estimates were similar for the TL and DLD groups except for errors and omissions, which showed overall higher levels of reliability in the DLD group and reached acceptable levels at 3 min. Percent grammatical utterances were reliable at 7 min in the DLD group and not reliable in shorter samples in the TL group. The subordination index was reliable at 10 min for both groups. Number of different words reached acceptable reliability at the 3-min length for the DLD group and at the 10-min length for the TL group. Utterances and words per minute were reliable at 3 min and mean length of utterance at 7 min in both groups.

CONCLUSIONS

Speech-language pathologists can obtain reliable LSA measures from shorter, 7-min conversational language samples from kindergarten to first-grade children with DLD. Shorter language samples may encourage increased use of LSA.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.19529287.

Dynamic Norming and Open Science

In a recent issue of JSLHR, Tucci et al. (2022) presented a method for assigning SEM scores to a language sample. However, this method is based on data that are not publicly available and uses a commercial analysis program that is not open source. The TalkBank system and the Child Language Data Exchange System database provides free analysis software based on openly accessible data, thereby adhering to Open Science standards, which represent an important next step for the fields of speech and hearing.

The Index of Productive Syntax: Psychometric Properties and Suggested Modifications

PURPOSE

The Index of Productive Syntax (IPSyn) is a well-known language sample analysis tool. However, its psychometric properties have not been assessed across a wide sample of typically developing preschool-age children and children with language disorders. We sought to determine the profile of IPSyn scores by age over early childhood. We additionally explored whether the IPSyn could be shortened to fewer items without loss of information and whether the required language sample could be shortened from a current required number of 100 utterances to 50.

METHOD

We used transcripts from the Child Language Data Exchange System, including 1,051 samples of adult-child conversational play with toys within the theoretical framework of item response theory. Samples included those from typically developing children as well as children with hearing loss, Down syndrome, and late language emergence.

RESULTS

The Verb Phrase and Sentence Structure subscales showed more stable developmental trajectories over the preschool years and greater differentiation between typical and atypical cohorts than did the Noun Phrase and Question/Negation subscales. A number of current IPSyn scoring items can be dropped without loss of information, and 50-utterance samples demonstrate most of the same psychometric properties of longer samples.

DISCUSSION

Our findings suggest ways in which the IPSyn can be automated and streamlined (proposed IPSyn-C) so as to provide useful clinical guidance with fewer items and a shorter required language sample. Reference values for the IPSyn-C are provided. Trajectories for one subscale (Question/Negation) appear inherently unstable and may require structured elicitation. Potential limitations, ramifications, and future directions are discussed.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.16915690.

An Evaluation of Expedited Transcription Methods for School-Age Children's Narrative Language: Automatic Speech Recognition and Real-Time Transcription

Purpose This study examined the accuracy and potential clinical utility of two expedited transcription methods for narrative language samples elicited from school-age children (7;5-11;10 [years;months]) with developmental language disorder. Transcription methods included real-time transcription produced by speech-language pathologists (SLPs) and trained transcribers (TTs) as well as Google Cloud Speech automatic speech recognition. Method The accuracy of each transcription method was evaluated against a gold-standard reference corpus. Clinical utility was examined by determining the reliability of scores calculated from the transcripts produced by each method on several language sample analysis (LSA) measures. Participants included seven certified SLPs and seven TTs. Each participant was asked to produce a set of six transcripts in real time, out of a total 42 language samples. The same 42 samples were transcribed using Google Cloud Speech. Transcription accuracy was evaluated through word error rate. Reliability of LSA scores was determined using correlation analysis. Results Results indicated that Google Cloud Speech was significantly more accurate than real-time transcription in transcribing narrative samples and was not impacted by speech rate of the narrator. In contrast, SLP and TT transcription accuracy decreased as a function of increasing speech rate. LSA metrics generated from Google Cloud Speech transcripts were also more reliably calculated. Conclusions Automatic speech recognition showed greater accuracy and clinical utility as an expedited transcription method than real-time transcription. Though there is room for improvement in the accuracy of speech recognition for the purpose of clinical transcription, it produced highly reliable scores on several commonly used LSA metrics. Supplemental Material https://doi.org/10.23641/asha.15167355.

Topic Modeling for Analyzing Topic Manipulation Skills

There are many ways to communicate with people, the most representative of which is a conversation. A smooth conversation should not only be written in a grammatically appropriate manner, but also deal with the subject of conversation; this is known as language ability. In the past, this ability has been evaluated by language analysis/therapy experts. However, this process is time-consuming and costly. In this study, the researchers developed a Hallym Systematic Analyzer of Korean language to automate the conversation analysis process traditionally conducted by language analysis/treatment experts. However, current morpheme analyzers or parsing analyzers can only evaluate certain elements of a conversation. Therefore, in this paper, we added the ability to analyze the topic manipulation skills (the number of topics and the rate of topic maintenance) using the existing Hallym Systematic Analyzer of Korean language. The purpose of this study was to utilize the topic modeling technique to automatically evaluate topic manipulation skills. By quantitatively evaluating the topic management capabilities that were previously evaluated in a conventional manner, it was possible to automatically analyze language ability in a wider range of aspects. The experimental results show that the automatic analysis methodology presented in this study achieved a very high level of correlation with language analysis/therapy professionals.

Using Free Computer-Assisted Language Sample Analysis to Evaluate and Set Treatment Goals for Children Who Speak African American English

Purpose Spoken language sample analysis (LSA) is widely considered to be a critical component of assessment for child language disorders. It is our best window into a preschool child's everyday expressive communicative skills. However, historically, the process can be cumbersome, and reference values against which LSA findings can be "benchmarked" are based on surprisingly little data. Moreover, current LSA protocols potentially disadvantage speakers of nonmainstream English varieties, such as African American English (AAE), blurring the line between language difference and disorder. Method We provide a tutorial on the use of free software (Computerized Language Analysis [CLAN]) enabled by the ongoing National Institute on Deafness and Other Communication Disorders-funded "Child Language Assessment Project." CLAN harnesses the advanced computational power of the Child Language Data Exchange System archive (www.childes.talkbank.org), with an aim to develop and test fine-grained and potentially language variety-sensitive benchmarks for a range of LSA measures. Using retrospective analysis of data from AAE-speaking children, we demonstrate how CLAN LSA can facilitate dialect-fair assessment and therapy goal setting. Results Using data originally collected to norm the Diagnostic Evaluation of Language Variation, we suggest that Developmental Sentence Scoring does not appear to bias against children who speak AAE but does identify children who have language impairment (LI). Other LSA measure scores were depressed in the group of AAE-speaking children with LI but did not consistently differentiate individual children as LI. Furthermore, CLAN software permits rapid, in-depth analysis using Developmental Sentence Scoring and the Index of Productive Syntax that can identify potential intervention targets for children with developmental language disorder.

Forum on Morphosyntax Assessment and Intervention for Children

Purpose

This forum consists of articles that address the need for and approaches to assessment and treatment of morphology and syntax in children. Drawing on papers submitted by diverse laboratories working with multiple populations, this forum includes several articles describing different approaches to treatment, guidelines for goal setting, and assessment methods. Populations described include monolingual and bilingual children who speak English, Dutch, and Spanish, who use oral language and/or augmentative and alternative communication to communicate.

Conclusion

The current tools available to support traditional grammar therapy are changing and increasing. An emphasis on manualized treatments, treatments that include drill and explicit instruction, and assessment and treatment tools for a variety of populations across a wide age span are included here. Further work is needed to fully develop these promising tools and approaches for the most effective use.