Psychometric Evaluation of Lexical Diversity Indices in Spanish Narrative Samples From Children With and Without Developmental Language Disorder

Should We Stop Using Lexical Diversity Measures in Children's Language Sample Analysis?

PURPOSE

Prior work has identified weaknesses in commonly used indices of lexical diversity in spoken language samples, such as type-token ratio (TTR) due to sample size and elicitation variation, we explored whether TTR and other diversity measures, such as number of different words/100 (NDW), vocabulary diversity (VocD), and the moving average TTR would be more sensitive to child age and clinical status (typically developing [TD] or developmental language disorder [DLD]) if samples were obtained from standardized prompts.

METHOD

We utilized archival data from the norming samples of the Test of Narrative Language and the Edmonton Narrative Norms Instrument. We examined lexical diversity and other linguistic properties of the samples, from a total of 1,048 children, ages 4-11 years; 798 of these were considered TD, whereas 250 were categorized as having a language learning disorder.

RESULTS

TTR was the least sensitive to child age or diagnostic group, with good potential to misidentify children with DLD as TD and TD children as having DLD. Growth slopes of NDW were shallow and not very sensitive to diagnostic grouping. The strongest performing measure was VocD. Mean length of utterance, TNW, and verbs/utterance did show both good growth trajectories and ability to distinguish between clinical and typical samples.

CONCLUSIONS

This study, the largest and best controlled to date, re-affirms that TTR should not be used in clinical decision making with children. A second popular measure, NDW, is not measurably stronger in terms of its psychometric properties. Because the most sensitive measure of lexical diversity, VocD, is unlikely to gain popularity because of reliance on computer-assisted analysis, we suggest alternatives for the appraisal of children's expressive vocabulary skill.

Indices of Narrative Language Associated with Disability

Narratives skills are associated with long-term academic and social benefits. While students with disabilities often struggle to produce complete and complex narratives, it remains unclear which aspects of narrative language are most indicative of disability. In this study, we examined the association between a variety of narrative contents and form indices and disability. Methodology involved drawing 50 K-3 students with Individual Education Programs (IEP) and reported language concerns from a large diverse sample (n = 1074). Fifty typically developing (TD) students were matched to the former group using propensity score matching based on their age, gender, grade, mother's education, and ethnicity. Narrative retells and generated language samples were collected and scored for Narrative Discourse and Sentence Complexity using a narrative scoring rubric. In addition, the number of different words (NDW), subordination index (SI), and percentage of grammatical errors (%GE) were calculated using computer software. Results of the Mixed effect model revealed that only Narrative Discourse had a significant effect on disability, with no significant effect revealed for Sentence Complexity, %GE, SI, and NDW. Additionally, Narrative Discourse emerged as the sole significant predictor of disability. At each grade, there were performance gaps between groups in the Narrative Discourse, Language Complexity, and SI. Findings suggest that difficulty in Narrative Discourse is the most consistent predictor of disability.

ADscreen: A speech processing-based screening system for automatic identification of patients with Alzheimer's disease and related dementia

Alzheimer's disease and related dementias (ADRD) present a looming public health crisis, affecting roughly 5 million people and 11 % of older adults in the United States. Despite nationwide efforts for timely diagnosis of patients with ADRD, >50 % of them are not diagnosed and unaware of their disease. To address this challenge, we developed ADscreen, an innovative speech-processing based ADRD screening algorithm for the protective identification of patients with ADRD. ADscreen consists of five major components: (i) noise reduction for reducing background noises from the audio-recorded patient speech, (ii) modeling the patient's ability in phonetic motor planning using acoustic parameters of the patient's voice, (iii) modeling the patient's ability in semantic and syntactic levels of language organization using linguistic parameters of the patient speech, (iv) extracting vocal and semantic psycholinguistic cues from the patient speech, and (v) building and evaluating the screening algorithm. To identify important speech parameters (features) associated with ADRD, we used the Joint Mutual Information Maximization (JMIM), an effective feature selection method for high dimensional, small sample size datasets. Modeling the relationship between speech parameters and the outcome variable (presence/absence of ADRD) was conducted using three different machine learning (ML) architectures with the capability of joining informative acoustic and linguistic with contextual word embedding vectors obtained from the DistilBERT (Bidirectional Encoder Representations from Transformers). We evaluated the performance of the ADscreen on an audio-recorded patients' speech (verbal description) for the Cookie-Theft picture description task, which is publicly available in the dementia databank. The joint fusion of acoustic and linguistic parameters with contextual word embedding vectors of DistilBERT achieved F1-score = 84.64 (standard deviation [std] = ±3.58) and AUC-ROC = 92.53 (std = ±3.34) for training dataset, and F1-score = 89.55 and AUC-ROC = 93.89 for the test dataset. In summary, ADscreen has a strong potential to be integrated with clinical workflow to address the need for an ADRD screening tool so that patients with cognitive impairment can receive appropriate and timely care.

Investigating Narrative Performance in Children With Developmental Language Disorder: A Systematic Review and Meta-Analysis

PURPOSE

Narrative assessment is one potentially underutilized and inconsistent method speech-language pathologists may use when considering a diagnosis of developmental language disorder (DLD). However, narration research encompasses many varied methodologies. This systematic review and meta-analysis aimed to (a) investigate how various narrative assessment types (e.g., macrostructure, microstructure, and internal state language) differentiate children with typical development (TD) from children with DLD, (b) identify specific narrative assessment measures that result in greater group differences, and (c) evaluate participant and sample characteristics that may influence performance differences.

METHOD

Electronic databases (PsycINFO, ERIC, and PubMed) and ASHAWire were searched on July 30, 2019, to locate studies that reported oral narrative language measures for both DLD and TD groups between ages 4 and 12 years; studies focusing on written narration or other developmental disorders only were excluded. We extracted data related to sample participants, narrative task(s) and assessment measures, and research design. Group differences were quantified using standardized mean differences. Analyses used mixed-effects meta-regression with robust variance estimation to account for effect size dependencies.

RESULTS

Searches identified 37 eligible studies published between 1987 and 2019, including 382 effect sizes. Overall meta-analysis showed that children with DLD had decreased narrative performance relative to TD peers, with an overall average effect of -0.82 SD, 95% confidence interval [-0.99, -0.66]. Effect sizes showed significant heterogeneity both between and within studies, even after accounting for effect size-, sample-, and study-level predictors. Across model specifications, grammatical accuracy (microstructure) and story grammar (macrostructure) yielded the most consistent evidence of TD-DLD group differences.

CONCLUSIONS

Present findings suggest some narrative assessment measures yield significantly different performance between children with and without DLD. However, researchers need to improve consistency of inclusionary criteria, descriptions of sample characteristics, and reporting of correlations between measures to determine which assessment measures reliably distinguish between groups.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.21200380.

Dual Language Profiles in Spanish-Speaking English Learners

PURPOSE

The purpose of this study was to identify and describe latent dual language profiles in a large sample of school-age Spanish-English bilingual children designated as English learners (ELs) by their school district.

METHOD

Data for this study include 847 Spanish-speaking ELs from kindergarten to third grade. Spanish and English narrative retell language samples were collected from all participants. Four oral language measures were calculated in Spanish and English, including the subordination index, moving average type-token ratio, narrative structure scheme (NSS), and words per minute using Systematic Analysis of Language Transcript. These indicator measures were used in a latent profile analysis to identify dual language profiles.

RESULTS

The optimal model represents a four-profile solution, including a Spanish-dominant group (average Spanish, low English), an English-dominant group (low Spanish, average English), and two balanced groups (a balanced-average group and a balanced-high group). Additionally, participants displayed uneven performance across language domains and distinct patterns of unique strength or weakness in a specific domain in one of their two languages.

CONCLUSIONS

Findings from this study highlight the large variability in English and Spanish oral language abilities in school-age Spanish-speaking ELs and suggest that a dichotomous classification of ELs versus English-proficient students may not be sufficient to determine the type of educational program that best fits a specific bilingual child's need. These findings highlight the need to assess both languages across multiple language domains to paint a representative picture of a bilingual child's language abilities. The dual language profiles identified may be used to guide the educational program selection process to improve the congruence among the linguistic needs of an individual child, teachers' use of instructional language, and the goals of the educational program (i.e., improving English proficiency vs. supporting dual language development).

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.20151836.

Measurement of Lexical Diversity in Children's Spoken Language: Computational and Conceptual Considerations

Background

Type-Token Ratio (TTR), given its relatively simple hand computation, is one of the few LSA measures calculated by clinicians in everyday practice. However, it has significant well-documented shortcomings; these include instability as a function of sample size, and absence of clear developmental profiles over early childhood. A variety of alternative measures of lexical diversity have been proposed; some, such as Number of Different Words/100 (NDW) can also be computed by hand. However, others, such as Vocabulary Diversity (VocD) and the Moving Average Type Token Ratio (MATTR) rely on complex resampling algorithms that cannot be conducted by hand. To date, no large-scale study of all four measures has evaluated how well any capture typical developmental trends over early childhood, or whether any reliably distinguish typical from atypical profiles of expressive child language ability.

Materials and Methods

We conducted linear and non-linear regression analyses for TTR, NDW, VocD, and MATTR scores for samples taken from 946 corpora from typically developing preschool children (ages 2-6 years), engaged in adult-child toy play, from the Child Language Data Exchange System (CHILDES). These were contrasted with 504 samples from children known to have delayed expressive language skills (total n = 1,454 samples). We also conducted a separate sub-analysis which examined possible contextual effects of sampling environment on lexical diversity.

Results

Only VocD showed significantly different mean scores between the typically -developing children and delayed developing children group. Using TTR would actually misdiagnose typical children and miss children with known language impairment. However, computation of VocD as a function of toy interactions was significant and emerges as a further caution in use of lexical diversity as a valid proxy index of children's expressive vocabulary skill.

Discussion

This large scale statistical comparison of computer-implemented algorithms for expressive lexical profiles in young children with traditional, hand-calculated measures showed that only VocD met criteria for evidence-based use in LSA. However, VocD was impacted by sample elicitation context, suggesting that non-linguistic factors, such as engagement with elicitation props, contaminate estimates of spoken lexical skill in young children. Implications and suggested directions are discussed.

Do vocabulary and narrative skills correlate in preschoolers with typical language development?

PURPOSE

To characterize the performance of preschoolers with typical language development in tasks of expressive vocabulary and oral narrative and to verify possible correlations.

METHODS

The study included 39 children aged 4 to 6 years old, of both genders, with no complaints about language development. Mothers answered a questionnaire of socioeconomic classification, while the ABFW Vocabulary Test was used to evaluate the vocabulary and the book "Frog, where are you?" was used to elicit the child's oral narrative. The data collected were submitted to descriptive and inferential statistical analysis.

RESULTS

Regarding expressive vocabulary, the majority of preschoolers (92.3%) had the usual verbal designation (UVD) suitable for the age group, and the semantic fields with the highest UVD were "animals", "shapes and colors", "toys and musical instruments", "transportation" and those with children were "professions" and "local". The predominant type of narrative was causal, followed by intentional. There was no correlation between UVD and the use of words in the narratives, but there was a positive correlation between the total and the number of different words used in the narrative.

CONCLUSION

There was no correlation between the expressive vocabulary (UVD) and the use of words in the narrative, but the preschoolers who used more words in their narratives also showed greater lexical variety in this sample.

A Longitudinal Study of Spanish Language Growth and Loss in Young Spanish-English Bilingual Children

This longitudinal study examined trajectories of Spanish language growth and loss in 34 Spanish-English bilingual children attending an English immersion school. Narrative retell language samples were collected in Spanish across 3 years using wordless, picture storybooks. Digital audio recordings were transcribed, coded, and analyzed for mean length of utterance in words, proportion of grammatical utterances, and moving-average type-token ratio. Code switching into English was also coded at the word level to determine its potential impact on moving-average type-token ratio. Growth curve models were used to estimate the change over time for each outcome measure. The findings indicated that the Spanish-English bilingual participants who attended an English immersion school demonstrated loss of Spanish grammatical and lexical production (as defined by encompassing maintenance and or significant deceleration) from preschool through kindergarten, and that the degree of loss in lexical production was impacted by whether code switching was included or excluded. The findings are discussed in the context of clinical decision-making when assessing the Spanish expressive language abilities of this specific population.

Vocabulary Size Influences Spontaneous Speech in Native Language Users: Validating the Use of Automatic Speech Recognition in Individual Differences Research

Previous research has shown that vocabulary size affects performance on laboratory word production tasks. Individuals who know many words show faster lexical access and retrieve more words belonging to pre-specified categories than individuals who know fewer words. The present study examined the relationship between receptive vocabulary size and speaking skills as assessed in a natural sentence production task. We asked whether measures derived from spontaneous responses to everyday questions correlate with the size of participants' vocabulary. Moreover, we assessed the suitability of automatic speech recognition (ASR) for the analysis of participants' responses in complex language production data. We found that vocabulary size predicted indices of spontaneous speech: individuals with a larger vocabulary produced more words and had a higher speech-silence ratio compared to individuals with a smaller vocabulary. Importantly, these relationships were reliably identified using manual and automated transcription methods. Taken together, our results suggest that spontaneous speech elicitation is a useful method to investigate natural language production and that automatic speech recognition can alleviate the burden of labor-intensive speech transcription.