Prevalence of Publication Bias Tests in Speech, Language, and Hearing Research

Guidance for Increasing Primary Study Inclusion and the Usability of Data in Meta-Analysis: A Reporting Tutorial

Meta-analyses can be used to comprehensively summarize the state of a given literature base, understand development and relations between constructs, and synthesize intervention effects to identify "what works for whom," all of which can directly inform research, practice, and policy. In this tutorial, we first argue that data reporting in primary studies can meaningfully affect the results of meta-analyses and, most importantly, the subsequent interpretations and use of the findings in research, practice, and policy in speech, language, and hearing sciences. Though reporting guidelines have been well established for the results of meta-analyses, there is limited focus on the essentials for primary study reporting that allow for inclusion in meta-analyses. To this end, we provide guidance for primary study authors to ensure their studies can maximize their contributions to research syntheses and, specifically, to meta-analyses. We discuss current and ongoing issues related to reporting, provide data-based examples of instances where lack of reporting or transparency has rendered a study ineligible from inclusion in a meta-analysis, encourage editorial teams and peer reviewers to be flexible in the inclusion of supplemental data reporting based on journal requirements and limits, and suggest being explicit and earnest about why these requests are important to advancing the field. Supplemental Material: https://doi.org/10.23641/asha.23117996.

Evaluating Research Transparency and Openness in Communication Sciences and Disorders Journals

PURPOSE

To improve the credibility, reproducibility, and clinical utility of research findings, many scientific fields are implementing transparent and open research practices. Such open science practices include researchers making their data publicly available and preregistering their hypotheses and analyses. A way to enhance the adoption of open science practices is for journals to encourage or require submitting authors to participate in such practices. Accordingly, the American Speech-Language-Hearing Association's Journals Program has recently announced their intention to promote open science practices. Here, we quantitatively assess the extent to which several journals in communication sciences and disorders (CSD) encourage or require participation in several open science practices by using the Transparency and Openness Promotion (TOP) Factor metric.

METHOD

TOP Factors were assessed for 34 CSD journals, as well as several journals in related fields. TOP Factors measure the level of implementation across 10 open science-related practices (e.g., data transparency, analysis plan preregistration, and replication) for a total possible score of 29 points.

RESULTS

Collectively, CSD journals had very low TOP Factors (M = 1.4, range: 0-8). The related fields of Psychology (M = 4.0), Rehabilitation (M = 3.2), Linguistics (M = 1.7), and Education (M = 1.6) also had low scores, though Psychology and Rehabilitation had higher scores than CSD.

CONCLUSION

CSD journals currently have low levels of encouraging or requiring participation in open science practices, which may impede adoption. Open Science Form: https://doi.org/10.23641/asha.21699458.

Investigating Narrative Performance in Children With Developmental Language Disorder: A Systematic Review and Meta-Analysis

PURPOSE

Narrative assessment is one potentially underutilized and inconsistent method speech-language pathologists may use when considering a diagnosis of developmental language disorder (DLD). However, narration research encompasses many varied methodologies. This systematic review and meta-analysis aimed to (a) investigate how various narrative assessment types (e.g., macrostructure, microstructure, and internal state language) differentiate children with typical development (TD) from children with DLD, (b) identify specific narrative assessment measures that result in greater group differences, and (c) evaluate participant and sample characteristics that may influence performance differences.

METHOD

Electronic databases (PsycINFO, ERIC, and PubMed) and ASHAWire were searched on July 30, 2019, to locate studies that reported oral narrative language measures for both DLD and TD groups between ages 4 and 12 years; studies focusing on written narration or other developmental disorders only were excluded. We extracted data related to sample participants, narrative task(s) and assessment measures, and research design. Group differences were quantified using standardized mean differences. Analyses used mixed-effects meta-regression with robust variance estimation to account for effect size dependencies.

RESULTS

Searches identified 37 eligible studies published between 1987 and 2019, including 382 effect sizes. Overall meta-analysis showed that children with DLD had decreased narrative performance relative to TD peers, with an overall average effect of -0.82 SD, 95% confidence interval [-0.99, -0.66]. Effect sizes showed significant heterogeneity both between and within studies, even after accounting for effect size-, sample-, and study-level predictors. Across model specifications, grammatical accuracy (microstructure) and story grammar (macrostructure) yielded the most consistent evidence of TD-DLD group differences.

CONCLUSIONS

Present findings suggest some narrative assessment measures yield significantly different performance between children with and without DLD. However, researchers need to improve consistency of inclusionary criteria, descriptions of sample characteristics, and reporting of correlations between measures to determine which assessment measures reliably distinguish between groups.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.21200380.

A Systematic Review of Interventions for Multilingual Preschoolers With Speech and Language Difficulties

Purpose There is a shortage of information on evidence-based interventions for supporting young multilingual children. The purpose of this review was to identify interventions that have been evaluated with preschool-age multilingual children with a speech and/or language disorder or who are at risk of poor speech, language, literacy, and/or educational outcomes. Method This review considered speech, language, and early literacy interventions evaluated with preschool-age multilingual children with a speech and/or language disorder or who have been identified as being at risk of language difficulties (PROSPERO ID: 165892). The following electronic databases were searched: EBSCO (CINAHL Plus, ERIC, PsycINFO, Medline, Education) and Linguistics, Language, and Behavior Abstracts. Data were extracted describing article, participant, methodological, and intervention variables, and effect sizes. The Council for Exceptional Children's (CEC) standards for evidence-based practice were used to examine the quality of studies. Results Fifty-six relevant studies were identified in 52 articles and these studies described 4,551 participants who had speech sound disorder (six articles), developmental language disorder (11 articles), or were considered to be at risk (36 articles). The interventions targeted speech production (seven studies), language (45 studies), and early literacy (11 studies) skills. Most studies reported positive effects. Only 15 studies met all quality indicators specified by the CEC (2014) and these described 18 interventions targeting language and literacy skills. The only intervention with sufficient evidence to be considered an evidence-based practice was Nuestros Niños [Our Children] for children's early literacy and phonological awareness skills. Conclusions A number of high-quality studies exist that describe speech, language and/or literacy interventions for preschool-age multilingual children with a speech and/or language disorder, or who have been identified as being at risk of language difficulties. However, there remains limited evidence for specific interventions as to their ability to inform evidence-based practices. Supplemental Material https://doi.org/10.23641/asha.16632649.

Reading Development in Children With Nonsyndromic Cleft Palate With or Without Cleft Lip: Meta-analysis and Systematic Review

OBJECTIVE

We conducted a meta-analysis and systematic review of literature comparing pre-reading and general reading in school-age children with nonsyndromic cleft palate with or without cleft lip (NSCP/L) to their peers without NSCP/L.

METHODS

Our literature search identified 1238 possible records. After screening we identified 11 samples for inclusion for systematic review and eight for meta-analysis. We compared 292 children with NSCP/L to 311 peers for 23 pre-reading effect sizes and 17 general reading effect sizes (EF_g). We conducted a random-effects metaregression using robust variance estimation.

RESULTS

On average school-age children with NSCP/L scored lower on pre-reading (EF_g = -0.36) and general reading measures (EF_g = -0.38) compared to their peers. We conducted post-hoc analyses on phonological awareness and word decoding effect sizes; children with NSCP/L performed lower on phonological awareness (EF_g = -0.22) and word decoding (EF_g = -0.39) compared to their peers. There was weak evidence that hearing status and/or speech-language functioning might moderate reading development. There was limited evidence that age or socioeconomic status moderated reading development. However, samples did not consistently report several characteristics that were coded for this project.

CONCLUSIONS

Our findings suggest that school-age children with NSCP/L have persistent reading problems. Further research is needed to explore reading development in children with NSCP/L, as well as the relationships among hearing, speech, language, and reading development.

Reporting and Reproducibility of Meta-Analysis in Speech, Language, and Hearing Research

Purpose The purposes of this meta-review are to (a) articulate the importance of transparency and reproducibility in meta-analysis, (b) assess the transparency and reproducibility of meta-analyses published in journals of the American Speech-Language-Hearing Association (ASHA), and (c) discuss the implications of our findings and recommendations for future research. Method We conducted a meta-review of all meta-analyses published in ASHA journals through December 31, 2020. Our systematic review yielded 47 meta-analyses for inclusion in this review. We coded all eligible reports on the core elements of transparency and reproducibility in meta-analysis. Results Our findings suggest that though reporting tendencies have improved over time, much work is needed to promote transparency and reproducibility in meta-analytic work. Key areas for future accountability include preregistering study protocol, using Preferred Reporting in Systematic Reviews and Meta-Analysis checklists, providing full data sets, and publishing analytic codes. Conclusions The state of reporting in meta-analysis is improving over time. We conclude with a discussion of specific areas that need further attention, and recommendations for researchers to consider when conducting future meta-analyses. Supplemental Material https://doi.org/10.23641/asha.14888481.

Intervention Effects on Language in Children With Autism: A Project AIM Meta-Analysis

Purpose This study synthesized effects of interventions on language outcomes of young children (ages 0-8 years) with autism and evaluated the extent to which summary effects varied by intervention, participant, and outcome characteristics. Method A subset of effect sizes gathered for a larger meta-analysis (the Autism Intervention Meta-analysis or Project AIM) examining the effects of interventions for young children with autism, which were specific to language outcomes, was analyzed. Robust variance estimation and metaregression were used to calculate summary and moderated effects while controlling for intercorrelation among outcomes within studies. Results A total of 221 outcomes were gathered from 60 studies. The summary effect of intervention on language outcomes was small but significant. Summary effects were larger for expressive and composite language outcomes compared to receptive language outcomes. Interventions implemented by clinicians, or by clinicians and caregivers together, had summary effects that were significantly larger than interventions implemented by caregivers alone. Participants' pretreatment language age equivalent scores positively and significantly moderated intervention effects, such that effects were significantly larger on average when samples of children had higher pretreatment language levels. Effects were not moderated by cumulative intervention intensity, intervention type, autism symptomatology, chronological age, or the proximity or boundedness of outcomes. Study quality concerns were apparent for a majority of included outcomes. Conclusions We found evidence that intervention can facilitate improvements in language outcomes for young children with autism. Effects were largest for expressive and composite language outcomes, for children with initially higher language abilities, and for interventions implemented by clinicians or by caregivers and clinicians combined. However, quality concerns of included studies and borderline significance of some results temper our conclusions regarding intervention effectiveness and corresponding moderators.

Early Speech and Language Development in Children With Nonsyndromic Cleft Lip and/or Palate: A Meta-Analysis

Objective The aim of the study was to conduct a meta-analysis of research examining the early speech and language functioning of young children, birth to age 8;11 (years;months), with nonsyndromic cleft lip and/or palate (NSCL/P) compared to their peers without NSCL/P. Method We conducted a random-effects metaregression using 241 effect sizes from 31 studies comparing 955 young children with NSCL/P to 938 typically developing peers on measures of speech and language functioning. Moderators were sample characteristics (i.e., age, cleft type, publication year, and study location) and measurement characteristics (i.e., speech sample material, language modality and domain, and assessment type). Results Young children with NSCL/P scored significantly lower on measures of speech and language compared to children without NSCL/P. Children with NSCL/P had smaller consonant inventories (standardized mean difference effect size [ESg] = -1.24), less accurate articulation (ESg = -1.13), and more speech errors (ESg = 0.93) than their peers. Additionally, children with NSCL/P had poorer expressive (ESg = -0.57) and receptive (ESg = -0.59) language skills than their peers. Age and assessment type moderated effect sizes for expressive language. As children with NSCL/P aged, their expressive language performance became more similar to their peers. Expressive language effect sizes from parent reports and observational language measures (estimated effect size = -0.74) were significantly lower than those from standardized norm-referenced tests (estimated effect size = -0.45). Conclusions These findings suggest that young children with NSCL/P experience delays relative to their peers across multiple speech and language constructs. Differences between children with NSCL/P and their typically developing peers appear to decrease with age. Supplemental Material https://doi.org/10.23641/asha.11356904.

Quality is Critical for Meaningful Synthesis of Afterschool Program Effects: A Systematic Review and Meta-analysis

Afterschool programs are designed to promote academic achievement and youth development as well as mitigate risk for students who attend. Systematic reviews have reported that afterschool programs are associated with increased academic proficiency, school bonding, prosocial behavior, and decreased problem behavior. However, the majority of meta-analyses that report on these outcomes limit their samples to only rigorously conducted studies, which is not representative of the larger literature base. This systematic review and meta-analysis is designed to extend the literature by meta-analyzing a comprehensive sample of studies and examine the impact of study quality on meta-analytical results. A comprehensive literature and effect size extraction process search yielded 130 effect sizes from 30 studies that examined the effects of afterschool programs on secondary students' academic achievement and social / behavioral development. The majority of studies were unpublished reports, and study quality in the sample represented a high risk of bias. The results confirm the hypothesis that the inclusion of lower quality studies significantly impacts overall outcomes in comparison to prior meta-analyses that have limited samples. As such, it is imperative that researchers and practitioners increase the rigor of study design across individual program evaluations so that fewer reports will be excluded. This will improve future meta-analytic outcomes by ensuring a more representative sample of the literature base. Only through improved rigor of study design will a more clear understanding of program impacts on secondary students be possible.