How do we determine the utility of screening tools?

The influence of loss to follow-up in autism screening research: Taking stock and moving forward

BACKGROUND

How best to improve the early detection of autism spectrum disorder (ASD) is the subject of significant controversy. Some argue that universal ASD screeners are highly accurate, whereas others argue that evidence for this claim is insufficient. Relatedly, there is no clear consensus as to the optimal role of screening for making referral decisions for evaluation and treatment. Published screening research can meaningfully inform these questions-but only through careful consideration of children who do not complete diagnostic follow-up.

METHODS

We developed two simulation models that re-analyze the results of a large-scale validation study of the M-CHAT-R/F by Robins et al. (2014, Pediatrics, 133, 37). Model #1 re-analyzes screener accuracy across six scenarios, each reflecting different assumptions regarding loss to follow-up. Model #2 builds on this by closely examining differential attrition at each point of the multi-step detection process.

RESULTS

Estimates of sensitivity ranged from 40% to 94% across scenarios, demonstrating that estimates of accuracy depend on assumptions regarding the diagnostic status of children who were lost to follow-up. Across a range of plausible assumptions, data also suggest that children with undiagnosed ASD may be more likely to complete follow-up than children without ASD, highlighting the role of clinicians and caregivers in the detection process.

CONCLUSIONS

Using simulation modeling as a quantitative method to examine potential bias in screening studies, analyses suggest that ASD screening tools may be less accurate than is often reported. Models also demonstrate the critical importance of every step in a detection process-including steps that determine whether children should complete an additional evaluation. We conclude that parent and clinician decision-making regarding follow-up may contribute more to detection than is widely assumed.

The Developmental Autism Early Screening (DAES): A Novel Test for Screening Autism Spectrum Disorder

This study was undertaken to set a novel developmental screening test for autism spectrum disorder (ASD) using the Griffiths Scales of Child Development (Griffith III) (Green et al., 2016; Stroud et al., 2016), in order to intercept the early atypical developmental patterns indicating ASD risk in the first 3 years of age. An observational and interactive ASD screener, the Developmental Autism Early Screening (DAES), was developed by detecting Griffiths III items differentiating toddlers with ASD risk from those with global developmental delay (DD) or neurotypical development. The DAES was validated with ASD-specific diagnostic instruments (ADOS-2) and the cut-off score based on sensitivity, specificity and positive predictive value that best differentiates between ASD and non-ASD children was identified. We enrolled a total sample of 297 subjects, including children at risk for ASD or DD and neurotypical children. At a cut-off score of 12.5, the DAES had a sensitivity of 93%, specificity of 98.4%, positive predictive value of 96.3% and negative predictive value of 96.9% for identifying children at risk for ASD from non-ASD participants (DD/neurotypical children). The DAES total score correlated significantly with the ADOS-2 calibrated severity scores (CSS) (R = 0.53, p < 0.001). Three ASD risk ranges were identified according to DAES total and ADOS-2 CSS: Little-to-no risk (CSS: 1-3, DAES: 1-7); Mild-to-moderate risk (CSS: 4-5, DAES: 8-14); Moderate-to-severe risk (CSS: 6-10, DAES ≥ 15). The DAES provides a direct approach based on developmental profiles to stratify risk for ASD in early childhood ensuring at risk children the most appropriate diagnostic procedures and targeted intervention.

Sex differences in early autism screening using the Modified Checklist for Autism in Toddlers, Revised, with Follow-Up (M-CHAT-R/F)

LAY ABSTRACT

This study examined a widely used autism screening tool, the Modified Checklist for Autism in Toddlers, Revised, with Follow-Up to identify differences in screening for autism between toddler males and females. Examining sex differences in screening for autism in toddlerhood is important as it determines who will be referred for evaluations and receive diagnoses, which is critical for access to autism-specific early intervention. This study found that females were less likely to screen positive and be invited for evaluations compared with males. Females at high likelihood for autism were less likely to be diagnosed with autism, which decreases confidence in the screener's results. Importantly, the Modified Checklist for Autism in Toddlers, Revised, with Follow-Up accurately identified both males and females with autism. Future research should examine ways to improve accuracy in screening results for females.

Meta-analysis of the Modified Checklist for Autism in Toddlers, Revised/Follow-up for Screening

CONTEXT

The Modified Checklist for Autism in Toddlers, Revised with Follow-up (M-CHAT-R/F) is used worldwide to screen for autism spectrum disorder (ASD).

OBJECTIVE

To calculate psychometric properties of the M-CHAT-R/F for subsequent diagnosis of ASD.

DATA SOURCES

Systematic searches of Medline, Embase, SCOPUS, and Trip Pro databases from January 2014 to November 2021.

STUDY SELECTION

Studies were included if they (1) used the M-CHAT-R/F (2) applied standard scoring protocol, (3) used a diagnostic assessment for ASD, and (4) reported at least 1 psychometric property of the M-CHAT-R/F.

DATA EXTRACTION

Two independent reviewers completed screening, full-text review, data extraction, and quality assessment, following Preferred Reporting Items for Systematic Reviews and Meta-analyses guidelines. A random-effects model was used to derive pooled estimates and assess for between-study heterogeneity.

RESULTS

Of 667 studies identified, 15 with 18 distinct samples from 10 countries (49 841 children) were used in the meta-analysis. Pooled positive predictive value (PPV), was 57.7% (95% confidence interval [CI] 48.6-66.8, τ2 = 0.031). PPV was higher among high-risk (75.6% [95% CI 66.0-85.2]) than low-risk samples (51.2% [95% CI 43.0-59.5]). Pooled negative predictive value was 72.5% (95% CI 62.5-82.4 τ2 = 0.031), sensitivity was 82.6% (95% CI 76.2-88.9) and specificity 45.7% (95% CI 25.0-66.4).

LIMITATIONS

Negative predictive value, sensitivity, and specificity were calculated based on small sample sizes because of limited or no evaluation of screen-negative children.

CONCLUSIONS

These results support use of the M-CHAT-R/F as a screening tool for ASD. Caregiver counseling regarding likelihood of an ASD diagnosis after positive screen should acknowledge the moderate PPV.

Sensitivity and Specificity of the Modified Checklist for Autism in Toddlers (Original and Revised): A Systematic Review and Meta-analysis

Importance

The Modified Checklist for Autism in Toddlers (M-CHAT) and the M-CHAT, Revised With Follow-up (M-CHAT-R/F)-henceforth referred to as M-CHAT(-R/F)-are the most commonly used toddler screeners for autism spectrum disorder (ASD). Their use often differs from that in the original validation studies, resulting in a range of estimates of sensitivity and specificity. Also, given the variability in reports of the clinical utility of the M-CHAT(-R/F), researchers and practitioners lack guidance to inform autism screening protocols.

Objective

To synthesize variability in sensitivity and specificity of M-CHAT(-R/F) across multiple factors, including procedures for identifying missed cases, likelihood level, screening age, and single compared with repeated screenings.

Data Sources

A literature search was conducted with PubMed, Web of Science, and Scopus to identify studies published between January 1, 2001, and August 31, 2022.

Study Selection

Articles were included if the studies used the M-CHAT(-R/F) (ie, original or revised version) to identify new ASD cases, were published in English-language peer-reviewed journals, included at least 10 ASD cases, reported procedures for false-negative case identification, screened children by 48 months, and included information (or had information provided by authors when contacted) needed to conduct the meta-analysis.

Data Extraction and Synthesis

The systematic review and meta-analysis was conducted within the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) reporting guideline. The Quality Assessment of Diagnostic Accuracy Studies-2 tool evaluated bias in sample selection. Data extraction and quality assessment were performed by 2 authors independently. The overall diagnostic accuracy of the M-CHAT(-R/F) was assessed with the hierarchic summary receiver operating characteristic (HSROC) model.

Main Outcomes and Measures

Sensitivity, specificity, diagnostic odds ratios, and HSROC curves of M-CHAT(-R/F).

Results

The review included 50 studies with 51 samples. The pooled sensitivity of M-CHAT(-R/F) was 0.83 (95% CI, 0.77-0.88), and the pooled specificity was 0.94 (95% CI, 0.89-0.97). Heterogeneity analyses revealed greater diagnostic accuracy for low- vs high-likelihood samples, a concurrent vs prospective case confirmation strategy, a large vs small sample size, use of M-CHAT(-R/F) Follow-up, and non-English vs English only.

Conclusions and Relevance

Overall, results of this study suggest the utility of the M-CHAT(-R/F) as an ASD screener. The wide variability in psychometric properties of M-CHAT(-R/F) highlights differences in screener use that should be considered in research and practice.

Do Autism-Specific and General Developmental Screens Have Complementary Clinical Value?

Prior studies suggest autism-specific and general developmental screens are complementary for identifying both autism and developmental delay (DD). Parents completed autism and developmental screens before 18-month visits. Children with failed screens for autism (n = 167) and age, gender, and practice-matched children passing screens (n = 241) completed diagnostic evaluations for autism and developmental delay. When referral for autism and/or DD was considered, overall false positives from the autism screens were less frequent than for referral for autism alone. Presence of a failed communication subscale in the developmental screen was a red flag for autism and/or DD. An ordinally-scored autism screen had more favorable characteristics when considering autism and/or DD, yet none of the screens achieved recommended standards at 18 months, reinforcing the need for recurrent screening as autism emerges in early development.

Autism screening at 18 months of age: a comparison of the Q-CHAT-10 and M-CHAT screeners

Background

Autism screening is recommended at 18- and 24-month pediatric well visits. The Modified Checklist for Autism in Toddlers—Revised (M-CHAT-R) authors recommend a follow-up interview (M-CHAT-R/F) when positive. M-CHAT-R/F may be less accurate for 18-month-olds than 24-month-olds and accuracy for identification prior to two years is not known in samples that include children screening negative. Since autism symptoms may emerge gradually, ordinally scoring items based on the full range of response options, such as in the 10-item version of the Quantitative Checklist for Autism in Toddlers (Q-CHAT-10), might better capture autism signs than the dichotomous (i.e., yes/no) items in M-CHAT-R or the pass/fail scoring of Q-CHAT-10 items. The aims of this study were to determine and compare the accuracy of the M-CHAT-R/F and the Q-CHAT-10 and to describe the accuracy of the ordinally scored Q-CHAT-10 (Q-CHAT-10-O) for predicting autism in a sample of children who were screened at 18 months.

Methods

This is a community pediatrics validation study with screen positive (n = 167) and age- and practice-matched screen negative children (n = 241) recruited for diagnostic evaluations completed prior to 2 years old.

Clinical diagnosis of autism was based on results of in-person diagnostic autism evaluations by research reliable testers blind to screening results and using the Autism Diagnostic Observation Schedule—Second Edition (ADOS-2) Toddler Module and Mullen Scales of Early Learning (MSEL) per standard guidelines.

Results

While the M-CHAT-R/F had higher specificity and PPV compared to M-CHAT-R, Q-CHAT-10-O showed higher sensitivity than M-CHAT-R/F and Q-CHAT-10.

Limitations

Many parents declined participation and the sample is over-represented by higher educated parents. Results cannot be extended to older ages.

Conclusions

Limitations of the currently recommended two-stage M-CHAT-R/F at the 18-month visit include low sensitivity with minimal balancing benefit of improved PPV from the follow-up interview. Ordinal, rather than dichotomous, scoring of autism screening items appears to be beneficial at this age. The Q-CHAT-10-O with ordinal scoring shows advantages to M-CHAT-R/F with half the number of items, no requirement for a follow-up interview, and improved sensitivity. Yet, Q-CHAT-10-O sensitivity is less than M-CHAT-R (without follow-up) and specificity is less than the two-stage procedure. Such limitations are consistent with recognition that screening needs to recur beyond this age.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13229-021-00480-4.

Leveraging telehealth to evaluate infants with prodromal autism spectrum disorder characteristics using the telehealth evaluation of development for infants

LAY ABSTRACT

Many families seeking early evaluations for autism spectrum disorder face long waitlists, must often travel to centers with appropriate expertise, and are frequently told by providers to "wait and see." This results in significant stress for families and delayed supports to infants and their caregivers who could benefit. This study evaluated whether telehealth could be used to identify and evaluate infants with early autism spectrum disorder characteristics in the first year of life. In this study, we evaluated 41 infants via telehealth using a standard set of probes and scored behavior related to social communication, play, imitation, and other developmental domains. We found the majority of infants demonstrated elevated likelihood of autism spectrum disorder on both parent-reported questionnaires and examiner-rated behavior. Caregiver ratings of the overall utility of the protocol used in this study were high. Overall, this study demonstrates the feasibility for telehealth-based approaches to evaluate infants' with elevated likelihood of autism spectrum disorder in the first year of life, which could help to improve families' access to care and to expand our capacity to conduct studies evaluating possible intervention supports.

Early and Repeated Screening Detects Autism Spectrum Disorder

OBJECTIVE

To evaluate timing and accuracy of early and repeated screening for autism spectrum disorder (ASD) during well-child visits.

STUDY DESIGN

Using a longitudinal study design, toddlers (n = 5784) were initially screened at 12 (n = 1504), 15 (n = 1228), or 18 (n = 3052) months during well-child visits, and rescreened at 18, 24, and 36 months. Of those screened, 368 toddlers attended an ASD evaluation after a positive screen and/or a provider concern for ASD at any visit.

RESULTS

Screens initiated at 12 months yielded an ASD diagnosis significantly earlier than at 15 months (P = .003, d = 0.99) and 18 months (P < .001, d = 0.97). Cross-group overall sensitivity of the initial screen was .715 and specificity was .959. Repeat screening improves sensitivity (82.1%), without notably decreasing specificity (all >93.5%). Screening at 18 months resulted in significantly higher positive predictive value than at 12 months (X² (1, n = 221) = 9.87, P = .002, OR = 2.60) and 15 months (X² (1, n = 208) = 14.57, P < .001, OR = 3.67). With repeat screening, positive predictive value increased for all screen groups, but the increase was not significant.

CONCLUSIONS

Screening as early as 12 months effectively identifies many children at risk for ASD. Children screened at 12 months receive a diagnosis of ASD significantly earlier than peers who are first screened at later ages, facilitating earlier intervention. However, as the sensitivity is lower for a single screen, screening needs to be repeated.

Validation of the Modified Checklist for Autism in Toddlers, Revised with Follow-up in a Population Sample of 30-Month-Old Children in Iceland: A Prospective Approach

The Modified Checklist for Autism in Toddlers, Revised with Follow-up was validated on a population sample in Reykjavik, Iceland. The participants (N = 1585) were screened in well-child care at age 30 months and followed up for at least 2 years to identify autism cases. The sensitivity, specificity, positive and negative predictive values were 0.62, 0.99, 0.72, and 0.99, respectively. True-positive children were diagnosed 10 months earlier than false-negative children. Autism symptom severity and the proportions of children with verbal and performance IQs/DQs < 70 were similar between groups. Although the sensitivity was suboptimal, the screening contributed to lowering the age at diagnosis for many children. Adding autism-specific screening to the well-child care program should be considered.