What is the optimal assessment of speech? A multicentre, international evaluation of speech assessment in 2500 patients with a cleft

OBJECTIVES

Speech problems in patients with a cleft palate are often complex and multifactorial. Finding the optimal way of monitoring these problems is challenging. The International Consortium of Health Outcomes Measurement (ICHOM) has developed a set of standardised outcome measures at specific ages for patients with a cleft lip and/or palate, including measures of speech assessment. This study evaluates the type and timing of speech outcome measures currently included in this ICHOM Standard Set. Additionally, speech assessments in other cleft protocols and initiatives are discussed.

DESIGN, SETTING AND PARTICIPANTS

An international, multicentre study was set up including centres from the USA and the Netherlands. Outcomes of clinical measures and Patient Reported Outcome Measures (PROMs) were collected retrospectively according to the ICHOM set. PROM data from a field test of the CLEFT-Q, a questionnaire developed and validated for patients with a cleft, were collected, including participants from countries with all sorts of income statuses, to examine the value of additional moments of measurement that are used in other cleft initiatives.Data from 2500 patients were included. Measured outcomes contained univariate regression analyses, trend analyses, t-tests, correlations and floor and ceiling effects.

RESULTS

PROMs correlated low to moderate with clinical outcome measures. Clinical outcome measures correlated low to moderate with each other too. In contrast, two CLEFT-Q Scales correlated strongly with each other. All PROMs and the Percent Consonants Correct (PCC) showed an effect of age. In patients with an isolated cleft palate, a ceiling effect was found in the Intelligibility in Context Scale.

CONCLUSION

Recommendations for an optimal speech outcome assessment in cleft patients are made. Measurement moments of different cleft protocols and initiatives are considered in this proposition. Concerning the type of measures, adjustment of the current PCC score outcome seems appropriate. For centres with adequate resources and specific interest in research, translation and validation of an upcoming tool, the Cleft Audit Protocol for Speech Augmented, is recommended.

Effect of perceptive-auditory training on the classification of speech hypernasality

PURPOSE

To analyze the effect of auditory-perceptual training by inexperienced speech-language pathologists in the classification of hypernasality in individuals with cleft lip and palate and compare their classification of hypernasality individually, with the gold standard evaluation, before and after this training.

METHODS

Three inexperienced speech-language pathologists used a four-point scale to assess 24 high-pressure speech samples from individuals with cleft lip and palate, before and after auditory-perceptual training. The speech samples corresponded to six samples of each degree of hypernasality. The speech-language pathologists received auditory-perceptual training during the assessments. They had access to anchor samples and immediate feedback of correct answers regarding the degree of hypernasality in training.

RESULTS

There was no significant difference in the overall percentage of correct answers when comparing before and after the auditory-perceptual training. There was a significant association and agreement of the three evaluators with a gold standard evaluation after training, with an increase in agreement for a single evaluator for absent and mild degrees of hypernasality. The dichotomous analysis of the data showed an increase in the Kappa Index of Agreement of this evaluator. Although there was an increase in the Index of Agreement between evaluators for absent, mild, and severe hypernasality, this increase did not reach statistical significance.

CONCLUSION

The auditory-perceptual training provided did not result in a significant improvement in the hypernasality classification for the inexperienced speech-language pathologists, even though the individual data analysis showed that the training favored one of the evaluators. Further studies involving gradual and more extensive auditory-perceptual training may favor the classification of hypernasality by inexperienced SLPs.

Training speech pathology students in the perceptual evaluation of speech in patients with cleft palate: Reliability results and the students' perspective

INTRODUCTION

Listener training is necessary to achieve agreement between perceptual ratings. However, evidence is limited regarding the effect of a training in the perceptual evaluation of cleft palate speech on the reliability of students in speech pathology. The aim of this study was to evaluate the short-term and longer-term effect of a training using the Belgian Dutch outcome tool for cleft palate speech on reliability and the students' experiences.

METHODS

A 2-h training was provided during a specialist course for 31 first year master students in speech pathology. Information regarding listener characteristics, speech documentation and speech analysis (speech parameters and rating scales) was discussed. This last part focused on the Belgian Dutch outcome tool for speech in patients with cleft palate, including reference samples and consensus listening exercises. Pre training (T1) and post-training reliability (immediately following the training (T2) and one month later (T3)) were collected for the speech variables speech understandability, speech acceptability, hypernasality, hyponasality, nasal emission and nasal turbulence that were rated by means of ordinal scales. Inter-rater reliability was determined with interclass correlation coefficients whereas percentages of agreement were calculated for the intra-rater reliability.

RESULTS

Qualitative inter-rater ICC categories improved for hyponasality and speech acceptability at T2 and T3, whereas the inter-rater reliability of the other variables remained stable regarding their interpreted categories. All ICC values quantitatively improved at T2, except hypernasality and nasal emission. At T3, three of those variables showed a small drop one month later. Regarding intra-rater reliability, a significant decrease for hypernasality and nasal turbulence was found at T2, whereas a significant increase was observed for nasal turbulence and speech acceptability at T3. All variables, except hypernasality, had improved intra-rater reliability at T3 in comparison with T1. Students experienced fewer difficulties with most variables after training. Evaluation of nasal emission and nasal turbulence remained however difficult and required, in the students' opinion, more attention.

DISCUSSION & CONCLUSION

Overall a positive training effect was noted, especially regarding the inter-rater reliability and the students' confidence in perceptual ratings. However, it remains a challenge to achieve a positive training effect for all target variables that is maintained over-time.

Speech Outcomes of a National Cohort of Children with Orofacial Cleft at 5 and 10 Years of age

OBJECTIVE

To assess speech outcomes at five and ten years of age in a nationwide study of children with orofacial cleft.

DESIGN

Prospective study.

PARTICIPANTS

Children born with orofacial cleft and having primary surgery in New Zealand. Speech samples were available for 151 five-year-old, and 163 ten-year-old children.

MAIN OUTCOME MEASURES

Intelligibility, Acceptability, Velopharyngeal function, Hypernasality, Hyponasality, severity of airflow evaluated by perceptual speech assessment (using the standardised Rhinocleft assessment), and overall assessment of requirement for clinical intervention.

RESULTS

A large proportion of five-year-old children had speech that was considered to be not completely intelligible, was not acceptable, and had inadequate velopharyngeal function. The noted deficiencies led to a clinical judgement that further speech and/or surgical intervention was required in 85% with cleft lip and palate, 65% with cleft palate and 26% with cleft lip. The proportion of children with poor speech outcomes in the ten-year-old children was lower, though of clinical importance, further intervention required for 25% with CLP, 15% with CP and 3% with CL. The number of sound production errors in both age groups followed the same pattern with fewest in those with CL and most in those with CLP.

CONCLUSIONS

A significant proportion of children with orofacial cleft were found to have poor speech outcomes requiring further treatment. The outcomes are poor compared to centres reported in the UK and Scandinavia. New Zealand requires a review of the current services for individuals born with cleft to improve speech outcomes and interdisciplinary care.

Speech Outcomes Following Orticochea Pharyngoplasty in Patients With History of Cleft Palate and Noncleft Velopharyngeal Dysfunction

OBJECTIVE

To report speech outcomes following Orticochea pharyngoplasty in 43 patients with cleft palate and noncleft velopharyngeal dysfunction.

DESIGN

A retrospective surgical audit of patients undergoing Orticochea pharyngoplasty between 2004 and 2012, with speech as a primary outcome measure.

SETTING

Patients known to a regional UK cleft center.

METHODS

Forty-three patients underwent Orticochea pharyngoplasty by a single surgeon in a UK regional cleft center. Twenty-one patients had undergone a prior procedure for velopharyngeal dysfunction. Pre- and postoperative speech samples were assessed blindly using the Cleft Audit Protocol for Speech-Augmented by a specialist cleft speech and language therapist, external to the team. Speech samples were rated on the following parameters: hypernasality, hyponasality, audible nasal emission, nasal, turbulence, and passive cleft speech characteristics. Statistical differences in pre- and postoperative speech scores were tested using the Wilcoxon matched-pairs signed-ranks test. Inter- and intrareliability scores were calculated using weighted Cohen κ.

RESULTS

Whole group: A statistically significant difference in pre- and postoperative scores for hypernasality (P < .001), hyponasality (P < .05), nasal emission (P < .01), and passive cleft speech characteristics (P < .01) were reported. Patients with cleft diagnoses: A statistically significant difference in scores for hypernasality (P < .001), nasal emission (P < .01), and passive cleft speech characteristics (P < .01) were reported for this group of patients. Patients with noncleft diagnoses: The only parameter to demonstrate a statistically significant difference was hypernasality (P < .01) in this group.

CONCLUSIONS

Orticochea pharyngoplasty is a successful surgical procedure in treating velopharyngeal dysfunction in both the cleft and noncleft populations.

Reliability of Outcome Measures to Assess Consonant Proficiency Following Cleft Palate Speech Intervention: The Percentage of Consonants Correct Metric and the Probe Scoring System

Purpose This study compared the inter- and intrarater reliability of the percentage of consonants correct (PCC) metrics and the probe scoring system between an experienced and a less experienced rater and between two experienced raters. In addition, these outcome measures' ability to reflect changes following speech intervention was measured. Method During Phase 1, two raters (Rater 1 with 5 years of experience in cleft-related speech disorders and Rater 2 with limited experience in cleft-related speech disorders) independently assessed 134 speech samples at the word and sentence levels, which were collected on different data points before, during, and following a cleft palate speech intervention. During Phase 2, a third rater (with 8 years of experience) analyzed 34 speech samples. The percentage of consonants correct-revised, the percentage of correct places and manners, and probe scores at the word and sentence levels were measured. Results Poor-to-moderate interreliability between Raters 1 and 2 was found due to differences in error classification. Interrater reliability between Raters 1 and 3 was very good for both the PCC metrics and the probe scores. The interrater reliability for the amount of targets elicited was lower compared to the interrater reliability for the amount of targets correct. The probe scoring system demonstrated a greater ability to detect changes toward the correct production of the target consonant compared to the PCC metrics. Conclusions Having an experience with the assessment of cleft-related speech disorders is a crucial factor to gain reliable results. The interrater reliability for the PCC metrics and the probe scoring system between two experienced raters did not differ, suggesting that both outcome measures can be used in cleft palate speech intervention studies. Despite the ability of the probe scoring system to detect changes, further research should provide insight in the benefits of this system both for research and clinical purposes.

Optimal Outcomes Reporting (OOR): A New Value-Based Metric for Outcome Reporting Following Cleft Palate Repair

OBJECTIVE

Palatoplasty outcome measurements vary widely among institutions. A standardized outcome metric would help provide quality benchmarks.

DESIGN

Retrospective review of primary palatoplasty patients from 2007 to 2013.

SETTING

Tertiary care children's hospital.

MAIN OUTCOME MEASURES

We created a novel conceptual quality metric called "OOR" (Optimal Outcome Reporting). Optimal Outcome Reporting is designed to reflect the percentage of patients with cleft palate who experience the best outcomes: one operation, velar competence by age 5 years, and no unintended palatal fistula.

RESULTS

Optimal Outcome Reporting was 72.3% (68/94). Eight patients had "suboptimal" outcomes for having undergone more than one operation. Eighteen patients failed for velar incompetence. No additional patients fell out of the algorithm for fistula. A significantly higher proportion of nonsyndromic patients demonstrated an "optimal" result compared to syndromic patients (61/80, 76.3% vs 7/14, 50.0%; P = .04). Patients who required more than one procedure had significantly more clinic visits (32.6 vs 14.9; P < .01) and accrued higher costs compared to "optimal" patients (US$34 019.88 vs US$15 357.25; P < .01).

CONCLUSIONS

Optimal Outcome Reporting represents a novel quality metric that can provide meaningful information for patients with cleft palate. Optimal Outcome Reporting utilization can help cleft centers adopt changes that matter to patients and their families. By allowing for cross-institutional comparisons in a clear and objective manner, OOR can promote competition, innovation, and value in cleft palate care.

Crowdsourcing to Assess Speech Quality Associated With Velopharyngeal Dysfunction

OBJECTIVE

To assess crowdsourced responses in the evaluation of speech outcomes in children with velopharyngeal dysfunction (VPD).

DESIGN

Fifty deidentified speech samples were compiled. Multiple pairwise comparisons obtained by crowdsourcing were used to produce a rank order of speech quality. Ratings of overall and specific speech characteristics were also collected. Twelve speech-language pathologists (SLPs) who specialize in VPD were asked to complete the same tasks. Crowds and experts completed each task on 2 separate occasions at least 1 week apart.

SETTING

On-line crowdsourcing platform.

PARTICIPANTS

Crowdsource raters were anonymous and at least 18 years of age, North American English speakers with self-reported normal hearing. Speech-language pathologists were recruited from multiple cleft/craniofacial teams.

INTERVENTIONS

None.

MAIN OUTCOME MEASURE(S)

Correlation of repeated assessments and comparison of crowd and SLP assessments.

RESULTS

We obtained 6331 lay person assessments that met inclusion criteria via crowdsourcing within 8 hours. The crowds provided reproducible Elo rankings of speech quality, ρ(48) = .89; P <.0001, and consistent ratings of intelligibility and acceptability (intraclass correlation coefficient [ICC] = .87 and .92) on repeated assessments. There was a significant correlation of those crowd rankings, ρ(10) = .86; P = .0003, and ratings (ICC = .75 and .79) with those of SLPs. The correlation of more specific speech characteristics by the crowds and SLPs was moderate to weak (ICC < 0.65).

CONCLUSIONS

Crowdsourcing shows promise as a rapid way to obtain large numbers of speech assessments. Reliability of repeated assessments was acceptable. Large groups of naive raters yield comparable evaluations of overall speech acceptability, intelligibility, and quality, but are not consistent with expert raters for specific speech characteristics such as resonance and nasal air emission.

Benchmarking Speech, Velopharyngeal Function Outcomes and Surgical Characteristics Following the Sommerlad Protocol and Palate Repair Technique

Objective:

To report speech and velopharyngeal function (VPF) outcomes, and surgical characteristics, at age 5 following early complete palate closure using the Sommerlad protocol.

Design:

A retrospective, descriptive, cross-sectional consecutive series.

Setting:

A regional twin site center; a district general hospital and tertiary children’s hospital.

Participants:

Between 1993 and 2006, 877 participants underwent surgery; 712 (81%) were eligible for inclusion; 391 (55%) were included, 321 (45%) excluded. Thirteen percent had bilateral cleft lip and palate (CLP), 40% unilateral CLP, and 47% isolated cleft palate.

Intervention:

Lip and vomerine flap repair at 3 months of age (BCLP, UCLP) with soft palate closure using radical muscle dissection and retropositioning at a mean age of 6.6 months, range 4 to 23 months, as described and undertaken by Sommerlad.

Outcome Measures:

Velopharyngeal Composite (VPC) CAPS-A and articulation summary scores derived from analysis using the Cleft Audit Protocol for Speech-Augmented.

Results:

A VPC-SUM CAPS-A score of “0” was found in 97% reflecting adequate VPF; 73% had no cleft articulation difficulties. The BCLP group had the poorest articulation. Secondary speech surgery rate was 2.6% and 10.7% had fistula repair. Levator muscle quality and degree of retropositioning was associated with speech outcomes related to VPF. 2.6% had current features of VPI.

Conclusions:

Complete palate closure by around 6 months of age by the surgeon who described the technique led to excellent perceived VPF and very good speech outcomes at age 5, with less speech therapy and secondary speech surgery than commonly reported. These findings serve as a benchmark for the Sommerlad protocol and technique.

Reliability of Speech Variables and Speech-Related Quality Indicators in the Swedish Cleft Lip and Palate Registry

OBJECTIVE

To assess the reliability of speech variables and speech-related quality indicators in the Swedish quality registry for cleft lip and palate (CLP).

DESIGN

Retrospective study.

SETTING

Primary care university hospitals.

PARTICIPANTS

Fifty-two 5-year-old children with unilateral CLP and 41 with bilateral CLP.

MAIN OUTCOME MEASURES

Registry data for "percent nonoral errors" and "perceived velopharyngeal competence" (VPC) were compared to reassessments by 4 independent judges based on audio recordings. Interjudge agreement for "percent consonants correct" (PCC) and the reliability of 3 quality indicators were also assessed. Agreement was calculated with single measures intraclass correlation coefficient (ICC) for articulation outcomes, quadratic weighted κ and ICC for VPC, and percentage agreement and κ for quality indicators.

RESULTS

When the agreement between registry data and the judges' reassessments was assessed, the ICC was 0.79 for percent nonoral errors. For VPC, the κ coefficient was 0.66 to 0.75 and the ICC was 0.73. Interjudge agreement for PCC calculated with ICC was 0.85. For the quality indicator "proportion of children with ≥86% correct consonants," all 4 judges were in agreement for 72% of the cases. For "proportion of children without nonoral speech errors" and "proportion of children with competent or marginally incompetent velopharyngeal function," the agreement between registry data and the 4 judges was 89% and 85%, respectively.

CONCLUSIONS

The results indicate that registry data on PCC, percent nonoral errors, VPC, and the quality indicators "proportion of children without nonoral speech errors" and "proportion of children with competent or marginally incompetent velopharyngeal function" are reliable.

Five-Fold Variation Among Surgeons and Hospitals in the Use of Secondary Palate Surgery

OBJECTIVE

To identify child-, surgeon- and hospital-specific factors at the time of primary cleft palate repair that are associated with the use of secondary palate surgery.

DESIGN

Retrospective cohort study.

SETTING

Forty-nine pediatric hospitals.

PARTICIPANTS

Children who underwent cleft palate repair between 1998 and 2015.

MAIN OUTCOME MEASURE

Time from primary cleft palate repair to secondary palate surgery.

RESULTS

By 5 years after the primary palate repair, 27.5% of children had undergone secondary palate surgery. In multivariable analysis, cleft type and age at primary palate repair were both associated with secondary surgery ( P < .01). Children with unilateral cleft lip and palate had a 1.69-fold increased hazard of secondary surgery (95% confidence interval [CI]: 1.54-1.85) compared to children with cleft palate alone. Primary palate repair before 9 months had a 3.99-fold increased hazard of secondary surgery (95% CI: 3.39-4.07) compared to repair at 16 to 24 months of age. After adjusting for cleft type, age at repair, and procedure volume, there remained substantial variation in secondary surgery use among surgeons and hospitals ( P < .01). For children with isolated cleft palate, the predicted proportion of children undergoing secondary surgery within 5 years of primary repair ranged from 8.5% to 46.0% across surgeons and 9.1% to 49.4% across hospitals.

CONCLUSIONS

There are substantial differences among surgeons and hospitals in the rates of secondary palate surgery. Further work is needed to identify causes for this variation among providers and develop interventions to reduce the need for secondary surgery.