Reliability of Speech Variables and Speech-Related Quality Indicators in the Swedish Cleft Lip and Palate Registry

Reliability of Variables of Consonant Production and Velopharyngeal Competence in 10-year-olds with Cleft Palate Using Multiple Raters

OBJECTIVE

To assess the reliability of speech data and speech-related quality indicators in the Swedish quality registry for cleft lip and palate (CLP) at 10 years of age.

DESIGN

Retrospective study.

SETTING

University hospitals.

PARTICIPANTS

One hundred twenty-one 10-year-olds with unilateral or bilateral CLP.

MAIN OUTCOME MEASURES

Six independent raters reassessed audio recordings for comparison with registry data. For calculation of agreement, the single measures intraclass correlation coefficient (ICC) was used for percentage of consonants correct (PCC) and non-oral speech errors, quadratic weighted kappa for velopharyngeal competence (VPC), and percentage agreement and kappa for quality indicators. The results of the three to four raters with the highest intra-rater and inter-rater reliability were used for comparison with registry data.

RESULTS

There was excellent agreement between registry data and reassessments for PCC (ICC, 0.93) and percentage of non-oral errors (ICC, 0.80). For VPC, one rater and registry data had good agreement (k, 0.704); the remaining cases had fair agreement (k, 0.476-0.554). The percentage agreement between registry data and reassessments for quality indicators ranged from fair to excellent. When calculated with kappa, agreement was good to excellent (mean of all k values, 0.67-0.70).

CONCLUSIONS

The CLP registry variables PCC and percentage of non-oral errors and the quality indicators without non-oral speech errors and competent/marginally incompetent velopharyngeal function are reliable for use in clinical audits and research of 10-year-olds. The three-tier ratings of VPC have weaker reliability but can still be useful in more detailed analyses if interpreted with caution.

Using Artificial Intelligence for Assessment of Velopharyngeal Competence in Children Born With Cleft Palate With or Without Cleft Lip

OBJECTIVE

Development of an AI tool to assess velopharyngeal competence (VPC) in children with cleft palate, with/without cleft lip.

DESIGN

Innovation of an AI tool using retrospective audio recordings and assessments of VPC.

SETTING

Two datasets were used. The first, named the SR dataset, included data from follow-up visits to Skåne University Hospital, Sweden. The second, named the SC + IC dataset, was a combined dataset (SC + IC dataset) with data from the Scandcleft randomized trials across five countries and an intercenter study performed at six Swedish CL/P centers.

PARTICIPANTS

SR dataset included 153 recordings from 162 children, and SC + IC dataset included 308 recordings from 399 children. All recordings were from ages 5 or 10, with corresponding VPC assessments.

INTERVENTIONS

Development of two networks, a convolutional neural network (CNN) and a pre-trained CNN (VGGish). After initial testing using the SR dataset, the networks were re-tested using the SC + IC dataset and modified to improve performance.

MAIN OUTCOME MEASURES

Accuracy of the networks' VPC scores, with speech and language pathologistś scores seen as the true values. A three-point scale was used for VPC assessments.

RESULTS

VGGish outperformed CNN, achieving 57.1% accuracy compared to 39.8%. Minor adjustments in data pre-processing and network characteristics improved accuracies.

CONCLUSIONS

Network accuracies were too low for the networks to be useful alternatives for VPC assessment in clinical practice. Suggestions for future research with regards to study design and dataset optimization were discussed.

Foreign-born 5-year-old children with cleft palate had poorer speech outcomes than their native-born peers

AIM

Speech difficulties are common in children with cleft palate, but research on foreign-born children is limited. This study aimed to compare speech outcomes, surgery and speech intervention in 5-year-old foreign-born and Swedish-born children with cleft palate with or without cleft lip.

METHODS

This retrospective study analysed data from the Swedish cleft lip and palate registry for children born between 2009 and 2016 using Pearson's Chi-squared test and binary logistic regression.

RESULTS

Among 160 foreign-born (106 boys, 54 girls) and 847 Swedish-born (479 boys, 368 girls) 5-year-olds, foreign-born children had significantly lower rates of sufficient velopharyngeal competence (77% vs. 86%), age-appropriate consonant production (28% vs. 60%), and speech without non-oral speech errors (70% vs. 86%). Differences remained after adjustment for cleft type, gender and additional diagnosed conditions. After further adjustments for age at completed primary palatal surgery, differences in age-appropriate consonant production and speech without non-oral speech errors remained significant. Foreign-born children underwent completed primary palatal surgery at older ages and received more secondary palatal surgery and speech intervention than Swedish-born peers.

CONCLUSION

Foreign-born children showed poorer speech outcomes than Swedish-born peers, despite more secondary palatal surgery and speech intervention. Age at completed primary palatal surgery could partly explain these differences.

Comment on Ombashi, van der Goes, Versnel, Khonsari, van der Molen: guidance to develop a multidisciplinary, international, pediatric registry: a systematic review, Orphanet Journal of Rare diseases, 2023

Recently, Ombashi et al. published a systematic review aiming to identify the pitfalls in the development and implementation as well as factors influencing long-term success of a multidisciplinary, international registry for cleft care on a global scale. The purpose of this letter to the editor is to highlight that the review failed to include the Swedish quality registry for patients born with cleft lip and palate, which fulfils the inclusion criteria. The Swedish cleft lip and palate registry is multidisciplinary, has a high coverage and reporting degree, and most outcome measures have been checked for reliability and validity. It is regularly used for open comparisons between treatment centers. Several research studies have been published based on the Swedish cleft lip and palate registry, and more are ongoing. The information we provide about the Swedish cleft lip and palate registry complements and expands the information of the results reported by Ombashi et al. in their research.

Reliability of data on percent consonants correct and its associated quality indicator in the Swedish cleft lip and palate registry

BACKGROUND

Data in national health care quality registries must be valid and reliable in order to enable open comparisons of results.

AIM

To assess the reliability of data on percent consonants correct (PCC) and its associated quality indicator ≥86% correct consonants in the Swedish quality registry for patients born with cleft lip and palate (CLP) registry.

METHODS

Six independent speech-language pathologists re-assessed the audio recordings of 96 five-year-olds with PCC data in the CLP registry. Target consonants of a single-word picture-naming test were phonetically transcribed, and PCC was calculated. The reliability of PCC data was assessed with the intraclass correlation coefficient (ICC). The reliability of the quality indicator ≥86% correct consonants was assessed with point-by-point percentage agreement and Cohen's kappa.

RESULTS

Intra- and inter-judge agreement for PCC was excellent with ICCs above 0.9, and so was the agreement of data from the CLP registry and the six judges' re-assessments. The percentage agreement between all judges and the CLP registry for the quality indicator ≥86% correct consonants was poor (67%). However, in 88% of the cases, results from four judges and the CLP registry agreed, corresponding to good agreement. The mean of all kappa values for six judges and the CLP registry corresponded to good agreement (0.72).

CONCLUSIONS

The results indicate the PCC data in the CLP registry and the quality indicator ≥86% correct consonants to be reliable. When differences in outcome between treatment centres are detected, the raw data collected should always be re-examined before drawing definitive conclusions.

Speech in 7- and 10-year-olds born with a unilateral cleft lip and palate: a continued prospective Swedish intercentre study

The aim of this study was to report longitudinal speech results in consecutively selected children from each of the six cleft centres in Sweden and to compare the results between centres. The children were born with a non-syndromic unilateral cleft lip and palate, and results from the same cohort at 5 years of age have previously been reported. Background data on medical care in terms of surgery, speech therapy, and hearing between 5 and 10 years of age were collected. Speech recordings of 56 children at 7 years and 54 at 10 years of age were blindly and independently assessed by four speech-language pathologists experienced in cleft palate speech. This resulted in measures of percent consonant correct (PCC) and perceived velopharyngeal competence rated on a three-tier scale. No statistically significant differences were found between centres. PCC scores at 7 years of age ranged from 44-100% (median 97.5) and at 10 years of age from 86-100% (median 100). Competent or marginally incompetent velopharyngeal function was found in 95% of the 7-year-olds and 98% of the 10-year-olds. Speech results were slightly better than previous reports of speech in children born with a unilateral cleft lip and palate.

Using "Real-World Data" to Study Cleft Lip/Palate Care: An Exploration of Speech Outcomes from a Multi-Center US Learning Health Network

OBJECTIVE

To assess the ability of a cleft-specific multi-site learning health network registry to describe variations in cleft outcomes by cleft phenotypes, ages, and treatment centers. Observed variations were assessed for coherence with prior study findings.

DESIGN

Cross-sectional analysis of prospectively collected data from 2019-2022.

SETTING

Six cleft treatment centers collected data systematically during routine clinic appointments according to a standardized protocol.

PARTICIPANTS

714 English-speaking children and adolescents with non-syndromic cleft lip/palate.

INTERVENTION

Routine multidisciplinary care and systematic outcomes measurement by cleft teams.

OUTCOME MEASURES

Speech outcomes included articulatory accuracy measured by Percent Consonants Correct (PCC), velopharyngeal function measured by Velopharyngeal Competence (VPC) Rating Scale (VPC-R), intelligibility measured by caregiver-reported Intelligibility in Context Scale (ICS), and two CLEFT-Q™ surveys, in which patients rate their own speech function and level of speech distress.

RESULTS

12year-olds exhibited high median PCC scores (91-100%), high frequency of velopharyngeal competency (62.50-100%), and high median Speech Function (80-91) relative to younger peers parsed by phenotype. Patients with bilateral cleft lip, alveolus, and palate reported low PCC scores (51-91%) relative to peers at some ages and low frequency of velopharyngeal competency (26.67%) at 5 years. ICS scores ranged from 3.93-5.0 for all ages and phenotypes. Speech Function and Speech Distress were similar across phenotypes.

CONCLUSIONS

This exploration of speech outcomes demonstrates the current ability of the cleft-specific registry to support cleft research efforts as a source of "real-world" data. Further work is focused on developing robust methodology for hypothesis-driven research and causal inference.

Long-term speech outcome after anterior distraction osteogenesis of the maxilla in patients with cleft lip and palate

Advancement of the maxilla may increase the distance between the soft palate and the posterior pharyngeal wall in patients with cleft lip and palate, implying a risk of velopharyngeal dysfunction. The aim was to evaluate long-term speech outcome in a consecutive series of patients treated with distraction osteogenesis (DO). Fourteen out of the 16 patients agreed to participate. A long-term speech follow-up was performed 1.5 to 13.5 years after DO. For two participants, audio recordings before DO were missing, and for another one, it was incomplete. The percentage of consonants correct (PCC) based on phonetic transcription and perceived velopharyngeal competence rated on a three-point scale were assessed before and after DO by three independent judges, based on audio recordings of reading of standardised sentences. Also, the participants were asked how they perceived their speech after DO. Changes in PCC were insignificant. Four participants perceived deteriorated speech related to DO. In two cases, the subjective deterioration did not correlate to results from perceptual assessment. In two others, the subjective deterioration correlated with the perceptual assessment, and the velopharyngeal function was judged as being incompetent after DO. After secondary velopharyngeal surgery, velopharyngeal function improved to competent in one case and marginally incompetent in the other. The results need to be interpreted with caution due to methodological limitations but indicate that some patients develop deteriorated velopharyngeal function after DO. The impact on articulation needs to be further explored. It is important that patients are informed before treatment of the risk of velopharyngeal dysfunction after DO.

Inter-centre comparison of data on surgery and speech outcomes at 5 years of age based on the Swedish quality registry for patients born with cleft palate with or without cleft lip

BACKGROUND

The objective of the Swedish cleft lip and palate registry (CLP registry) is to promote quality control, research and improvement of treatment, by the comparison of long-term results. The aim was to compare data from the CLP registry among the six treatment centres, regarding data on surgery and speech outcomes at 5 years of age.

METHODS

The participants were 430 children born in Sweden from 2009 to 2014, with cleft palate with or without cleft lip and without known syndromes and/or additional malformations. The number of primary and secondary palatal surgeries up to 5 years of age, timing of the last primary palatal surgery, percentage consonants correct, percentage non-oral speech errors and perceived velopharyngeal competence at 5 years were assessed. Multivariable binary logistic regression adjusted for sex and cleft type was used to compare results between the six centres.

RESULTS

At one centre (centre 4), the palate was closed in one to three stages, and at the remaining centres in one or two stages. At centre 4, more children underwent a higher number of palatal surgeries, and the last primary palatal surgery was performed at a higher age. Children in centre 4 were also less likely to achieve ≥86% correct consonants (OR = 0.169, P = < 0.001), have no non-oral speech errors (OR = 0.347, P = < 0.001), or have competent or marginally incompetent velopharyngeal competence (OR = 0.244, P = < 0.001), compared to the average results of the other centres. No clear association between patient volume and speech outcome was observed.

CONCLUSIONS

The results indicated the risk of a negative speech result if the last primary palatal surgery was performed after 25 months of age. Whether the cleft in the palate was closed in one or two stages did not affect speech outcome. The Swedish CLP registry can be used for open comparisons of treatment results to provide the basis for improvements of treatment methods. If deviating negative results are seen consistently at one centre, this information should be acted upon by further investigation and analysis, making changes to the treatment protocol as needed.

Speech in a consecutive series of children born with cleft lip and palate with and without syndromes and/or additional malformations

Background

When evaluating speech in children with cleft palate with or without cleft lip (CP/L), children with known syndromes and/or additional malformations (CP/L+) are usually excluded. The aim of this study was to present speech outcome of a consecutive series of 5-year-olds born with CP/L, and to compare speech results of children with CP/L + and children with CP/L without known syndromes and/or additional malformations (CP/L-).

Methods

One hundred 5-year-olds (20 with CP/L+; 80 with CP/L-) participated. All children were treated with primary palatal surgery in one stage with the same procedure for muscle reconstruction. Three independent judges performed phonetic transcriptions and rated perceived velopharyngeal competence from audio recordings. Based on phonetic transcriptions, percent consonants correct (PCC) and percent non-oral errors were investigated. Group comparisons were performed.

Results

In the total group, mean PCC was 88.2 and mean percent non-oral errors 1.5. The group with bilateral cleft lip and palate (BCLP) had poorer results on both measures compared to groups with other cleft types. The average results of PCC and percent non-oral errors in the CP/L + group indicated somewhat poorer speech, but no significant differences were observed. In the CP/L + group, 25 % were judged as having incompetent velopharyngeal competence, compared to 15 % in the CP/L- group.

Conclusions

The results indicated relatively good speech compared to speech of children with CP/L in previous studies. Speech was poorer in many children with more extensive clefts. No significant differences in speech outcomes were observed between CP/L + and CP/L- groups.

Reliability of Outcome Measures to Assess Consonant Proficiency Following Cleft Palate Speech Intervention: The Percentage of Consonants Correct Metric and the Probe Scoring System

Purpose This study compared the inter- and intrarater reliability of the percentage of consonants correct (PCC) metrics and the probe scoring system between an experienced and a less experienced rater and between two experienced raters. In addition, these outcome measures' ability to reflect changes following speech intervention was measured. Method During Phase 1, two raters (Rater 1 with 5 years of experience in cleft-related speech disorders and Rater 2 with limited experience in cleft-related speech disorders) independently assessed 134 speech samples at the word and sentence levels, which were collected on different data points before, during, and following a cleft palate speech intervention. During Phase 2, a third rater (with 8 years of experience) analyzed 34 speech samples. The percentage of consonants correct-revised, the percentage of correct places and manners, and probe scores at the word and sentence levels were measured. Results Poor-to-moderate interreliability between Raters 1 and 2 was found due to differences in error classification. Interrater reliability between Raters 1 and 3 was very good for both the PCC metrics and the probe scores. The interrater reliability for the amount of targets elicited was lower compared to the interrater reliability for the amount of targets correct. The probe scoring system demonstrated a greater ability to detect changes toward the correct production of the target consonant compared to the PCC metrics. Conclusions Having an experience with the assessment of cleft-related speech disorders is a crucial factor to gain reliable results. The interrater reliability for the PCC metrics and the probe scoring system between two experienced raters did not differ, suggesting that both outcome measures can be used in cleft palate speech intervention studies. Despite the ability of the probe scoring system to detect changes, further research should provide insight in the benefits of this system both for research and clinical purposes.

Coverage, reporting degree and design of the Swedish quality registry for patients born with cleft lip and/or palate

Background

The objective of the Swedish cleft lip and palate (CLP) registry is to promote quality control, research and improvement of treatment, by comparison of the long-term results of surgery, orthodontics and speech from all six Swedish CLP centres. The purpose of the study was to investigate the coverage and reporting degree of the Swedish CLP registry, and to describe the design of the registry and discuss questions of reliability and validity of the data included.

Methods

All six Swedish CLP centres participate in the registry. All children in Sweden with cleft lip and/or cleft palate, born from 2009 onwards, are included in the registry. Baseline data such as cleft type (ICD-10 diagnosis), heredity, birth weight and additional deformities and/or syndromes, as well as pre-surgical treatment, are recorded at first visit. Data on surgical treatment are recorded continuously. Treatment outcome regarding dentofacial development and speech are recorded at follow-ups at 5, 10, 16 and 19 years of age. Data on dentofacial development are also recorded 1 year after orthognathic surgery. In addition, data on babbling and speech are recorded at 18 months of age. Coverage degree and reporting degree of surgery was assessed by comparison with registrations in the Swedish Central patient registry. Reporting degree of orthodontic and speech registrations at 5 years of age was assessed by comparison with registrations at baseline.

Results

The average coverage degree for children born 2009 to 2018 was 95.1%. For cleft-related surgeries, the average reporting degree was 92.4%. Average reporting degree of orthodontic registrations and speech registrations at age 5 years was 92 and 97.5% respectively.

Conclusion

In order to achieve valid and reliable data in a healthcare quality registry, the degree of coverage and reporting needs to be high, the variables included should be limited and checked for reliability, and the professionals must calibrate themselves regularly. The Swedish CLP registry fulfils these requirements.