Crowdsourcing to Assess Speech Quality Associated With Velopharyngeal Dysfunction

Assessing novelty, feasibility and value of creative ideas with an unsupervised approach using GPT-4

Creativity is defined by three key factors: novelty, feasibility and value. While many creativity tests focus primarily on novelty, they often neglect feasibility and value, thereby limiting their reflection of real-world creativity. In this study, we employ GPT-4, a large language model, to assess these three dimensions in a Japanese-language Alternative Uses Test (AUT). Using a crowdsourced evaluation method, we acquire ground truth data for 30 question items and test various GPT prompt designs. Our findings show that asking for multiple responses in a single prompt, using an 'explain first, rate later' design, is both cost-effective and accurate (r = .62, .59 and .33 for novelty, feasibility and value, respectively). Moreover, our method offers comparable accuracy to existing methods in assessing novelty, without the need for training data. We also evaluate additional models such as GPT-4 Turbo, GPT-4 Omni and Claude 3.5 Sonnet. Comparable performance across these models demonstrates the universal applicability of our prompt design. Our results contribute a straightforward platform for instant AUT evaluation and provide valuable ground truth data for future methodological research.

Crowdsourced Perceptual Ratings of Voice Quality in People With Parkinson's Disease Before and After Intensive Voice and Articulation Therapies: Secondary Outcome of a Randomized Controlled Trial

PURPOSE

Limited research has examined the suitability of crowdsourced ratings to measure treatment effects in speakers with Parkinson's disease (PD), particularly for constructs such as voice quality. This study obtained measures of reliability and validity for crowdsourced listeners' ratings of voice quality in speech samples from a published study. We also investigated whether aggregated listener ratings would replicate the original study's findings of treatment effects based on the Acoustic Voice Quality Index (AVQI) measure.

METHOD

This study reports a secondary outcome measure of a randomized controlled trial with speakers with dysarthria associated with PD, including two active comparators (Lee Silverman Voice Treatment [LSVT LOUD] and LSVT ARTIC), an inactive comparator (untreated PD), and a healthy control group. Speech samples from three time points (pretreatment, posttreatment, and 6-month follow-up) were presented in random order for rating as "typical" or "atypical" with respect to voice quality. Untrained listeners were recruited through the Amazon Mechanical Turk crowdsourcing platform until each sample had at least 25 ratings.

RESULTS

Intrarater reliability for tokens presented repeatedly was substantial (Cohen's κ = .65-.70), and interrater agreement significantly exceeded chance level. There was a significant correlation of moderate magnitude between the AVQI and the proportion of listeners classifying a given sample as "typical." Consistent with the original study, we found a significant interaction between group and time point, with the LSVT LOUD group alone showing significantly higher perceptually rated voice quality at posttreatment and follow-up relative to the pretreatment time point.

CONCLUSIONS

These results suggest that crowdsourcing can be a valid means to evaluate clinical speech samples, even for less familiar constructs such as voice quality. The findings also replicate the results of the study by Moya-Galé et al. (2022) and support their functional relevance by demonstrating that the effects of treatment measured acoustically in that study are perceptually apparent to everyday listeners.

The Feasibility of Cross-Linguistic Speech Evaluation in the Care of International Cleft Palate Patients

ABSTRACT

Many patients with cleft palate in developing countries never receive postoperative speech assessment or therapy. The use of audiovisual recordings could improve access to post-repair speech care. The present study evaluated whether English-speaking speech-language pathologists (SLPs) could assess cleft palate patients speaking an unfamiliar language (Tamil) using recorded media. Recordings obtained from Tamil-speaking participants were rated by 1 Tamil-speaking SLP and 3 English-speaking SLPs. Ratings were analyzed for inter-rater reliability and scored for percent correct. Accuracy of the English SLPs was compared with independent t tests and Analysis of Variance. Sixteen participants (mean age 14.5 years, standard deviation [SD] 7.4 years; mean age of surgery of 2.7 years, SD 3.7 years; time since surgery: 10.8 years, SD 5.7 years) were evaluated. Across the 4 SLPs, 5 speech elements were found to have moderate agreement, and the mean kappa was 0.145 (slight agreement). Amongst the English-speaking SLPs, 10 speech elements were found to have substantial or moderate agreement, and the mean kappa was 0.333 (fair agreement). Speech measures with the highest inter-rater reliability were hypernasality and consonant production errors. The average percent correct of the English SLPs was 60.7% (SD 20.2%). English SLPs were more accurate if the participant was female, under eighteen, bilingual, or had speech therapy. The results demonstrate that English SLPs without training in a specific language (Tamil) have limited potential to assess speech elements accurately. This research could guide training interventions to augment the ability of SLPs to conduct cross-linguistic evaluations and improve international cleft care by global health teams.

Crowdsourcing as a tool in the clinical assessment of intelligibility in dysarthria: How to deal with excessive variation

PURPOSE

Independent laypersons are essential in the assessment of intelligibility in persons with dysarthria (PWD), as they reflect intelligibility limitations in the most ecologically valid way, without being influenced by familiarity with the speaker. The present work investigated online crowdsourcing as a convenient method to involve lay people as listeners, with the objective of exploring how to constrain the expected variability of crowd-based judgements to make them applicable in clinical diagnostics.

METHOD

Intelligibility was assessed using a word transcription task administered via crowdsourcing. In study 1, speech samples of 23 PWD were transcribed by 18 crowdworkers each. Four methods of aggregating the intelligibility scores of randomly sampled panels of 4 to 14 listeners were compared for accuracy, i.e. the stability of the resulting intelligibility estimates across different panels, and their validity, i.e. the degree to which they matched data obtained under controlled laboratory conditions ("gold standard"). In addition, we determined an economically acceptable number of crowdworkers per speaker which is needed to obtain accurate and valid intelligibility estimates. Study 2 examined the robustness of the chosen aggregation method against downward outliers due to spamming in a larger sample of 100 PWD.

RESULTS

In study 1, an interworker aggregation method based on negative exponential weightings of the scores as a function of their distance from the "best" listener's score (exponentially weighted mean) outperformed three other methods (median value, arithmetic mean, maximum). Under cost-benefit considerations, an optimum panel size of 9 crowd listeners per examination was determined. Study 2 demonstrated the robustness of this aggregation method against spamming crowd listeners.

CONCLUSION

Though intelligibility data collected through online crowdsourcing are noisy, accurate and valid intelligibility estimates can be obtained by appropriate aggregation of the raw data. This makes crowdsourcing a suitable method for incorporating real-world perspectives into clinical dysarthria assessment.