Quantifying the Uncertainty of Parameters Measured in Spontaneous Speech of Speakers With Dementia

Toward Generalizable Machine Learning Models in Speech, Language, and Hearing Sciences: Estimating Sample Size and Reducing Overfitting

PURPOSE

Many studies using machine learning (ML) in speech, language, and hearing sciences rely upon cross-validations with single data splitting. This study's first purpose is to provide quantitative evidence that would incentivize researchers to instead use the more robust data splitting method of nested k-fold cross-validation. The second purpose is to present methods and MATLAB code to perform power analysis for ML-based analysis during the design of a study.

METHOD

First, the significant impact of different cross-validations on ML outcomes was demonstrated using real-world clinical data. Then, Monte Carlo simulations were used to quantify the interactions among the employed cross-validation method, the discriminative power of features, the dimensionality of the feature space, the dimensionality of the model, and the sample size. Four different cross-validation methods (single holdout, 10-fold, train-validation-test, and nested 10-fold) were compared based on the statistical power and confidence of the resulting ML models. Distributions of the null and alternative hypotheses were used to determine the minimum required sample size for obtaining a statistically significant outcome (5% significance) with 80% power. Statistical confidence of the model was defined as the probability of correct features being selected for inclusion in the final model.

RESULTS

ML models generated based on the single holdout method had very low statistical power and confidence, leading to overestimation of classification accuracy. Conversely, the nested 10-fold cross-validation method resulted in the highest statistical confidence and power while also providing an unbiased estimate of accuracy. The required sample size using the single holdout method could be 50% higher than what would be needed if nested k-fold cross-validation were used. Statistical confidence in the model based on nested k-fold cross-validation was as much as four times higher than the confidence obtained with the single holdout-based model. A computational model, MATLAB code, and lookup tables are provided to assist researchers with estimating the minimum sample size needed during study design.

CONCLUSION

The adoption of nested k-fold cross-validation is critical for unbiased and robust ML studies in the speech, language, and hearing sciences.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.25237045.

What Drives Task Performance in Animal Fluency in Individuals Without Dementia? The SMART-MR Study

PURPOSE

In this study, we aim to understand whether and how performance in animal fluency (i.e., total correct word count) relates to linguistic levels and/or executive functions by looking at sequence information and item-level metrics (i.e., clusters, switches, and word properties).

METHOD

Seven hundred thirty-one Dutch-speaking individuals without dementia from the Second Manifestations of ARTerial disease-Magnetic Resonance study responded to an animal fluency task (120 s). We obtained cluster size and number of switches for the task, and eight different word properties for each correct word produced. We detected variables that determine total word count with random forests, and used conditional inference trees to assess points along the scales of such variables, at which total word count changes significantly.

RESULTS

Number of switches, average cluster size, lexical decision response times, word frequency, and concreteness determined total correct word count in animal fluency. People who produced more correct words produced more switches and bigger clusters. People who produced fewer words produced fewer switches and more frequent words.

CONCLUSIONS

Concurrent with existing literature, individuals without dementia rely on language and executive functioning to produce words in animal fluency. The novelty of our work is that such results were shown based on a data-driven approach using sequence information and item-level metrics.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.23713269.