A psychometric investigation of "macroscopic" speech measures for clinical and psychological science

Evaluating speech latencies during structured psychiatric interviews as an automated objective measure of psychomotor slowing

We sought to derive an objective measure of psychomotor slowing from speech analytics during a psychiatric interview to avoid potential burden of dedicated neurophysiological testing. Speech latency, which reflects response time between speakers, shows promise from the literature. Speech data was obtained from 274 subjects with a diagnosis of bipolar I depression enrolled in a randomized, doubleblind, 6-week phase 2 clinical trial. Audio recordings of structured Montgomery-Åsberg Depression Rating Scale (MADRS) interviews at 6 time points were examined (k = 1,352). We evaluated speech latencies, and other aspects of speech, for temporal stability, convergent validity, sensitivity/responsivity to clinical change, and generalization across seven socio-linguistically diverse countries. Speech latency was minimally associated with demographic features, and explained nearly a third of the variance in depression (categorically defined). Speech latency significantly decreased as depression symptoms improved over time, explaining nearly 20 % of variance in depression remission. Classification for differentiating people with versus without concurrent depression was high (AUCs > 0.85) both cross-sectionally and longitudinally. Results replicated across countries. Other speech features offered modest incremental contribution. Neurophysiological speech parameters with face validity can be derived from psychiatric interviews without the added patient burden of additional testing.

Lexical Speech Features of Spontaneous Speech in Older Persons With and Without Cognitive Impairment: Reliability Analysis

Background

Speech analysis data are promising digital biomarkers for the early detection of Alzheimer disease. However, despite its importance, very few studies in this area have examined whether older adults produce spontaneous speech with characteristics that are sufficiently consistent to be used as proxy markers of cognitive status.

Objective

This preliminary study seeks to investigate consistency across lexical characteristics of speech in older adults with and without cognitive impairment.

Methods

A total of 39 older adults from a larger, ongoing study (age: mean 81.1, SD 5.9 years) were included. Participants completed neuropsychological testing and both picture description tasks and expository tasks to elicit speech. Participants with T-scores of ≤40 on ≥2 cognitive tests were categorized as having mild cognitive impairment (MCI). Speech features were computed automatically by using Python and the Natural Language Toolkit.

Results

Reliability indices based on mean correlations for picture description tasks and expository tasks were similar in persons with and without MCI (with r ranging from 0.49 to 0.65 within tasks). Intraindividual variability was generally preserved across lexical speech features. Speech rate and filler rate were the most consistent indices for the cognitively intact group, and speech rate was the most consistent for the MCI group.

Conclusions

Our findings suggest that automatically calculated lexical properties of speech are consistent in older adults with varying levels of cognitive impairment. These findings encourage further investigation of the utility of speech analysis and other digital biomarkers for monitoring cognitive status over time.

Alogia and pressured speech do not fall on a continuum of speech production using objective speech technologies

Speech production is affected in a variety of serious mental illnesses (SMI; e.g., schizophrenia, unipolar depression, bipolar disorders) and at its extremes can be observed in the gross reduction of speech (e.g., alogia) or increase of speech (e.g., pressured speech). The present study evaluated whether clinically-rated alogia and pressured speech represent antithetical constructs when analyzed using objective metrics of speech production. We examined natural speech using acoustic and natural language processing features from two archival studies using several different speaking tasks and a combined 107 patients meeting criteria for SMI. Contrary to expectations, we did not find that alogia and pressured speech presented as opposing ends of a speech production continuum. Objective speech markers were associated with clinically rated alogia but not pressured speech, and these results were consistent across speaking tasks and studies. Implications for our understanding of speech production symptoms in SMI are discussed, as well as implications for Natural Language Processing and digital phenotyping efforts more generally.

Automated assessment of speech production and prediction of MCI in older adults

The population of older adults is growing dramatically and, with it comes increased prevalence of neurological disorders, including Alzheimer's disease (AD). Though existing cognitive screening tests can aid early detection of cognitive decline, these methods are limited in their sensitivity and require trained administrators. The current study sought to determine whether it is possible to identify persons with mild cognitive impairment (MCI) using automated analysis of spontaneous speech. Participants completed a brief neuropsychological test battery and a spontaneous speech task. MCI was classified using established research criteria, and lexical-semantic features were calculated from spontaneous speech. Logistic regression analyses compared the predictive ability of a commonly-used cognitive screening instrument (the Modified Mini Mental Status Exam, 3MS) and speech indices for MCI classification. Testing against constant-only logistic regression models showed that both the 3MS [χ2(1) = 6.18, p = .013; AIC = 41.46] and speech indices [χ2(16) = 32.42, p = .009; AIC = 108.41] were able to predict MCI status. Follow-up testing revealed the full speech model better predicted MCI status than did 3MS (p = .049). In combination, the current findings suggest that spontaneous speech may have value as a potential screening measure for the identification of cognitive deficits, though confirmation is needed in larger, prospective studies.

Improving the Applicability of AI for Psychiatric Applications through Human-in-the-loop Methodologies

OBJECTIVES

Machine learning (ML) and natural language processing have great potential to improve efficiency and accuracy in diagnosis, treatment recommendations, predictive interventions, and scarce resource allocation within psychiatry. Researchers often conceptualize such an approach as operating in isolation without much need for human involvement, yet it remains crucial to harness human-in-the-loop practices when developing and implementing such techniques as their absence may be catastrophic. We advocate for building ML-based technologies that collaborate with experts within psychiatry in all stages of implementation and use to increase model performance while simultaneously increasing the practicality, robustness, and reliability of the process.

METHODS

We showcase pitfalls of the traditional ML framework and explain how it can be improved with human-in-the-loop techniques. Specifically, we applied active learning strategies to the automatic scoring of a story recall task and compared the results to a traditional approach.

RESULTS

Human-in-the-loop methodologies supplied a greater understanding of where the model was least confident or had knowledge gaps during training. As compared to the traditional framework, less than half of the training data were needed to reach a given accuracy.

CONCLUSIONS

Human-in-the-loop ML is an approach to data collection and model creation that harnesses active learning to select the most critical data needed to increase a model's accuracy and generalizability more efficiently than classic random sampling would otherwise allow. Such techniques may additionally operate as safeguards from spurious predictions and can aid in decreasing disparities that artificial intelligence systems otherwise propagate.

Ambulatory audio and video recording for digital phenotyping in schizophrenia: Adherence & data usability

Ambulatory audio and video recording provides a wealth of information which can be used for a broad range of applications, including digital phenotyping, telepsychiatry, and telepsychology. However, these technologies are in their infancy, and guidelines for their use and analysis have yet to be established. The current project used ambulatory assessment data from individuals with schizophrenia (N = 52) and controls (N = 55) over a week to assess factors influencing sufficiency and useability of video and audio data. Logistic multilevel models examined the effect of relevant variables on video provision and video quality. There was no difference by group in video provision or quality. Videos were less likely to be provided later in the study and later in the day. Video quality was lower later in the day, particularly for controls. Participants were more likely to provide videos if alone or at home than in other settings. Black participants were less likely to have analyzable video frames than White participants. These results suggest potential racial disparities in camera technologies and/or facial analysis algorithms. Implications of these findings and recommendations for future study development, such as instructions to provide to participants to optimize video quality, are discussed.

High Predictive Accuracy of Negative Schizotypy With Acoustic Measures

Negative schizotypal traits potentially can be digitally phenotyped using objective vocal analysis. Prior attempts have shown mixed success in this regard, potentially because acoustic analysis has relied on small, constrained feature sets. We employed machine learning to (a) optimize and cross-validate predictive models of self-reported negative schizotypy using a large acoustic feature set, (b) evaluate model performance as a function of sex and speaking task, (c) understand potential mechanisms underlying negative schizotypal traits by evaluating the key acoustic features within these models, and (d) examine model performance in its convergence with clinical symptoms and cognitive functioning. Accuracy was good (> 80%) and was improved by considering speaking task and sex. However, the features identified as most predictive of negative schizotypal traits were generally not considered critical to their conceptual definitions. Implications for validating and implementing digital phenotyping to understand and quantify negative schizotypy are discussed.

Monitoring the effects of therapeutic interventions in depression through self-assessments

The treatment of major psychiatric disorders is an arduous and thorny path for the patients concerned, characterized by polypharmacy, massive adverse side effects, modest prospects of success, and constantly declining response rates. The more important is the early detection of psychiatric disorders prior to the development of clinically relevant symptoms, so that people can benefit from early interventions. A well-proven approach to monitoring mental health relies on voice analysis. This method has been successfully used with psychiatric patients to 'objectively' document the progress of improvement or the onset of relapse. The studies with psychiatric patients over 2-4 weeks demonstrated that daily voice assessments have a notable therapeutic effect in themselves. Therefore, daily voice assessments appear to be a lowthreshold form of therapeutic means that may be realized through self-assessments. To evaluate performance and reliability of this approach, we have carried out a longitudinal study on 82 university students in 3 different countries with daily assessments over 2 weeks. The sample included 41 males (mean age 24.2±3.83 years) and 41 females (mean age 21.6±2.05 years). Unlike other research in the field, this study was not concerned with the classification of individuals in terms of diagnostic categories. The focus lay on the monitoring aspect and the extent to which the effects of therapeutic interventions or of behavioural changes are visible in the results of self-assessment voice analyses. The test persons showed an over-proportionally good adherence to the daily voice analysis scheme. The accumulated data were of generally high quality: sufficiently high signal levels, a very limited number of movement artifacts, and little to no interfering background noise. The method was sufficiently sensitive to detect: i) habituation effects when test persons became used to the daily procedure; and ii) short-term fluctuations that exceeded prespecified thresholds and reached significance. Results are directly interpretable and provide information about what is going well, what is going less well, and where there is a need for action. The proposed self-assessment approach was found to be well-suited to serve as a health-monitoring tool for subjects with an elevated vulnerability to psychiatric disorders or to stress-induced mental health problems. Daily voice assessments are in fact a low-threshold form of therapeutic means that can be realized through selfassessments, that requires only little effort, can be carried out in the test person's own home, and has the potential to strengthen resilience and to induce positive behavioural changes.

Validating Biobehavioral Technologies for Use in Clinical Psychiatry

The last decade has witnessed the development of sophisticated biobehavioral and genetic, ambulatory, and other measures that promise unprecedented insight into psychiatric disorders. As yet, clinical sciences have struggled with implementing these objective measures and they have yet to move beyond "proof of concept." In part, this struggle reflects a traditional, and conceptually flawed, application of traditional psychometrics (i.e., reliability and validity) for evaluating them. This paper focuses on "resolution," concerning the degree to which changes in a signal can be detected and quantified, which is central to measurement evaluation in informatics, engineering, computational and biomedical sciences. We define and discuss resolution in terms of traditional reliability and validity evaluation for psychiatric measures, then highlight its importance in a study using acoustic features to predict self-injurious thoughts/behaviors (SITB). This study involved tracking natural language and self-reported symptoms in 124 psychiatric patients: (a) over 5-14 recording sessions, collected using a smart phone application, and (b) during a clinical interview. Importantly, the scope of these measures varied as a function of time (minutes, weeks) and spatial setting (i.e., smart phone vs. interview). Regarding reliability, acoustic features were temporally unstable until we specified the level of temporal/spatial resolution. Regarding validity, accuracy based on machine learning of acoustic features predicting SITB varied as a function of resolution. High accuracy was achieved (i.e., ~87%), but only when the acoustic and SITB measures were "temporally-matched" in resolution was the model generalizable to new data. Unlocking the potential of biobehavioral technologies for clinical psychiatry will require careful consideration of resolution.

Using machine learning of computerized vocal expression to measure blunted vocal affect and alogia

Negative symptoms are a transdiagnostic feature of serious mental illness (SMI) that can be potentially “digitally phenotyped” using objective vocal analysis. In prior studies, vocal measures show low convergence with clinical ratings, potentially because analysis has used small, constrained acoustic feature sets. We sought to evaluate (1) whether clinically rated blunted vocal affect (BvA)/alogia could be accurately modelled using machine learning (ML) with a large feature set from two separate tasks (i.e., a 20-s “picture” and a 60-s “free-recall” task), (2) whether “Predicted” BvA/alogia (computed from the ML model) are associated with demographics, diagnosis, psychiatric symptoms, and cognitive/social functioning, and (3) which key vocal features are central to BvA/Alogia ratings. Accuracy was high (>90%) and was improved when computed separately by speaking task. ML scores were associated with poor cognitive performance and social functioning and were higher in patients with schizophrenia versus depression or mania diagnoses. However, the features identified as most predictive of BvA/Alogia were generally not considered critical to their operational definitions. Implications for validating and implementing digital phenotyping to reduce SMI burden are discussed.

Analyzing acoustic and prosodic fluctuations in free speech to predict psychosis onset in high-risk youths

The diagnosis and treatment of psychiatric disorders depends on the analysis of behavior through language by a clinical specialist. This analysis is subjective in nature and could benefit from automated, objective acoustic and linguistic processing methods. This integrated approach would convey a richer representation of patient speech, particularly for expression of emotion. In this work, we explore the potential of acoustic and prosodic metrics to infer clinical variables and predict psychosis, a condition which produces measurable derailment and tangentiality in patient language. To that purpose, we analyzed the recordings of 32 young patients at high risk of developing clinical psychosis. The subjects were evaluated using the Structured Interview for Prodromal Syndromes/Scale of Prodromal Symptoms (SIPS/SOPS) criteria. To analyze the recordings, we examined the variation of different acoustic and prosodic metrics across time. This preliminary analysis shows that these features can infer negative symptom severity ratings (i.e., SIPS-Btotal), obtaining a Pearson correlation of 0.77 for all the subjects after cross-validated evaluation. In addition, these features can predict development of psychosis with high accuracy above 90%, outperforming classification using clinical variables only. This improved predictive power ultimately can help provide early treatment and improve quality of life for those at risk for developing psychosis.

Ambulatory digital phenotyping of blunted affect and alogia using objective facial and vocal analysis: Proof of concept

Negative symptoms reflect one of the most debilitating aspects of one of the most debilitating diseases known to humankind. As yet, our treatments for negative symptoms are palliative at best and our understanding of their causes is relatively superficial. To address this, we are developing objective ambulatory tools for digitally phenotyping their severity which can be used outside the confines of the traditional clinical and research settings. The present study evaluated the feasibility, reliability and validity of ambulatory vocal acoustic and facial emotion expression analysis. Videos were provided by 25 patients with schizophrenia or schizoaffective disorder and 27 nonpsychiatric controls using inexpensive, non-invasive ambulatory recording methods. Controls provided 411 video recordings, and patients provided 377 video recordings; an average of 15.22 and 14.50 per participant per group respectively. The vast majority (over 80%) of these videos were usable for analysis. An empirically-supported, limited-feature vocal (7 features) and facial (3 features) set was examined. Within participants, these features varied considerably over time, but showed moderate to good test-retest reliability in many cases once contextual factors (e.g., activity involved in at the time of testing) were accounted for. Vocal and facial features showed statistically significant convergence with a "gold standard" negative symptom measure. Ambulatory vocal/facial features were more strongly associated with engagement in social or work activities in patients than negative symptom ratings. These data support the use of ambulatory vocal/facial analytic technologies for digital phenotyping of these negative symptoms.

Digital phenotyping of negative symptoms: the relationship to clinician ratings

Negative symptoms are a critical, but poorly understood, aspect of schizophrenia. Measurement of negative symptoms primarily relies on clinician ratings, an endeavor with established reliability and validity. There have been increasing attempts to digitally phenotype negative symptoms using objective biobehavioral technologies, eg, using computerized analysis of vocal, speech, facial, hand and other behaviors. Surprisingly, biobehavioral technologies and clinician ratings are only modestly inter-related, and findings from individual studies often do not replicate or are counterintuitive. In this article, we document and evaluate this lack of convergence in 4 case studies, in an archival dataset of 877 audio/video samples, and in the extant literature. We then explain this divergence in terms of "resolution"-a critical psychometric property in biomedical, engineering, and computational sciences defined as precision in distinguishing various aspects of a signal. We demonstrate how convergence between clinical ratings and biobehavioral data can be achieved by scaling data across various resolutions. Clinical ratings reflect an indispensable tool that integrates considerable information into actionable, yet "low resolution" ordinal ratings. This allows viewing of the "forest" of negative symptoms. Unfortunately, their resolution cannot be scaled or decomposed with sufficient precision to isolate the time, setting, and nature of negative symptoms for many purposes (ie, to see the "trees"). Biobehavioral measures afford precision for understanding when, where, and why negative symptoms emerge, though much work is needed to validate them. Digital phenotyping of negative symptoms can provide unprecedented opportunities for tracking, understanding, and treating them, but requires consideration of resolution.

Validity and features of spontaneous speech in acute aphasia as evaluated with the Brief Aphasia Evaluation: is fluent aphasia more severe than nonfluent aphasia?

PURPOSE

To explore the relationship between the two components of spontaneous speech in the Brief Aphasia Evaluation (BAE) and the rest of the scale represented by its three main factors: The Expression, Comprehension, and Complementary factors.

METHODS

BAE has proven validity and reliability. The evaluation of spontaneous speech in this scale comprises two components: Performance Rank (score: 0-3) and Type of Disorder (Fluency [F], Content [C], or Mixed [FC]) when rank < 3. Sixty-seven patients with left brain damage and 30 demographically matched healthy participants (HP) were studied. It was analyzed the correlation between Performance Rank and the three BAE factors and, recoding 3 as 0 and < 3 as 1, the sensitivity/specificity of this component for each factor. The effect of Type of Disorder on the three factors was analyzed.

RESULTS

1) Performance Rank: Correlations of 0.84 (Expression), 0.81 (Comprehension), and 0.76 (Complementary) were observed, with a sensitivity and specificity ≥ 78% for any factor; 2) Type of Disorder: The performance significantly decreased from FC to C and from C to F in Expression (FC < C < F), from FC to C and from FC to F also in Comprehension and Complementary, from patients with any type of disorder to HP.

CONCLUSION

Performance Rank was a relevant indicator of aphasia by its consistency with valid and comprehensive dimensions of acute language impairments. A degree difference between F and C was observed, being F a milder disorder; i.e., fluency problems were less severe than retrieval or anomia ones.

Comparing static and dynamic predictors of risk for hostility in serious mental illness: Preliminary findings

This study compared static predictors of hostility (e.g. demographics, clinician ratings) to subjective (i.e., self-reported affect on slider scales in response to written questions) and objective (i.e., vocal indicators of arousal from speech samples in a story-retelling task) dynamic predictors using ambulatory assessment over five days in a sample of 25 stable outpatients with diagnoses of a serious mental illness. Multilevel modeling showed that both subjective and objective dynamic predictors were significant, but none of the static predictors were. These results suggest that, in predicting hostility, it is more important to account for state variation than static traits.

Blunted vocal affect and expression is not associated with schizophrenia: A computerized acoustic analysis of speech under ambiguous conditions

INTRODUCTION

Patients with schizophrenia are consistently rated by clinicians as having high levels of blunted vocal affect and alogia. However, objective technologies have often failed to substantiate these abnormalities. It could be the case that negative symptoms are context-dependent.

OBJECTIVES

The present study examined speech elicited under conditions demonstrated to exacerbate thought disorder.

METHODS

The Rorschach Test was administered to 36 outpatients with schizophrenia and 25 nonpatient controls. Replies to separate "perceptual" and "memory" phases were analyzed using validated acoustic analytic methods.

RESULTS

Compared to nonpatient controls, schizophrenia patients did not display abnormal speech expression on objective measure of blunted vocal affect or alogia. Moreover, clinical ratings of negative symptoms were not significantly correlated with objective measures.

CONCLUSIONS

These findings suggest that in patients with schizophrenia, vocal affect/alogia is generally unremarkable under ambiguous conditions. Clarifying the nature of blunted vocal affect and alogia, and how objective measures correspond to what clinicians attend to when making clinical ratings are important directions for future research.

An open resource for transdiagnostic research in pediatric mental health and learning disorders

Technological and methodological innovations are equipping researchers with unprecedented capabilities for detecting and characterizing pathologic processes in the developing human brain. As a result, ambitions to achieve clinically useful tools to assist in the diagnosis and management of mental health and learning disorders are gaining momentum. To this end, it is critical to accrue large-scale multimodal datasets that capture a broad range of commonly encountered clinical psychopathology. The Child Mind Institute has launched the Healthy Brain Network (HBN), an ongoing initiative focused on creating and sharing a biobank of data from 10,000 New York area participants (ages 5–21). The HBN Biobank houses data about psychiatric, behavioral, cognitive, and lifestyle phenotypes, as well as multimodal brain imaging (resting and naturalistic viewing fMRI, diffusion MRI, morphometric MRI), electroencephalography, eye-tracking, voice and video recordings, genetics and actigraphy. Here, we present the rationale, design and implementation of HBN protocols. We describe the first data release (n=664) and the potential of the biobank to advance related areas (e.g., biophysical modeling, voice analysis).

A Diagnostic Marker to Discriminate Childhood Apraxia of Speech From Speech Delay: IV. The Pause Marker Index

Purpose

Three previous articles provided rationale, methods, and several forms of validity support for a diagnostic marker of childhood apraxia of speech (CAS), termed the pause marker (PM). Goals of the present article were to assess the validity and stability of the PM Index (PMI) to scale CAS severity.

Method

PM scores and speech, prosody, and voice precision-stability data were obtained for participants with CAS in idiopathic, neurogenetic, and complex neurodevelopmental disorders; adult-onset apraxia of speech consequent to stroke and primary progressive apraxia; and idiopathic speech delay. Three studies were completed including criterion and concurrent validity studies of the PMI and a temporal stability study of the PMI using retrospective case studies.

Results

PM scores were significantly correlated with other signs of CAS precision and stability. The best fit of the distribution of PM scores to index CAS severity was obtained by dividing scores into 4 ordinal severity classifications: mild, mild-moderate, moderate-severe, and severe. Severity findings for the 4 classifications and retrospective longitudinal findings from 8 participants with CAS supported the validity and stability of the PMI.

Conclusion

Findings support research and clinical use of the PMI to scale the severity of CAS.

Vocal expression in schizophrenia: Less than meets the ear

Abnormalities in nonverbal communication are a hallmark of schizophrenia. Results from studies using symptom rating scales suggest that these abnormalities are profound (i.e., 3-5 SDs) and occur across virtually every channel of vocal expression. Computerized acoustic analytic technologies, used to overcome practical and psychometric limitations with symptom rating scales, have found much more benign and isolated abnormalities. To better understand vocal deficits in schizophrenia and to advance acoustic analytic technologies for clinical and research applications, we examined archived speech samples from 5 separate studies, each using different speaking tasks (patient N = 309; control N = 117). We sought to: (a) use Principal Component Analysis (PCA) to identify independent vocal expression measures from a large set of variables, (b) quantify how patients with schizophrenia are abnormal with respect to these variables, (c) evaluate the impact of demographic and contextual factors (e.g., study site, speaking task), and (d) examine the relationship between clinically-rated psychiatric symptoms and vocal variables. PCA identified 7 independent markers of vocal expression. Most of these vocal variables varied considerably as a function of context and many were associated with demographic factors. After controlling for context and demographics, there were no meaningful differences in vocal expression between patients and controls. Within patients, vocal variables were associated with a range of psychiatric symptoms-though only pause length was significantly associated with clinically rated negative symptoms. The discussion centers on explaining the apparent discordance between clinical and computerized speech measures.