Burying Our Heads in the Sand: The Neglected Importance of Reporting Inter-Rater Reliability in Antipsychotic Medication Trials

Multimodal Mental Health Digital Biomarker Analysis From Remote Interviews Using Facial, Vocal, Linguistic, and Cardiovascular Patterns

OBJECTIVE

Psychiatric evaluation suffers from subjectivity and bias, and is hard to scale due to intensive professional training requirements. In this work, we investigated whether behavioral and physiological signals, extracted from tele-video interviews, differ in individuals with psychiatric disorders.

METHODS

Temporal variations in facial expression, vocal expression, linguistic expression, and cardiovascular modulation were extracted from simultaneously recorded audio and video of remote interviews. Averages, standard deviations, and Markovian process-derived statistics of these features were computed from 73 subjects. Four binary classification tasks were defined: detecting 1) any clinically-diagnosed psychiatric disorder, 2) major depressive disorder, 3) self-rated depression, and 4) self-rated anxiety. Each modality was evaluated individually and in combination.

RESULTS

Statistically significant feature differences were found between psychiatric and control subjects. Correlations were found between features and self-rated depression and anxiety scores. Heart rate dynamics provided the best unimodal performance with areas under the receiver-operator curve (AUROCs) of 0.68-0.75 (depending on the classification task). Combining multiple modalities provided AUROCs of 0.72-0.82.

CONCLUSION

Multimodal features extracted from remote interviews revealed informative characteristics of clinically diagnosed and self-rated mental health status.

SIGNIFICANCE

The proposed multimodal approach has the potential to facilitate scalable, remote, and low-cost assessment for low-burden automated mental health services.

Multimodal mental health assessment with remote interviews using facial, vocal, linguistic, and cardiovascular patterns

Objective

The current clinical practice of psychiatric evaluation suffers from subjectivity and bias, and requires highly skilled professionals that are often unavailable or unaffordable. Objective digital biomarkers have shown the potential to address these issues. In this work, we investigated whether behavioral and physiological signals, extracted from remote interviews, provided complimentary information for assessing psychiatric disorders.

Methods

Time series of multimodal features were derived from four conceptual modes: facial expression, vocal expression, linguistic expression, and cardiovascular modulation. The features were extracted from simultaneously recorded audio and video of remote interviews using task-specific and foundation models. Averages, standard deviations, and hidden Markov model-derived statistics of these features were computed from 73 subjects. Four binary classification tasks were defined: detecting 1) any clinically-diagnosed psychiatric disorder, 2) major depressive disorder, 3) self-rated depression, and 4) self-rated anxiety. Each modality was evaluated individually and in combination.

Results

Statistically significant feature differences were found between controls and subjects with mental health conditions. Correlations were found between features and self-rated depression and anxiety scores. Visual heart rate dynamics achieved the best unimodal performance with areas under the receiver-operator curve (AUROCs) of 0.68-0.75 (depending on the classification task). Combining multiple modalities achieved AUROCs of 0.72-0.82. Features from task-specific models outperformed features from foundation models.

Conclusion

Multimodal features extracted from remote interviews revealed informative characteristics of clinically diagnosed and self-rated mental health status.

Significance

The proposed multimodal approach has the potential to facilitate objective, remote, and low-cost assessment for low-burden automated mental health services.

Site-independent confirmation of primary site-based PANSS ratings in a schizophrenia trial

Blinded, site-independent (remote) ratings from audio-digital recordings of site-based Positive and Negative Syndrome Scale (PANSS) interviews were obtained in a 5-week, randomized, double-blinded study assessing the safety, tolerability, and efficacy of KarXT (a fixed combination of xanomeline and trospium chloride) in hospitalized adults with schizophrenia experiencing an acute exacerbation of psychosis (EMERGENT-1; ClinicalTrials.gov identifier: NCT3697252). The blinded site-independent raters had no knowledge of site location, study visit, drug vs. placebo assignment, or any treatment emergent adverse events (TEAEs). Concordance analyses of 561 paired site-based and site-independent PANSS ratings across all visits revealed a high correlation (ICC = 0.775). Paired scoring differences were positively correlated with the PANSS total score (Spearman's rho = 0.37, p < 0.0001). Paired PANSS scores were available from 148 subjects at both the baseline and end of study visits (KarXT = 72, Placebo = 76). Site-based PANSS total scores (primary aim) revealed a significantly greater improvement from baseline in the KarXT group compared to the placebo group (p < 0.0001). The blinded site-independent PANSS total scores derived from listening to and scoring the recorded site-based PANSS interviews replicated this finding (p < 0.001) and yielded an overall predictive value of 85.1% for matching the site-based response/non-response outcomes. TEAE's have the potential to "unblind" site-based ratings. In this study, the site-independent raters were blinded to TEAEs, affirmed the site-based PANSS ratings, and mitigated concerns about possible functional unblinding of site-based raters. This method of blinded assessment via audio-digital recordings may have utility for other studies concerned with ratings precision and/or functional unblinding.