Assessing Bipolar Episodes Using Speech Cues Derived from Phone Calls

Acoustic features from speech as markers of depressive and manic symptoms in bipolar disorder: A prospective study

INTRODUCTION

Voice features could be a sensitive marker of affective state in bipolar disorder (BD). Smartphone apps offer an excellent opportunity to collect voice data in the natural setting and become a useful tool in phase prediction in BD.

AIMS OF THE STUDY

We investigate the relations between the symptoms of BD, evaluated by psychiatrists, and patients' voice characteristics. A smartphone app extracted acoustic parameters from the daily phone calls of n = 51 patients. We show how the prosodic, spectral, and voice quality features correlate with clinically assessed affective states and explore their usefulness in predicting the BD phase.

METHODS

A smartphone app (BDmon) was developed to collect the voice signal and extract its physical features. BD patients used the application on average for 208 days. Psychiatrists assessed the severity of BD symptoms using the Hamilton depression rating scale -17 and the Young Mania rating scale. We analyze the relations between acoustic features of speech and patients' mental states using linear generalized mixed-effect models.

RESULTS

The prosodic, spectral, and voice quality parameters, are valid markers in assessing the severity of manic and depressive symptoms. The accuracy of the predictive generalized mixed-effect model is 70.9%-71.4%. Significant differences in the effect sizes and directions are observed between female and male subgroups. The greater the severity of mania in males, the louder (β = 1.6) and higher the tone of voice (β = 0.71), more clearly (β = 1.35), and more sharply they speak (β = 0.95), and their conversations are longer (β = 1.64). For females, the observations are either exactly the opposite-the greater the severity of mania, the quieter (β = -0.27) and lower the tone of voice (β = -0.21) and less clearly (β = -0.25) they speak - or no correlations are found (length of speech). On the other hand, the greater the severity of bipolar depression in males, the quieter (β = -1.07) and less clearly they speak (β = -1.00). In females, no distinct correlations between the severity of depressive symptoms and the change in voice parameters are found.

CONCLUSIONS

Speech analysis provides physiological markers of affective symptoms in BD and acoustic features extracted from speech are effective in predicting BD phases. This could personalize monitoring and care for BD patients, helping to decide whether a specialist should be consulted.

The Role of Selected Speech Signal Characteristics in Discriminating Unipolar and Bipolar Disorders

OBJECTIVE

The objective of this study is to explore and enhance the diagnostic process of unipolar and bipolar disorders. The primary focus is on leveraging automated processes to improve the accuracy and accessibility of diagnosis. The study aims to introduce an audio corpus collected from patients diagnosed with these disorders, annotated using the Clinical Global Impressions Scale (CGI) by psychiatrists.

METHODS AND PROCEDURES

Traditional diagnostic methods rely on the clinician's expertise and consideration of co-existing mental disorders. However, this study proposes the implementation of automated processes in the diagnosis, providing quantitative measures and enabling prolonged observation of patients. The paper introduces a speech signal pipeline for CGI state classification, with a specific focus on selecting the most discriminative features. Acoustic features such as prosodies, MFCC, and LPC coefficients are examined in the study. The classification process utilizes common machine learning methods.

RESULTS

The results of the study indicate promising outcomes for the automated diagnosis of bipolar and unipolar disorders using the proposed speech signal pipeline. The audio corpus annotated with CGI by psychiatrists achieved a classification accuracy of 95% for the two-class classification. For the four- and seven-class classifications, the results were 77.3% and 73%, respectively, demonstrating the potential of the developed method in distinguishing different states of the disorders.

Evaluating the Feasibility and Acceptability of an Artificial-Intelligence-Enabled and Speech-Based Distress Screening Mobile App for Adolescents and Young Adults Diagnosed with Cancer: A Study Protocol

Simple Summary

Adolescent and young adult (AYA) patients diagnosed with cancer are at a higher risk of psychological distress, which requires regular monitoring throughout their cancer journeys. Paper-and-pencil or digital surveys for psychological stress are often cumbersome to complete during a patient’s visit, and many patients find completing the same survey multiple times repetitive and boring. Recent advances in mobile technology and speech science have enabled flexible and engaging ways of monitoring psychological distress. This paper describes the scientific process we will use to evaluate an artificial intelligence (AI)-enabled mobile app to monitor depression and anxiety among AYAs diagnosed with cancer.

Abstract

Adolescents and young adults (AYAs) diagnosed with cancer are an age-defined population, with studies reporting up to 45% of the population experiencing psychological distress. Although it is essential to screen and monitor for psychological distress throughout AYAs’ cancer journeys, many cancer centers fail to effectively implement distress screening protocols largely due to busy clinical workflow and survey fatigue. Recent advances in mobile technology and speech science have enabled flexible and engaging methods to monitor psychological distress. However, patient-centered research focusing on these methods’ feasibility and acceptability remains lacking. Therefore, in this project, we aim to evaluate the feasibility and acceptability of an artificial intelligence (AI)-enabled and speech-based mobile application to monitor psychological distress among AYAs diagnosed with cancer. We use a single-arm prospective cohort design with a stratified sampling strategy. We aim to recruit 60 AYAs diagnosed with cancer and to monitor their psychological distress using an AI-enabled speech-based distress monitoring tool over a 6 month period. The primary feasibility endpoint of this study is defined by the number of participants completing four out of six monthly distress assessments, and the acceptability endpoint is defined both quantitatively using the acceptability of intervention measure and qualitatively using semi-structured interviews.

Behavioural and Self-Reported Data Collected from Smartphones in the Assessment of Depressive and Manic Symptoms for Bipolar Disorder Patients: Prospective Observational Study

BACKGROUND

Smartphones allow for real-time monitoring of patients' behavioural activities in a naturalistic setting. These data are suggested as markers of mental state in bipolar disorder (BD).

OBJECTIVE

We assess the relations between data collected from smartphones and the clinically rated depressive and manic symptoms together with the corresponding affective states in BD.

METHODS

BDmon - a dedicated mobile app was developed and installed on the patients' smartphones to automatically collect statistics about phone calls and text messages, as well as self-assessment of sleep and patient's mood. The final sample for the numerical analyses consisted of 51 eligible patients who participated in at least two psychiatric assessments and used the BDmon app (mean participation time: 208 days ± SD of 132 days). In total, 196 psychiatric assessments were performed using the Hamilton Depression Rating Scale (HDRS) and Young Mania Rating Scale (YMRS). Generalized linear mixed-effects models were applied to quantify the strength of the relation between the daily statistics about behavioural data collected automatically from smartphones and the affective symptoms and mood states in BD.

RESULTS

Objective behavioural data collected from smartphones and their relation to BD states were as follows: (1) depressed patients tended to make phone calls less frequently than in euthymia (β=-0.064, P=.01); (2) the number of incoming answered calls was lower in depression as compared to euthymia (β=-0.15, P=.01) and, at the same time, missed incoming calls were more frequent and increased as depressive symptoms intensified (β=4.431, P<.001; β=4.861, P<.001, respectively); (3) the fraction of outgoing calls was higher in manic states (β=2.73, P=.03); (4) the fraction of missed calls was higher in manic/mixed states as compared to euthymia (β=3.53, P=.01) and positively correlated to the severity of symptoms (β=2.991, P=.02); (5) variability of duration of outgoing calls was higher in manic/mixed states (β=1.22·10-3, P=.045) and positively correlated to the severity of symptoms (β=1.72·10-3, P=.02); (6) the number and length of sent text messages was higher in manic/mixed states as compared to euthymia (β=0.031, P=.01; β=0.015, P=.01, respectively) and positively correlated to the severity of manic symptoms (β=0.116, P<.001; β=0.022, P<.001). We also observed that self-assessment of mood was lower in depressive (β=-1.452, P<.001). and higher in manic states (β=0.509, P<.001).

CONCLUSIONS

Smartphone-based behavioural parameters are valid markers in assessing the severity of affective symptoms and discriminating between mood states. This opens a way toward early detection of worsening of the mental state and thereby increases the patient's chance of improving the course of the illness.

Internet of Things for Mental Health: Open Issues in Data Acquisition, Self-Organization, Service Level Agreement, and Identity Management

The increase of mental illness cases around the world can be described as an urgent and serious global health threat. Around 500 million people suffer from mental disorders, among which depression, schizophrenia, and dementia are the most prevalent. Revolutionary technological paradigms such as the Internet of Things (IoT) provide us with new capabilities to detect, assess, and care for patients early. This paper comprehensively survey works done at the intersection between IoT and mental health disorders. We evaluate multiple computational platforms, methods and devices, as well as study results and potential open issues for the effective use of IoT systems in mental health. We particularly elaborate on relevant open challenges in the use of existing IoT solutions for mental health care, which can be relevant given the potential impairments in some mental health patients such as data acquisition issues, lack of self-organization of devices and service level agreement, and security, privacy and consent issues, among others. We aim at opening the conversation for future research in this rather emerging area by outlining possible new paths based on the results and conclusions of this work.

Passive Sensing of Prediction of Moment-To-Moment Depressed Mood among Undergraduates with Clinical Levels of Depression Sample Using Smartphones

Prior research has recently shown that passively collected sensor data collected within the contexts of persons daily lives via smartphones and wearable sensors can distinguish those with major depressive disorder (MDD) from controls, predict MDD severity, and predict changes in MDD severity across days and weeks. Nevertheless, very little research has examined predicting depressed mood within a day, which is essential given the large amount of variation occurring within days. The current study utilized passively collected sensor data collected from a smartphone application to future depressed mood from hour-to-hour in an ecological momentary assessment study in a sample reporting clinical levels of depression (N = 31). Using a combination of nomothetic and idiographically-weighted machine learning models, the results suggest that depressed mood can be accurately predicted from hour to hour with an average correlation between out of sample predicted depressed mood levels and observed depressed mood of 0.587, CI [0.552, 0.621]. This suggests that passively collected smartphone data can accurately predict future depressed mood among a sample reporting clinical levels of depression. If replicated in other samples, this modeling framework may allow just-in-time adaptive interventions to treat depression as it changes in the context of daily life.

Clinical state tracking in serious mental illness through computational analysis of speech

Individuals with serious mental illness experience changes in their clinical states over time that are difficult to assess and that result in increased disease burden and care utilization. It is not known if features derived from speech can serve as a transdiagnostic marker of these clinical states. This study evaluates the feasibility of collecting speech samples from people with serious mental illness and explores the potential utility for tracking changes in clinical state over time. Patients (n = 47) were recruited from a community-based mental health clinic with diagnoses of bipolar disorder, major depressive disorder, schizophrenia or schizoaffective disorder. Patients used an interactive voice response system for at least 4 months to provide speech samples. Clinic providers (n = 13) reviewed responses and provided global assessment ratings. We computed features of speech and used machine learning to create models of outcome measures trained using either population data or an individual's own data over time. The system was feasible to use, recording 1101 phone calls and 117 hours of speech. Most (92%) of the patients agreed that it was easy to use. The individually-trained models demonstrated the highest correlation with provider ratings (rho = 0.78, p<0.001). Population-level models demonstrated statistically significant correlations with provider global assessment ratings (rho = 0.44, p<0.001), future provider ratings (rho = 0.33, p<0.05), BASIS-24 summary score, depression sub score, and self-harm sub score (rho = 0.25,0.25, and 0.28 respectively; p<0.05), and the SF-12 mental health sub score (rho = 0.25, p<0.05), but not with other BASIS-24 or SF-12 sub scores. This study brings together longitudinal collection of objective behavioral markers along with a transdiagnostic, personalized approach for tracking of mental health clinical state in a community-based clinical setting.

Speech-based markers for posttraumatic stress disorder in US veterans

BACKGROUND

The diagnosis of posttraumatic stress disorder (PTSD) is usually based on clinical interviews or self-report measures. Both approaches are subject to under- and over-reporting of symptoms. An objective test is lacking. We have developed a classifier of PTSD based on objective speech-marker features that discriminate PTSD cases from controls.

METHODS

Speech samples were obtained from warzone-exposed veterans, 52 cases with PTSD and 77 controls, assessed with the Clinician-Administered PTSD Scale. Individuals with major depressive disorder (MDD) were excluded. Audio recordings of clinical interviews were used to obtain 40,526 speech features which were input to a random forest (RF) algorithm.

RESULTS

The selected RF used 18 speech features and the receiver operating characteristic curve had an area under the curve (AUC) of 0.954. At a probability of PTSD cut point of 0.423, Youden's index was 0.787, and overall correct classification rate was 89.1%. The probability of PTSD was higher for markers that indicated slower, more monotonous speech, less change in tonality, and less activation. Depression symptoms, alcohol use disorder, and TBI did not meet statistical tests to be considered confounders.

CONCLUSIONS

This study demonstrates that a speech-based algorithm can objectively differentiate PTSD cases from controls. The RF classifier had a high AUC. Further validation in an independent sample and appraisal of the classifier to identify those with MDD only compared with those with PTSD comorbid with MDD is required.

Objective smartphone data as a potential diagnostic marker of bipolar disorder

OBJECTIVE

Currently, the diagnosis in bipolar disorder relies on patient information and careful clinical evaluations and judgements with a lack of objective tests. Core clinical features of bipolar disorder include changes in behaviour. We aimed to investigate objective smartphone data reflecting behavioural activities to classify patients with bipolar disorder compared with healthy individuals.

METHODS

Objective smartphone data were automatically collected from 29 patients with bipolar disorder and 37 healthy individuals. Repeated measurements of objective smartphone data were performed during different affective states in patients with bipolar disorder over 12 weeks and compared with healthy individuals.

RESULTS

Overall, the sensitivity of objective smartphone data in patients with bipolar disorder versus healthy individuals was 0.92, specificity 0.39, positive predictive value 0.88 and negative predictive value 0.52. In euthymic patients versus healthy individuals, the sensitivity was 0.90, specificity 0.56, positive predictive value 0.85 and negative predictive value 0.67. In mixed models, automatically generated objective smartphone data (the number of text messages/day, the duration of phone calls/day) were increased in patients with bipolar disorder (during euthymia, depressive and manic or mixed states, and overall) compared with healthy individuals. The amount of time the smartphone screen was 'on' per day was decreased in patients with bipolar disorder (during euthymia, depressive state and overall) compared with healthy individuals.

CONCLUSION

Objective smartphone data may represent a potential diagnostic behavioural marker in bipolar disorder and may be a candidate supplementary method to the diagnostic process in the future. Further studies including larger samples, first-degree relatives and patients with other psychiatric disorders are needed.

Correlations Between Objective Behavioral Features Collected From Mobile and Wearable Devices and Depressive Mood Symptoms in Patients With Affective Disorders: Systematic Review

Background

Several studies have recently reported on the correlation between objective behavioral features collected via mobile and wearable devices and depressive mood symptoms in patients with affective disorders (unipolar and bipolar disorders). However, individual studies have reported on different and sometimes contradicting results, and no quantitative systematic review of the correlation between objective behavioral features and depressive mood symptoms has been published.

Objective

The objectives of this systematic review were to (1) provide an overview of the correlations between objective behavioral features and depressive mood symptoms reported in the literature and (2) investigate the strength and statistical significance of these correlations across studies. The answers to these questions could potentially help identify which objective features have shown most promising results across studies.

Methods

We conducted a systematic review of the scientific literature, reported according to the preferred reporting items for systematic reviews and meta-analyses guidelines. IEEE Xplore, ACM Digital Library, Web of Sciences, PsychINFO, PubMed, DBLP computer science bibliography, HTA, DARE, Scopus, and Science Direct were searched and supplemented by hand examination of reference lists. The search ended on April 27, 2017, and was limited to studies published between 2007 and 2017.

Results

A total of 46 studies were eligible for the review. These studies identified and investigated 85 unique objective behavioral features, covering 17 various sensor data inputs. These features were divided into 7 categories. Several features were found to have statistically significant and consistent correlation directionality with mood assessment (eg, the amount of home stay, sleep duration, and vigorous activity), while others showed directionality discrepancies across the studies (eg, amount of text messages [short message service] sent, time spent between locations, and frequency of mobile phone screen activity).

Conclusions

Several studies showed consistent and statistically significant correlations between objective behavioral features collected via mobile and wearable devices and depressive mood symptoms. Hence, continuous and everyday monitoring of behavioral aspects in affective disorders could be a promising supplementary objective measure for estimating depressive mood symptoms. However, the evidence is limited by methodological issues in individual studies and by a lack of standardization of (1) the collected objective features, (2) the mood assessment methodology, and (3) the statistical methods applied. Therefore, consistency in data collection and analysis in future studies is needed, making replication studies as well as meta-analyses possible.

The impact and measurement of social dysfunction in late-life depression: an evaluation of current methods with a focus on wearable technology

OBJECTIVE

Depression is known to negatively impact social functioning, with patients commonly reporting difficulties maintaining social relationships. Moreover, a large body of evidence suggests poor social functioning is not only present in depression but that social functioning is an important factor in illness course and outcome. In addition, good social relationships can play a protective role against the onset of depressive symptoms, particularly in late-life depression. However, the majority of research in this area has employed self-report measures of social function. This approach is problematic, as due to their reliance on memory, such measures are prone to error from the neurocognitive impairments of depression, as well as mood-congruent biases.

METHOD

Narrative review based on searches of the Web of Science and PubMed database(s) from the start of the databases, until the end of 2015.

RESULTS

The present review provides an overview of the literature on social functioning in (late-life) depression and discusses the potential for new technologies to improve the measurement of social function in depressed older adults. In particular, the use of wearable technology to collect direct, objective measures of social activity, such as physical activity and speech, is considered.

CONCLUSION

In order to develop a greater understanding of social functioning in late-life depression, future research should include the development and validation of more direct, objective measures in conjunction with subjective self-report measures. Copyright © 2016 John Wiley & Sons, Ltd.

Big data for bipolar disorder

The delivery of psychiatric care is changing with a new emphasis on integrated care, preventative measures, population health, and the biological basis of disease. Fundamental to this transformation are big data and advances in the ability to analyze these data. The impact of big data on the routine treatment of bipolar disorder today and in the near future is discussed, with examples that relate to health policy, the discovery of new associations, and the study of rare events. The primary sources of big data today are electronic medical records (EMR), claims, and registry data from providers and payers. In the near future, data created by patients from active monitoring, passive monitoring of Internet and smartphone activities, and from sensors may be integrated with the EMR. Diverse data sources from outside of medicine, such as government financial data, will be linked for research. Over the long term, genetic and imaging data will be integrated with the EMR, and there will be more emphasis on predictive models. Many technical challenges remain when analyzing big data that relates to size, heterogeneity, complexity, and unstructured text data in the EMR. Human judgement and subject matter expertise are critical parts of big data analysis, and the active participation of psychiatrists is needed throughout the analytical process.

Voice quality in patients suffering from bipolar disease

People suffering from bipolar disease are more and more common. Such pathology can severely affect patients' lifestyle by wide, and sometimes extreme, mood swings. Biosignals can be very useful to understand this disease. Specifically, speech-related features have been seen to vary in depressed people with respect to healthy subjects. Usually prosodic, spectral and energy-related features are studied. Some further information, instead, can be provided studying voice quality. According to Laver's model, voice quality is sensitive and depends on both anatomic/physiologic issues and long-term muscular adjustments of the larynx or the supraglottal vocal tract. A pilot study on both bipolar patients and healthy control subjects, performed by means of the Long-Term Average Spectrum (LTAS) is presented. The effects on LTAS estimation of a F0-correction procedure are discussed. Pairwise statistical comparisons between subjects in euthymic and depressed states and euthymic and hypomanic states were performed. Significant differences were found in some frequency intervals in both cases. The F0-correction procedure modified the values of the significant frequency intervals in the euthymic/depressed comparison, that also was characterized by a change of F0. Noticeably, no statistically significant differences were found in control subjects acquired in the same mood state. Though the number of subjects is small, the results are encouraging given their coherence across patients and the lack of differences in the control group. Finally, this work suggests that particular vocal settings might be involved in different mood states.

Voice analysis as an objective state marker in bipolar disorder

Changes in speech have been suggested as sensitive and valid measures of depression and mania in bipolar disorder. The present study aimed at investigating (1) voice features collected during phone calls as objective markers of affective states in bipolar disorder and (2) if combining voice features with automatically generated objective smartphone data on behavioral activities (for example, number of text messages and phone calls per day) and electronic self-monitored data (mood) on illness activity would increase the accuracy as a marker of affective states. Using smartphones, voice features, automatically generated objective smartphone data on behavioral activities and electronic self-monitored data were collected from 28 outpatients with bipolar disorder in naturalistic settings on a daily basis during a period of 12 weeks. Depressive and manic symptoms were assessed using the Hamilton Depression Rating Scale 17-item and the Young Mania Rating Scale, respectively, by a researcher blinded to smartphone data. Data were analyzed using random forest algorithms. Affective states were classified using voice features extracted during everyday life phone calls. Voice features were found to be more accurate, sensitive and specific in the classification of manic or mixed states with an area under the curve (AUC)=0.89 compared with an AUC=0.78 for the classification of depressive states. Combining voice features with automatically generated objective smartphone data on behavioral activities and electronic self-monitored data increased the accuracy, sensitivity and specificity of classification of affective states slightly. Voice features collected in naturalistic settings using smartphones may be used as objective state markers in patients with bipolar disorder.

Smartphone-based recognition of states and state changes in bipolar disorder patients

Today's health care is difficult to imagine without the possibility to objectively measure various physiological parameters related to patients' symptoms (from temperature through blood pressure to complex tomographic procedures). Psychiatric care remains a notable exception that heavily relies on patient interviews and self-assessment. This is due to the fact that mental illnesses manifest themselves mainly in the way patients behave throughout their daily life and, until recently there were no "behavior measurement devices." This is now changing with the progress in wearable activity recognition and sensor enabled smartphones. In this paper, we introduce a system, which, based on smartphone-sensing is able to recognize depressive and manic states and detect state changes of patients suffering from bipolar disorder. Drawing upon a real-life dataset of ten patients, recorded over a time period of 12 weeks (in total over 800 days of data tracing 17 state changes) by four different sensing modalities, we could extract features corresponding to all disease-relevant aspects in behavior. Using these features, we gain recognition accuracies of 76% by fusing all sensor modalities and state change detection precision and recall of over 97%. This paper furthermore outlines the applicability of this system in the physician-patient relations in order to facilitate the life and treatment of bipolar patients.