Fusing Ambient and Mobile Sensor Features Into a Behaviorome for Predicting Clinical Health Scores

Multimodal Fusion of Smart Home and Text-based Behavior Markers for Clinical Assessment Prediction

New modes of technology are offering unprecedented opportunities to unobtrusively collect data about people's behavior. While there are many use cases for such information, we explore its utility for predicting multiple clinical assessment scores. Because clinical assessments are typically used as screening tools for impairment and disease, such as mild cognitive impairment (MCI), automatically mapping behavioral data to assessment scores can help detect changes in health and behavior across time. In this paper, we aim to extract behavior markers from two modalities, a smart home environment and a custom digital memory notebook app, for mapping to ten clinical assessments that are relevant for monitoring MCI onset and changes in cognitive health. Smart home-based behavior markers reflect hourly, daily, and weekly activity patterns, while app-based behavior markers reflect app usage and writing content/style derived from free-form journal entries. We describe machine learning techniques for fusing these multimodal behavior markers and utilizing joint prediction. We evaluate our approach using three regression algorithms and data from 14 participants with MCI living in a smart home environment. We observed moderate to large correlations between predicted and ground-truth assessment scores, ranging from r = 0.601 to r = 0.871 for each clinical assessment.

Automated Cognitive Health Assessment Using Partially Complete Time Series Sensor Data

BACKGROUND

Behavior and health are inextricably linked. As a result, continuous wearable sensor data offer the potential to predict clinical measures. However, interruptions in the data collection occur, which create a need for strategic data imputation.

OBJECTIVE

The objective of this work is to adapt a data generation algorithm to impute multivariate time series data. This will allow us to create digital behavior markers that can predict clinical health measures.

METHODS

We created a bidirectional time series generative adversarial network to impute missing sensor readings. Values are imputed based on relationships between multiple fields and multiple points in time, for single time points or larger time gaps. From the complete data, digital behavior markers are extracted and are mapped to predicted clinical measures.

RESULTS

We validate our approach using continuous smartwatch data for n = 14 participants. When reconstructing omitted data, we observe an average normalized mean absolute error of 0.0197. We then create machine learning models to predict clinical measures from the reconstructed, complete data with correlations ranging from r = 0.1230 to r = 0.7623. This work indicates that wearable sensor data collected in the wild can be used to offer insights on a person's health in natural settings.

A systems approach towards remote health-monitoring in older adults: Introducing a zero-interaction digital exhaust

Using connected sensing devices to remotely monitor health is a promising way to help transition healthcare from a rather reactive to a more precision medicine oriented proactive approach, which could be particularly relevant in the face of rapid population ageing and the challenges it poses to healthcare systems. Sensor derived digital measures of health, such as digital biomarkers or digital clinical outcome assessments, may be used to monitor health status or the risk of adverse events like falls. Current research around such digital measures has largely focused on exploring the use of few individual measures obtained through mobile devices. However, especially for long-term applications in older adults, this choice of technology may not be ideal and could further add to the digital divide. Moreover, large-scale systems biology approaches, like genomics, have already proven beneficial in precision medicine, making it plausible that the same could also hold for remote-health monitoring. In this context, we introduce and describe a zero-interaction digital exhaust: a set of 1268 digital measures that cover large parts of a person’s activity, behavior and physiology. Making this approach more inclusive of older adults, we base this set entirely on contactless, zero-interaction sensing technologies. Applying the resulting digital exhaust to real-world data, we then demonstrate the possibility to create multiple ageing relevant digital clinical outcome assessments. Paired with modern machine learning, we find these assessments to be surprisingly powerful and often on-par with mobile approaches. Lastly, we highlight the possibility to discover novel digital biomarkers based on this large-scale approach.