A Novel Chewing Detection System Based on PPG, Audio, and Accelerometry

Factors associated with eating rate: a systematic review and narrative synthesis informed by socio-ecological model

Accumulating evidence shows associations between rapid eating and overweight. Modifying eating rate might be a potential weight management strategy without imposing additional dietary restrictions. A comprehensive understanding of factors associated with eating speed will help with designing effective interventions. The aim of this review was to synthesise the current state of knowledge on the factors associated with eating rate. The socio-ecological model (SEM) was utilised to scaffold the identified factors. A comprehensive literature search of eleven databases was conducted to identify factors associated with eating rate. The 104 studies that met the inclusion criteria were heterogeneous in design and methods of eating rate measurement. We identified thirty-nine factors that were independently linked to eating speed and mapped them onto the individual, social and environmental levels of the SEM. The majority of the reported factors pertained to the individual characteristics (n = 20) including demographics, cognitive/psychological factors and habitual food oral processing behaviours. Social factors (n = 11) included eating companions, social and cultural norms, and family structure. Environmental factors (n = 8) included food texture and presentation, methods of consumption or background sounds. Measures of body weight, food form and characteristics, food oral processing behaviours and gender, age and ethnicity were the most researched and consistent factors associated with eating rate. A number of other novel and underresearched factors emerged, but these require replication and further research. We highlight directions for further research in this space and potential evidence-based candidates for interventions targeting eating rate.

Eating Speed Measurement Using Wrist-Worn IMU Sensors Towards Free-Living Environments

Eating speed is an important indicator that has been widely investigated in nutritional studies. The relationship between eating speed and several intake-related problems such as obesity, diabetes, and oral health has received increased attention from researchers. However, existing studies mainly use self-reported questionnaires to obtain participants' eating speed, where they choose options from slow, medium, and fast. Such a non-quantitative method is highly subjective and coarse at the individual level. This study integrates two classical tasks in automated food intake monitoring domain: bite detection and eating episode detection, to advance eating speed measurement in near-free-living environments automatically and objectively. Specifically, a temporal convolutional network combined with a multi-head attention module (TCN-MHA) is developed to detect bites (including eating and drinking gestures) from IMU data. The predicted bite sequences are then clustered into eating episodes. Eating speed is calculated by using the time taken to finish the eating episode to divide the number of bites. To validate the proposed approach on eating speed measurement, a 7-fold cross validation is applied to the self-collected fine-annotated full-day-I (FD-I) dataset, and a holdout experiment is conducted on the full-day-II (FD-II) dataset. The two datasets are collected from 61 participants with a total duration of 513 h, which are publicly available. Experimental results show that the proposed approach achieves a mean absolute percentage error (MAPE) of 0.110 and 0.146 in the FD-I and FD-II datasets, respectively, showcasing the feasibility of automated eating speed measurement in near-free-living environments.

Automated detection and recognition system for chewable food items using advanced deep learning models

Identifying and recognizing the food on the basis of its eating sounds is a challenging task, as it plays an important role in avoiding allergic foods, providing dietary preferences to people who are restricted to a particular diet, showcasing its cultural significance, etc. In this research paper, the aim is to design a novel methodology that helps to identify food items by analyzing their eating sounds using various deep learning models. To achieve this objective, a system has been proposed that extracts meaningful features from food-eating sounds with the help of signal processing techniques and deep learning models for classifying them into their respective food classes. Initially, 1200 audio files for 20 food items labeled have been collected and visualized to find relationships between the sound files of different food items. Later, to extract meaningful features, various techniques such as spectrograms, spectral rolloff, spectral bandwidth, and mel-frequency cepstral coefficients are used for the cleaning of audio files as well as to capture the unique characteristics of different food items. In the next phase, various deep learning models like GRU, LSTM, InceptionResNetV2, and the customized CNN model have been trained to learn spectral and temporal patterns in audio signals. Besides this, the models have also been hybridized i.e. Bidirectional LSTM + GRU and RNN + Bidirectional LSTM, and RNN + Bidirectional GRU to analyze their performance for the same labeled data in order to associate particular patterns of sound with their corresponding class of food item. During evaluation, the highest accuracy, precision,F1 score, and recall have been obtained by GRU with 99.28%, Bidirectional LSTM + GRU with 97.7% as well as 97.3%, and RNN + Bidirectional LSTM with 97.45%, respectively. The results of this study demonstrate that deep learning models have the potential to precisely identify foods on the basis of their sound by computing the best outcomes.

Benchmarking of Sensor Configurations and Measurement Sites for Out-of-the-Lab Photoplethysmography

Photoplethysmography (PPG) is used for heart-rate monitoring in a variety of contexts and applications due to its versatility and simplicity. These applications, namely studies involving PPG data acquisition during day-to-day activities, require reliable and continuous measurements, which are often performed at the index finger or wrist. However, some PPG sensors are susceptible to saturation, motion artifacts, and discomfort upon their use. In this paper, an off-the-shelf PPG sensor was benchmarked and modified to improve signal saturation. Moreover, this paper explores the feasibility of using an optimized sensor in the lower limb as an alternative measurement site. Data were collected from 28 subjects with ages ranging from 18 to 59 years. To validate the sensors' performance, signal saturation and quality, wave morphology, performance of automatic systolic peak detection, and heart-rate estimation, were compared. For the upper and lower limb locations, the index finger and the first toe were used as reference locations, respectively. Lowering the amplification stage of the PPG sensor resulted in a significant reduction in signal saturation, from 18% to 0.5%. Systolic peak detection at rest using an automatic algorithm showed a sensitivity and precision of 0.99 each. The posterior wrist and upper arm showed pulse wave morphology correlations of 0.93 and 0.92, respectively. For these locations, peak detection sensitivity and precision were 0.95, 0.94 and 0.89, 0.89, respectively. Overall, the adjusted PPG sensors are a good alternative for obtaining high-quality signals at the fingertips, and for new measurement sites, the posterior pulse and the upper arm allow for high-quality signal extraction.

Evaluation of the Chewing Pattern through an Electromyographic Device

Chewing is essential in regulating metabolism and initiating digestion. Various methods have been used to examine chewing, including analyzing chewing sounds and using piezoelectric sensors to detect muscle contractions. However, these methods struggle to distinguish chewing from other movements. Electromyography (EMG) has proven to be an accurate solution, although it requires sensors attached to the skin. Existing EMG devices focus on detecting the act of chewing or classifying foods and do not provide self-awareness of chewing habits. We developed a non-invasive device that evaluates a personalized chewing style by analyzing various aspects, like chewing time, cycle time, work rate, number of chews and work. It was tested in a case study comparing the chewing pattern of smokers and non-smokers, as smoking can alter chewing habits. Previous studies have shown that smokers exhibit reduced chewing speed, but other aspects of chewing were overlooked. The goal of this study is to present the device and provide additional insights into the effects of smoking on chewing patterns by considering multiple chewing features. Statistical analysis revealed significant differences, as non-smokers had more chews and higher work values, indicating more efficient chewing. The device provides valuable insights into personalized chewing profiles and could modify unhealthy chewing habits.

Health-Related Indicators Measured Using Earable Devices: Systematic Review

BACKGROUND

Earable devices are novel, wearable Internet of Things devices that are user-friendly and have potential applications in mobile health care. The position of the ear is advantageous for assessing vital status and detecting diseases through reliable and comfortable sensing devices.

OBJECTIVE

Our study aimed to review the utility of health-related indicators derived from earable devices and propose an improved definition of disease prevention. We also proposed future directions for research on the health care applications of earable devices.

METHODS

A systematic review was conducted of the PubMed, Embase, and Web of Science databases. Keywords were used to identify studies on earable devices published between 2015 and 2020. The earable devices were described in terms of target health outcomes, biomarkers, sensor types and positions, and their utility for disease prevention.

RESULTS

A total of 51 articles met the inclusion criteria and were reviewed, and the frequency of 5 health-related characteristics of earable devices was described. The most frequent target health outcomes were diet-related outcomes (9/51, 18%), brain status (7/51, 14%), and cardiovascular disease (CVD) and central nervous system disease (5/51, 10% each). The most frequent biomarkers were electroencephalography (11/51, 22%), body movements (6/51, 12%), and body temperature (5/51, 10%). As for sensor types and sensor positions, electrical sensors (19/51, 37%) and the ear canal (26/51, 51%) were the most common, respectively. Moreover, the most frequent prevention stages were secondary prevention (35/51, 69%), primary prevention (12/51, 24%), and tertiary prevention (4/51, 8%). Combinations of ≥2 target health outcomes were the most frequent in secondary prevention (8/35, 23%) followed by brain status and CVD (5/35, 14% each) and by central nervous system disease and head injury (4/35, 11% each).

CONCLUSIONS

Earable devices can provide biomarkers for various health outcomes. Brain status, healthy diet status, and CVDs were the most frequently targeted outcomes among the studies. Earable devices were mostly used for secondary prevention via monitoring of health or disease status. The potential utility of earable devices for primary and tertiary prevention needs to be investigated further. Earable devices connected to smartphones or tablets through cloud servers will guarantee user access to personal health information and facilitate comfortable wearing.

Red carbon dot directed biocrystalline alignment for piezoelectric energy harvesting

Herein, using chitin-derived chitosan, we first demonstrate the luminous carbon dot-directed large-scale biocrystalline piezo-phase alignment. This further significantly facilitates the piezo-energy harvesting of Earth-abundant natural biopolymers. A very small, yet moderate, number of red-emission carbon quantum dots (R-CQDs) allow a highly preferential macroscopic alignment of chitosan based, electrospun hybrid nanofibers and a highly preferential microscopic alignment of internal chitosan piezo-phase crystalline lamellae. Meanwhile, R-CQD hybridized bionanofibers maintain the long-wavelength photoluminescence excitation/emission of encapsulated, monodisperse R-CQDs. The piezoelectric voltage output and piezoelectric current output of hybrid bionanofibers reach up to 125 V cm^-3 and 1.5 μA cm^-3, respectively. They are more than 5 and 6 times higher than those of the state-of-the-art pristine ones, respectively. Moreover, the proof-of-concept red-emission bionanofibrous piezoelectric nanogenerator shows a highly durable, highly stable, and highly reproducible piezoresponse in over 10 000 continuous load cycles. As a reliable renewable energy source, it demonstrates the fast charging of external capacitors and the direct operation of commercial electronics. In particular, as a self-powered wearable tactile healthcare sensor, it attains ultrahigh mechanosensitivity in sensing a broad range of human biophysiological pressures and strains.

iHearken: Chewing sound signal analysis based food intake recognition system using Bi-LSTM softmax network

BACKGROUND AND OBJECTIVE

Food ingestion is an integral part of health and wellness. Continues monitoring of different food types and observing the amount being consumed prevents gastrointestinal diseases and weight-related issues. Food intake recognition (FIR) systems, thus have significant impact on everyday life. The purpose of this study is to develop an automatic approach for the FIR using a contemporary wearable hardware and machine learning technique. This will assist clinicians and concern person to manage health issues associated with food intake.

METHODS

In this work, we present a novel hardware iHearken, a headphone-like wearable sensor-based system to monitor eating activities and recognize food intake type in the free-living condition. State-of-the-art hardware is designed for data acquisition where 16 subjects are recruited and 20 different food items are used for data collection. Further, chewing sound signals are analyzed for FIR using bottleneck features. The proposed model is divided into 4 distinct phases: data acquisition, event detection using a pre-trained model, bottleneck feature extraction, and classification based on bidirectional long short-term memory (Bi-LSTM) softmax model. The Bi-LSTM network with softmax function is applied to calculate the identification score for apiece chewing signal which further classifies the chewing signal data into liquid / solid food classes.

RESULTS

The results of proposed model performance is evaluated in (%) for accuracy, precision, recall and F-score as 97.422, 96.808, 98.0, and 97.512, respectively, and root mean square error (RMSE), and mean absolute percentage error (MAPE) as 0.160 1.030 respectively for numbers of correct food type recognized. Further, we also evaluated our model's performance for food classification into solid and liquid and achieved an accuracy (96.66%), precision (96.40%), recall (95.230%), F-score (95.79%), RMSE (0.182), and MAPE (2.22). We also demonstrated that the food recognition accuracy of different models with the proposed model differed statistically.

CONCLUSION

An informatics complexity study of the proposed model was subsequently explored to review the effectiveness of the proposed wearable device and the methodology. The medical importance of this investigation is the reliable monitoring of the clinical development of the food intake classification methods via food chew event detection in the ambulatory environment has been justified.

Enhancing Nutrition Care Through Real-Time, Sensor-Based Capture of Eating Occasions: A Scoping Review

As food intake patterns become less structured, different methods of dietary assessment may be required to capture frequently omitted snacks, smaller meals, and the time of day when they are consumed. Incorporating sensors that passively and objectively detect eating behavior may assist in capturing these eating occasions into dietary assessment methods. The aim of this study was to identify and collate sensor-based technologies that are feasible for dietitians to use to assist with performing dietary assessments in real-world practice settings. A scoping review was conducted using the PRISMA extension for scoping reviews (PRISMA-ScR) framework. Studies were included if they were published between January 2016 and December 2021 and evaluated the performance of sensor-based devices for identifying and recording the time of food intake. Devices from included studies were further evaluated against a set of feasibility criteria to determine whether they could potentially be used to assist dietitians in conducting dietary assessments. The feasibility criteria were, in brief, consisting of an accuracy ≥80%; tested in settings where subjects were free to choose their own foods and activities; social acceptability and comfort; a long battery life; and a relatively rapid detection of an eating episode. Fifty-four studies describing 53 unique devices and 4 device combinations worn on the wrist (n = 18), head (n = 16), neck (n = 9), and other locations (n = 14) were included. Whilst none of the devices strictly met all feasibility criteria currently, continuous refinement and testing of device software and hardware are likely given the rapidly changing nature of this emerging field. The main reasons devices failed to meet the feasibility criteria were: an insufficient or lack of reporting on battery life (91%), the use of a limited number of foods and behaviors to evaluate device performance (63%), and the device being socially unacceptable or uncomfortable to wear for long durations (46%). Until sensor-based dietary assessment tools have been designed into more inconspicuous prototypes and are able to detect most food and beverage consumption throughout the day, their use will not be feasible for dietitians in practice settings.

Current Perspectives on Food Oral Processing

Food oral processing (FOP) is a fast-emerging research area in the food science discipline. Since its first introduction about a decade ago, a large amount of literature has been published in this area, forming new frontiers and leading to new research opportunities. This review aims to summarize FOP research progress from current perspectives. Food texture, food flavor (aroma and taste), bolus swallowing, and eating behavior are covered in this review. The discussion of each topic is organized into three parts: a short background introduction, reflections on current research findings and achievements, and future directions and implications on food design. Physical, physiological, and psychological principles are the main concerns of discussion for each topic. The last part of the review shares views on the research challenges and outlooks of future FOP research. It is hoped that the review not only helps readers comprehend what has been achieved in the past decade but also, more importantly, identify where the knowledge gaps are and in which direction the FOP research will go.

Bite-Weight Estimation Using Commercial Ear Buds

While automatic tracking and measuring of our physical activity is a well established domain, not only in research but also in commercial products and every-day lifestyle, automatic measurement of eating behavior is significantly more limited. Despite the abundance of methods and algorithms that are available in bibliography, commercial solutions are mostly limited to digital logging applications for smart-phones. One factor that limits the adoption of such solutions is that they usually require specialized hardware or sensors. Based on this, we evaluate the potential for estimating the weight of consumed food (per bite) based only on the audio signal that is captured by commercial ear buds (Samsung Galaxy Buds). Specifically, we examine a combination of features (both audio and non-audio features) and trainable estimators (linear regression, support vector regression, and neural-network based estimators) and evaluate on an in-house dataset of 8 participants and 4 food types. Results indicate good potential for this approach: our best results yield mean absolute error of less than 1 g for 3 out of 4 food types when training food-specific models, and 2.1 g when training on all food types together, both of which improve over an existing literature approach.

Self-Supervised Feature Learning of 1D Convolutional Neural Networks with Contrastive Loss for Eating Detection Using an In-Ear Microphone

The importance of automated and objective monitoring of dietary behavior is becoming increasingly accepted. The advancements in sensor technology along with recent achievements in machine-learning-based signal-processing algorithms have enabled the development of dietary monitoring solutions that yield highly accurate results. A common bottleneck for developing and training machine learning algorithms is obtaining labeled data for training supervised algorithms, and in particular ground truth annotations. Manual ground truth annotation is laborious, cumbersome, can sometimes introduce errors, and is sometimes impossible in free-living data collection. As a result, there is a need to decrease the labeled data required for training. Additionally, unlabeled data, gathered in-the-wild from existing wearables (such as Bluetooth earbuds) can be used to train and fine-tune eating-detection models. In this work, we focus on training a feature extractor for audio signals captured by an in-ear microphone for the task of eating detection in a self-supervised way. We base our approach on the SimCLR method for image classification, proposed by Chen et al. from the domain of computer vision. Results are promising as our self-supervised method achieves similar results to supervised training alternatives, and its overall effectiveness is comparable to current state-of-the-art methods. Code is available at https://github.com/mug-auth/ssl-chewing.

Short and Long-Term Innovations on Dietary Behavior Assessment and Coaching: Present Efforts and Vision of the Pride and Prejudice Consortium

Overweight, obesity and cardiometabolic diseases are major global health concerns. Lifestyle factors, including diet, have been acknowledged to play a key role in the solution of these health risks. However, as shown by numerous studies, and in clinical practice, it is extremely challenging to quantify dietary behaviors as well as influencing them via dietary interventions. As shown by the limited success of 'one-size-fits-all' nutritional campaigns catered to an entire population or subpopulation, the need for more personalized coaching approaches is evident. New technology-based innovations provide opportunities to further improve the accuracy of dietary assessment and develop approaches to coach individuals towards healthier dietary behaviors. Pride & Prejudice (P&P) is a unique multi-disciplinary consortium consisting of researchers in life, nutrition, ICT, design, behavioral and social sciences from all four Dutch Universities of Technology. P&P focuses on the development and integration of innovative technological techniques such as artificial intelligence (AI), machine learning, conversational agents, behavior change theory and personalized coaching to improve current practices and establish lasting dietary behavior change.

Counting Bites and Recognizing Consumed Food from Videos for Passive Dietary Monitoring

Assessing dietary intake in epidemiological studies are predominantly based on self-reports, which are subjective, inefficient, and also prone to error. Technological approaches are therefore emerging to provide objective dietary assessments. Using only egocentric dietary intake videos, this work aims to provide accurate estimation on individual dietary intake through recognizing consumed food items and counting the number of bites taken. This is different from previous studies that rely on inertial sensing to count bites, and also previous studies that only recognize visible food items but not consumed ones. As a subject may not consume all food items visible in a meal, recognizing those consumed food items is more valuable. A new dataset that has 1,022 dietary intake video clips was constructed to validate our concept of bite counting and consumed food item recognition from egocentric videos. 12 subjects participated and 52 meals were captured. A total of 66 unique food items, including food ingredients and drinks, were labelled in the dataset along with a total of 2,039 labelled bites. Deep neural networks were used to perform bite counting and food item recognition in an end-to-end manner. Experiments have shown that counting bites directly from video clips can reach 74.15% top-1 accuracy (classifying between 0-4 bites in 20-second clips), and a MSE value of 0.312 (when using regression). Our experiments on video-based food recognition also show that recognizing consumed food items is indeed harder than recognizing visible ones, with a drop of 25% in F1 score.

Deep Learning-Based Multimodal Data Fusion: Case Study in Food Intake Episodes Detection Using Wearable Sensors

BACKGROUND

Multimodal wearable technologies have brought forward wide possibilities in human activity recognition, and more specifically personalized monitoring of eating habits. The emerging challenge now is the selection of most discriminative information from high-dimensional data collected from multiple sources. The available fusion algorithms with their complex structure are poorly adopted to the computationally constrained environment which requires integrating information directly at the source. As a result, more simple low-level fusion methods are needed.

OBJECTIVE

In the absence of a data combining process, the cost of directly applying high-dimensional raw data to a deep classifier would be computationally expensive with regard to the response time, energy consumption, and memory requirement. Taking this into account, we aimed to develop a data fusion technique in a computationally efficient way to achieve a more comprehensive insight of human activity dynamics in a lower dimension. The major objective was considering statistical dependency of multisensory data and exploring intermodality correlation patterns for different activities.

METHODS

In this technique, the information in time (regardless of the number of sources) is transformed into a 2D space that facilitates classification of eating episodes from others. This is based on a hypothesis that data captured by various sensors are statistically associated with each other and the covariance matrix of all these signals has a unique distribution correlated with each activity which can be encoded on a contour representation. These representations are then used as input of a deep model to learn specific patterns associated with specific activity.

RESULTS

In order to show the generalizability of the proposed fusion algorithm, 2 different scenarios were taken into account. These scenarios were different in terms of temporal segment size, type of activity, wearable device, subjects, and deep learning architecture. The first scenario used a data set in which a single participant performed a limited number of activities while wearing the Empatica E4 wristband. In the second scenario, a data set related to the activities of daily living was used where 10 different participants wore inertial measurement units while performing a more complex set of activities. The precision metric obtained from leave-one-subject-out cross-validation for the second scenario reached 0.803. The impact of missing data on performance degradation was also evaluated.

CONCLUSIONS

To conclude, the proposed fusion technique provides the possibility of embedding joint variability information over different modalities in just a single 2D representation which results in obtaining a more global view of different aspects of daily human activities at hand, and yet preserving the desired performance level in activity recognition.

A Data Driven End-to-End Approach for In-the-Wild Monitoring of Eating Behavior Using Smartwatches

The increased worldwide prevalence of obesity has sparked the interest of the scientific community towards tools that objectively and automatically monitor eating behavior. Despite the study of obesity being in the spotlight, such tools can also be used to study eating disorders (e.g. anorexia nervosa) or provide a personalized monitoring platform for patients or athletes. This paper presents a complete framework towards the automated i) modeling of in-meal eating behavior and ii) temporal localization of meals, from raw inertial data collected in-the-wild using commercially available smartwatches. Initially, we present an end-to-end Neural Network which detects food intake events (i.e. bites). The proposed network uses both convolutional and recurrent layers that are trained simultaneously. Subsequently, we show how the distribution of the detected bites throughout the day can be used to estimate the start and end points of meals, using signal processing algorithms. We perform extensive evaluation on each framework part individually. Leave-one-subject-out (LOSO) evaluation shows that our bite detection approach outperforms four state-of-the-art algorithms towards the detection of bites during the course of a meal (0.923 F1 score). Furthermore, LOSO and held-out set experiments regarding the estimation of meal start/end points reveal that the proposed approach outperforms a relevant approach found in the literature (Jaccard Index of 0.820 and 0.821 for the LOSO and held-out experiments, respectively). Experiments are performed using our publicly available FIC and the newly introduced FreeFIC datasets.

NeckSense: A Multi-Sensor Necklace for Detecting Eating Activities in Free-Living Conditions

We present the design, implementation, and evaluation of a multi-sensor, low-power necklace, NeckSense, for automatically and unobtrusively capturing fine-grained information about an individual's eating activity and eating episodes, across an entire waking day in a naturalistic setting. NeckSense fuses and classifies the proximity of the necklace from the chin, the ambient light, the Lean Forward Angle, and the energy signals to determine chewing sequences, a building block of the eating activity. It then clusters the identified chewing sequences to determine eating episodes. We tested NeckSense on 11 participants with and 9 participants without obesity, across two studies, where we collected more than 470 hours of data in a naturalistic setting. Our results demonstrate that NeckSense enables reliable eating detection for individuals with diverse body mass index (BMI) profiles, across an entire waking day, even in free-living environments. Overall, our system achieves an F1-score of 81.6% in detecting eating episodes in an exploratory study. Moreover, our system can achieve an F1-score of 77.1% for episodes even in an all-day-long free-living setting. With more than 15.8 hours of battery life, NeckSense will allow researchers and dietitians to better understand natural chewing and eating behaviors. In the future, researchers and dietitians can use NeckSense to provide appropriate real-time interventions when an eating episode is detected or when problematic eating is identified.

Retrieval and Timing Performance of Chewing-Based Eating Event Detection in Wearable Sensors

We present an eating detection algorithm for wearable sensors based on first detecting chewing cycles and subsequently estimating eating phases. We term the corresponding algorithm class as a bottom-up approach. We evaluated the algorithm using electromyographic (EMG) recordings from diet-monitoring eyeglasses in free-living and compared the bottom-up approach against two top-down algorithms. We show that the F1 score was no longer the primary relevant evaluation metric when retrieval rates exceeded approx. 90%. Instead, detection timing errors provided more important insight into detection performance. In 122 hours of free-living EMG data from 10 participants, a total of 44 eating occasions were detected, with a maximum F1 score of 99.2%. Average detection timing errors of the bottom-up algorithm were 2.4 ± 0.4 s and 4.3 ± 0.4 s for the start and end of eating occasions, respectively. Our bottom-up algorithm has the potential to work with different wearable sensors that provide chewing cycle data. We suggest that the research community report timing errors (e.g., using the metrics described in this work).

Counting Bites With Bits: Expert Workshop Addressing Calorie and Macronutrient Intake Monitoring

BACKGROUND

Conventional diet assessment approaches such as the 24-hour self-reported recall are burdensome, suffer from recall bias, and are inaccurate in estimating energy intake. Wearable sensor technology, coupled with advanced algorithms, is increasingly showing promise in its ability to capture behaviors that provide useful information for estimating calorie and macronutrient intake.

OBJECTIVE

This paper aimed to summarize current technological approaches to monitoring energy intake on the basis of expert opinion from a workshop panel and to make recommendations to advance technology and algorithms to improve estimation of energy expenditure.

METHODS

A 1-day invitational workshop sponsored by the National Science Foundation was held at Northwestern University. A total of 30 participants, including population health researchers, engineers, and intervention developers, from 6 universities and the National Institutes of Health participated in a panel discussing the state of evidence with regard to monitoring calorie intake and eating behaviors.

RESULTS

Calorie monitoring using technological approaches can be characterized into 3 domains: (1) image-based sensing (eg, wearable and smartphone-based cameras combined with machine learning algorithms); (2) eating action unit (EAU) sensors (eg, to measure feeding gesture and chewing rate); and (3) biochemical measures (eg, serum and plasma metabolite concentrations). We discussed how each domain functions, provided examples of promising solutions, and highlighted potential challenges and opportunities in each domain. Image-based sensor research requires improved ground truth (context and known information about the foods), accurate food image segmentation and recognition algorithms, and reliable methods of estimating portion size. EAU-based domain research is limited by the understanding of when their systems (device and inference algorithm) succeed and fail, need for privacy-protecting methods of capturing ground truth, and uncertainty in food categorization. Although an exciting novel technology, the challenges of biochemical sensing range from a lack of adaptability to environmental effects (eg, temperature change) and mechanical impact, instability of wearable sensor performance over time, and single-use design.

CONCLUSIONS

Conventional approaches to calorie monitoring rely predominantly on self-reports. These approaches can gain contextual information from image-based and EAU-based domains that can map automatically captured food images to a food database and detect proxies that correlate with food volume and caloric intake. Although the continued development of advanced machine learning techniques will advance the accuracy of such wearables, biochemical sensing provides an electrochemical analysis of sweat using soft bioelectronics on human skin, enabling noninvasive measures of chemical compounds that provide insight into the digestive and endocrine systems. Future computing-based researchers should focus on reducing the burden of wearable sensors, aligning data across multiple devices, automating methods of data annotation, increasing rigor in studying system acceptability, increasing battery lifetime, and rigorously testing validity of the measure. Such research requires moving promising technological solutions from the controlled laboratory setting to the field.

The potential of photoplethysmogram and galvanic skin response in emotion recognition using nonlinear features

Recently, developing an accurate automatic emotion recognition system using a minimum number of bio-signals has become a challenging issue in "affective computing." This study aimed to propose a reliable system by examining nonlinear dynamics of photoplethysmogram (PPG) and galvanic skin response (GSR). To address this goal, two strategies were adopted. First, the efficiency of each signal in valence/arousal based emotion categorization was examined. Then, the proficiency of a hybrid feature, by combining both GSR and PPG features was studied. Lyapunov exponents, lagged Poincare's measures, and approximate entropy were extracted to characterize the irregularity and chaotic behavior of the phase space. To discriminate two levels of arousal and two levels of the valence, a probabilistic neural network (PNN) with different sigma adjustment parameter was examined. The results showed that the phase space geometry and consequently, the signal dynamics are influenced by the emotional music video. Additionally, distinctive patterns of the phase space behavior were observed under the influence of different lags. For both signals, the most irregularity was observed during the high valence, and the least irregularity was seen during the low valence. Consequently, signals' irregularity is affected by the valence dimension. The results showed that the fusion has more potential for emotion recognition than that of using each signal separately. For sigma = 0.1, the highest recognition rate was 100% in a subject-dependent mode. In a subject-independent mode, the maximum accuracies of 88.57 and 86.8% were obtained for arousal and valence dimensions, respectively.

End-to-end Learning for Measuring in-meal Eating Behavior from a Smartwatch

In this paper, we propose an end-to-end neural network (NN) architecture for detecting in-meal eating events (i.e., bites), using only a commercially available smartwatch. Our method combines convolutional and recurrent networks and is able to simultaneously learn intermediate data representations related to hand movements, as well as sequences of these movements that appear during eating. A promising F-score of 0.884 is achieved for detecting bites on a publicly available dataset with 10 subjects.

Modeling Wrist Micromovements to Measure In-Meal Eating Behavior From Inertial Sensor Data

Overweight and obesity are both associated with in-meal eating parameters such as eating speed. Recently, the plethora of available wearable devices in the market ignited the interest of both the scientific community and the industry toward unobtrusive solutions for eating behavior monitoring. In this paper, we present an algorithm for automatically detecting the in-meal food intake cycles using the inertial signals (acceleration and orientation velocity) from an off-the-shelf smartwatch. We use five specific wrist micromovements to model the series of actions leading to and following an intake event (i.e., bite). Food intake detection is performed in two steps. In the first step, we process windows of raw sensor streams and estimate their micromovement probability distributions by means of a convolutional neural network. In the second step, we use a long short-term memory network to capture the temporal evolution and classify sequences of windows as food intake cycles. Evaluation is performed using a challenging dataset of 21 meals from 12 subjects. In our experiments, we compare the performance of our algorithm against three state-of-the-art approaches, where our approach achieves the highest F1 detection score (0.913 in the leave-one-subject-out experiment). The dataset used in the experiments is available at https://mug.ee.auth.gr/intake-cycle-detection/.

Personalised meal eating behaviour analysis via semi-supervised learning

Automated monitoring and analysis of eating behaviour patterns, i.e., "how one eats", has recently received much attention by the research community, owing to the association of eating patterns with health-related problems and especially obesity and its comorbidities. In this work, we introduce an improved method for meal micro-structure analysis. Stepping on a previous methodology of ours that combines feature extraction, SVM micro-movement classification and LSTM sequence modelling, we propose a method to adapt a pretrained IMU-based food intake cycle detection model to a new subject, with the purpose of improving model performance for that subject. We split model training into two stages. First, the model is trained using standard supervised learning techniques. Then, an adaptation step is performed, where the model is fine-tuned on unlabeled samples of the target subject via semisupervised learning. Evaluation is performed on a publicly available dataset that was originally created and used in [1] and has been extended here to demonstrate the effect of the semisupervised approach, where the proposed method improves over the baseline method.

The SPLENDID Eating Detection Sensor: Development and Feasibility Study

Background

The available methods for monitoring food intake—which for a great part rely on self-report—often provide biased and incomplete data. Currently, no good technological solutions are available. Hence, the SPLENDID eating detection sensor (an ear-worn device with an air microphone and a photoplethysmogram [PPG] sensor) was developed to enable complete and objective measurements of eating events. The technical performance of this device has been described before. To date, literature is lacking a description of how such a device is perceived and experienced by potential users.

Objective

The objective of our study was to explore how potential users perceive and experience the SPLENDID eating detection sensor.

Methods

Potential users evaluated the eating detection sensor at different stages of its development: (1) At the start, 12 health professionals (eg, dieticians, personal trainers) were interviewed and a focus group was held with 5 potential end users to find out their thoughts on the concept of the eating detection sensor. (2) Then, preliminary prototypes of the eating detection sensor were tested in a laboratory setting where 23 young adults reported their experiences. (3) Next, the first wearable version of the eating detection sensor was tested in a semicontrolled study where 22 young, overweight adults used the sensor on 2 separate days (from lunch till dinner) and reported their experiences. (4) The final version of the sensor was tested in a 4-week feasibility study by 20 young, overweight adults who reported their experiences.

Results

Throughout all the development stages, most individuals were enthusiastic about the eating detection sensor. However, it was stressed multiple times that it was critical that the device be discreet and comfortable to wear for a longer period. In the final study, the eating detection sensor received an average grade of 3.7 for wearer comfort on a scale of 1 to 10. Moreover, experienced discomfort was the main reason for wearing the eating detection sensor <2 hours a day. The participants reported having used the eating detection sensor on 19/28 instructed days on average.

Conclusions

The SPLENDID eating detection sensor, which uses an air microphone and a PPG sensor, is a promising new device that can facilitate the collection of reliable food intake data, as shown by its technical potential. Potential users are enthusiastic, but to be successful wearer comfort and discreetness of the device need to be improved.

Automatic Analysis of Food Intake and Meal Microstructure Based on Continuous Weight Measurements

The structure of the cumulative food intake (CFI) curve has been associated with obesity and eating disorders. Scales that record the weight loss of a plate from which a subject eats food are used for capturing this curve; however, their measurements are contaminated by additive noise and are distorted by certain types of artifacts. This paper presents an algorithm for automatically processing continuous in-meal weight measurements in order to extract the clean CFI curve and in-meal eating indicators, such as total food intake and food intake rate. The algorithm relies on the representation of the weight-time series by a string of symbols that correspond to events such as bites or food additions. A context-free grammar is next used to model a meal as a sequence of such events. The selection of the most likely parse tree is finally used to determine the predicted eating sequence. The algorithm is evaluated on a dataset of 113 meals collected using the Mandometer, a scale that continuously samples plate weight during eating. We evaluate the effectiveness for seven indicators and for bite-instance detection. We compare our approach with three state-of-the-art algorithms, and achieve the lowest error rates for most indicators (24 g for total meal weight). The proposed algorithm extracts the parameters of the CFI curve automatically, eliminating the need for manual data processing, and thus facilitating large-scale studies of eating behavior.